Local First and TypeScript’s Missing Library with Johannes Schickling

Welcome to SITAC. So today, we've got an awesome show for you today. We have Johannes Schickling on today. You may know him, founder of Prisma, but we're not here today to talk about Prisma.

Johannes JS, honestly, like, one of the people that I watch for what's coming in web development, because he was early on GraphQL, very early on TypeScript.

Seems like he's very early on local first right now. So quite honestly, when he's working on something, we're gonna watch. You know? So, he's working on 2 things right now, which is I'll I'll allow him to explain a little bit more, but Overtone is a local first music app that sits on top of Spotify, Apple Music, things like that. And Effect, which is a TypeScript's missing, standard library. It's kind of a library for writing TypeScript apps that are composable or reusable, schema validation, error handling, some pretty nifty things in there.

So we're gonna talk to him about both of those types of things as well as, like, diving a little bit more into to local first. So welcome. Thanks so much for coming on.

Yeah. Thanks so much for having me. Super excited to be on the show.

Yeah. Yeah. Before well, before we get into it, Wes, one of the things that affect us really well is it makes working with your errors really organized and easy. And you know what? So does Sentry. This show is brought you by Sentry, the perfect place to track and log all of your errors and solve them for the bugs that effect is going to save you from having in the 1st place. So we'll be talking about effect a little bit more in just a second. But let's get into 1st and foremost, overtone. Do you wanna talk a little bit about, what overtone is and maybe why it's interesting and unique?

Yeah. Totally. May I I'll start with the the why first and then then to the what. Since, like, in the past, as you've mentioned, in the past, I was working on on Prisma, and I built Prisma in order to help other app developers deal with some of the the insufficiencies that I've been facing when building apps.

But after I was able to take a sabbatical from Prisma, I was really urging to get back into the app development shoes myself again.

And there was, like, Node particular thing that's been on my on my mind for a long time as I'm a big music enthusiast. And, like, I listen to music from a lot of different places.

And I never really found, like, a good home for my music. So I was hoping there eventually to be, like, the right music app for me in the same way I'm using Superhuman, for example, as my email client. Or for so many things, I can have my own client and like a client does for me. But I never found that for for music.

And then I also, at the same time, came across, like, some other apps which really inspired me in terms of being high quality, super fast. And since no one built the right music app for me, I figured, okay. This is a good app for me to kind of just, get up and running as an app developer again. Like, I started building a little prototype, etcetera. And one thing led to another.

And I really fell in love with the entire use case and within the entire approach. And it's see, it developed more and Node. And I really saw Otus has a lot of potential.

So this is me working coming from a little prototype into developing this into a, ambitious idea. And right now, the idea is really to, like, as you can use Superhuman or a mailbox or whatever email client you want to use, I want to have the same experience for music.

Since, like, I listen to music on YouTube, on Spotify, on SoundCloud.

I buy my own stuff from Bandcamp and from from other platforms.

And, this is kinda scratching my own itch, and it's such a huge community.

Yeah. And and so you mentioned Superhuman there. For people who might not know how great superhuman is, I was one of those people who before I used it, I was like, there's no way it's that good. Can you can you maybe go into just a little bit more detail in terms of, like, what what does that experience feel like? Not necessarily superhuman, but, like, the experience you're trying to reach that you get in things like superhuman.

Yeah. I mean, I start out with, like, to just underline your anecdote. It's it's Yarn. Some things you have to try out for yourself to really Yeah. Appreciate, what the rave is all about. For I think for Superhuman, they advertise with just like the speed, the productivity, and just most people don't love doing email. So whatever helps you get through that more quickly and more delightfully is a huge deal.

And for for me about music, music for me JS, like, all about joy. All about, like, the emotions that music evoke in me and, like, that I can, like, go from 1 track to another, what people share with me.

But right now listening to music, either you're all locked ESLint, like, a single platform like Spotify or something else. And like someone sends me a link of like, oh, like, I don't have access to that platform or or so this is like the joy gets lost very, very quickly. And when I find something good, that just like becomes another browser tab, maybe a bookmark, whatever. So I never really had that platform or like the tool that natively works for music. So I wanted to have a joyful place where it can just, like, let my music stay. In the past, we had, like, maybe on a shelf ahead, like, my music collection. I would feel really proud about that. And we no longer have that. We maybe have, like, a Spotify ESLint.

Literally, like, a track that I fell in love with, like, 3 weeks ago.

I went on a road trip, saved it for offline. I came back, and Spotify since removed it. So I can no longer listen to it. And this sort of stuff key JS, like, a 1,000 paper cuts. And I just want to make sure that music is enjoyable.

The paper Scott are Node. And it's it's it's nice to listen to music and and deal with it. And so but for emails, all about productivity. I think for music, it's all about joy. And I want to make sure that it just works and the software is is doing what you would expect. And that means I want to raise the level, the the quality bar to a level where maybe, like, macOS has set it once with the finder or where Figma has set it in terms of, like, how you can work with a design tool. And I feel like so many apps are being rebuilt as web apps.

And the what was once like a native app, and you could really strike for high Sanity.

That's becoming so much harder, and things rather feel like websites. You scroll somewhere, reload the the Node. Like, it has forgotten where you were before.

And so I wanna, turn the dial to all the way max, all the way to best experience in the domain of music.

Wow. And what makes a good music app for you then JS and and does that play into it being local first?

So what I found JS the the best enabling paradigm is like, my theory is that local first is the best foundation to build high quality apps in general.

Superhuman is one example. Figma is also actually, pretty local first inspired.

Another one that a lot of software developers use is like a linear. That's also local first. So a bit of like a theory, that's unfolding is that I think the the next generation of, like, all the next apps, I think, will all, if they want to be the best, I think they will, one way or another, go in a local first direction as you already see with linear.

And Yeah. I think in the case of what makes JS important for Overturn as an as a music app. 1st and foremost, it's like, it should play back music in a way that you have just nothing to be annoyed about. If you hit that play button, it should start immediately. If you hit the pause button, it should stop immediately. If you, like, select something, it should everything should just be very immediate.

No loading spinners. All joy.

Yeah. But that's harder harder,

Yarn, easier said than done. What are what are your thoughts on on starting a mobile or a local first app as a local first app instead of taking an app that exists and trying to add local first features? In my experience, trying to make an existing app local first was very, very painful. Is that something that you've noticed? Is there any improvement in that that world?

So a bit tongue in cheek, I would say, is, like, the entire term local first is right. So you start with that way. And if you don't, that becomes kinda tricky. So if you have, like, a more, like, say, like, Rails inspired app JS, like, we build most web app these days, whether we use something like Next. Js or Remix. They all sort of followed a similar paradigm where, like, the user experience is on the client, but more the most important stuff is all happening on the server. And this is like almost this tug of war in terms of like, oh, like, who who has, like, the authority in this? And, this creates this tension. And what you're trying to reconcile now is, like, move more, like, empower the client Node. But, fundamentally, you've, at some point, made a decision, Node. Like, the entire authority sits with the server.

And that creates, like, that really tough tension that's tough to tough to reconcile.

And what Local First is all about is really say, like, hey. Let's empower the client as much as possible, as much as the domain allows.

And in many domains such as like something like teal drawer, notes app, etcetera, all the stuff that's being created is being created on the client to begin with. The server just plays like a mediating role that, like, helps you to to get stuff from Node device to your collaborator.

And those make great use cases. And if you flip it really and ideal maybe from the start, maybe through a bigger rewrite, empower the client as much as possible. That makes it really easy. But once if you start out from, like, a server authority all the way, then Yeah. The like, it's it rather you you try to take a step in that direction, but that's typically just done as like a more and more glorious cache. But you need to flip it and, like, say, no. This is actually not a cache. This is we have, like, a a data architecture where this is the source of truth.

And the server is almost like a a cache in in some way.

Interesting.

And let's let's talk about the actual tech behind something like this because you think, like, you just said, empower the ESLint, meaning they could do as much work on the client as possible, and then the server is is just a cache, which is a really interesting way to think about it versus, like, when I think of a website, I think, like, yeah, the server does a lot of the the work, you know, and then the client is we're just clicking and displaying data there. So if you're building something like Vercel, you've Scott, like you obviously have, like, a database. You need somewhere to do some sort of, like, processing of your work. Like, what are the other parts of of building in an app maybe in the context of something like overtone?

So what might be surprising to to most app developers is, like, for overtone JS it is right now, I don't have a server. It's all the client.

Like, good old Winamp.

Fun fact, Winamp apparently still has, like, 42,000,000 active users. But aside from that, like, Winamp That's crazy. Is just like a local client app that plays music through like, Winamp was built in a certain time. By now Wes have cloud. Most music comes through some, like, a platform like Spotify, etcetera. But for my music client, the Spotify servers are doing the work. I just need to be a client for that. So the use cases for which I need like a dedicated overtone server are so minimal that I have started out building overtone and not yet having that server. A bit further down the road, I will add 1. And that's only responsibility will be to facilitate the syncing between the Vercel. Since we're not yet living in a world where peer to peer works everywhere, like you might be in a, like, a cafe, and you have your laptop, and you have your iPhone, and you can't quite rely on the does communicate directly with each other.

So they might need to go through some server that facilitates that.

But, the syncing, which is very intuitive, I feel like I have stuff on Node device and another. And now they need to talk to each other.

And a server facilitates that. But the server is like Node authority in that regard. So I try to do as much as possible on the client. What you've mentioned, like, the the sort of like little check Bos that we associate with, building an app, like a database or a server. All of those jobs to be done happen in the client. I literally have a database baked into the client. Okay. And what what when are you using there? Yeah. So this is where I had to reinvent the wheel a little bit for my use case. I'm, I consider myself to, to be kind of like a enjoying pioneering work, technical pioneering work Nice. Where I feel like, oh, there has to be a better way. Is this crazy? Probably, yes. Let me try. And then, kinda like 2 steps forward, 1 step backwards, and, like, then rinse and repeat.

So, and I was lucky enough in 2020 to be connected, to some other great people, namely Jeffrey Litt and Nicholas Schiefer, who have done a 2 Yarn, research project together with that was called Riffle. You can find it in Riffle dot Systems.

The idea was exactly that. Like, what if we want to build richer client side apps? What do we need for that? And an idea was like, hey. Let's actually what do we need to build apps? We need to deal with data. And what best primitive there is there to deal with data? Well, it's a database.

But traditionally, you always had like these databases in a server or possibly even more traditional where we had only desktop upgrade software. Yeah. This is where we had a database actually in the client side apps. We kind of forgot about that over a few generations.

Moved it all to the server. But now I think the pendulum is swinging back a little bit. So the idea was like, hey, let's bring this primitive to manage data, a database, into the client directly. There were a few technical breakthroughs or innovations that now facilitate that in the web environment, a browser environment, namely WASM.

So the idea was to use SQLite as an embeddable database, bring it directly into the web app. SQLite typically is rather kind of more request response. You send it a query, gives you the result back, end of story.

But what you want in a web app is that it's reactive. This is what we're used to from Mhmm. Like Redux, MobX, Sushtant, all of those sort of, like, client side state management tools.

But so this is what we had to do then. Like, we added a layer on top of SQLite that made it reactive.

We also didn't just want to express everything as SQL queries.

By now, we also use things like signals, etcetera.

So we added like a signals like layer also on top of it. So we have reactive SQLite. You can compose it together with things like signals, but also add a layer of like transactionality there. That's really important, when you, for example, click an overtone on play.

And you might see the same track maybe in a track ESLint, but you also see it at the play bar Mhmm. In your track cue. You don't want that to kind of like take many renders that, like, one thing shows a play state, the other still say, shows a pause state, the other shows maybe another track. You want, like, almost asset guarantees from a traditional server side database Okay. In your client side app that everything happens at the same time. So all of this, you're like, set out for this goal. We want better state management locally.

And this is what led to Riffle, which is the foundation for overturn.

Wow. Wow. How do you handle authentication in this kind of setup?

So the I would also like invite to switch the perspective a little bit. In a server side context, the server is not yours. The server is possibly exposed to everyone.

You you absolutely need authentication.

In a client side context, you like, how do you do let's say you have, like, text ESLint or notepad or something like that. You you'd not think about how do I do authentication for Node. You just open it. You put the stuff in it. And, like, you you save a file on your on your hard drive. And that's it. Like you having physical access to your machine is like all the authentication you need. And for a lot of local first software, that's enough. If the data is on your device or maybe you piggyback on something like Icloud, Then you don't even need authentication if you can embed yourself in an existing system.

If you want to build applications that where that looks a little bit more like a social network Wes everyone's connected to everyone, that's less of a good fit for for Local First. But typically, if you're, like, in a circle of trust, like, in the context of, like, a company or a family, etcetera. This is where Local First works really, really well. And this is you just Node, like, a one time getting access. You could, like, punt on that and say, like, you could just have access to that in your like, maybe you're using Tailscale or something.

So you really don't necessarily need authentication as much. It can be almost more of like a binary thing, that you can handle very easily.

And in the case of overtone, like, Spotify does authentication. You just enter your Spotify credentials. I connect to you. And there there's nothing more.

Yeah. So the authentication is you have implicit access to the database. Right? Like, if you are It's your database.

Yeah. It it is your your database. So it's it's either on your computer, which is protected by other things, or, it's either on a local network, which is already a privileged environment. Right? Or you said, like, use Tailscale, which is, like, you get VPN, and then you have access to that type of thing. Yeah. Or even, I guess, for syncing, you would potentially, if you're not using Icloud or whatever, you could potentially

auth into a a server somewhere to sync or to set up syncing, I Wes. Exactly. Yeah. Okay. Yeah. And it's very use case specific. What does your domain require

and then you do it? Whereas, like, if you're building an app with Next. Js, etcetera, this is where we have all the checkboxes. How do you do file upload? How do you do, like, authentication? How do you do Yeah. And so now we can rather focus on the domain and then really see, like, oh, we don't need to do any of those things because it just works. Like, if you open teal draw, there's like you can just use it right away and still, it's still there in your browser.

And that's so liberating. Yeah. You know what? It's it's funny that I I noticed sorry, Wes. One thing I noticed is that people oftentimes will start with the tech first and then work to the app that they're building. Alright. I need to build a site. I'm gonna build it on Next. Js, and these are the features that I need instead of starting with the problem in hand first and then picking the tech from that problem. And that really the way you're talking about you know, thinking about which apps specifically work well with local first, I think that's a a big consideration a lot of people need to have when even choosing even just the conversation of server side versus client side rendering in those aspects of what you actually need out of the network. I I think those are all big considerations.

Exactly.

One cool thing about this entire thing is also that aside from it being a bit new, it can also there's so much stuff that we invented over the Yarn. And then, like, we haven't even touched on, like, infrastructure. Where do you deploy? Do you put it in AWS? All of, like, the different AWS things.

Local first can really kinda back to the roots, back to the what you really Node. And it collapses so much of, like, the stack Node that we have created over the years that it can really, like out of 10 things that you need for a traditional app, with local first, you can get away with doing 2 things and then spend the rest of your effort on making the upgrade.

I'm I'm curious where, like, heavy lifting happens when you're doing a local first application. Right? So you you spin up. You have a scripts dot JS file. You do work in there. And at some point, you're either going to need to do something JavaScript. Right? And if that's the case, you usually will think, okay. Well, could I use a Wes Worker for this? Can I use some sort of WASM package that I can run some non JavaScript language but still run it in the browser? What's your approach to that?

So I think it it always depends. Right? So depending on the, on your use case, depending on the the shape of your data, the cardinality of your data, etcetera, you might need different trade offs. So I think something like tldraw, for example, can get away with an entirely different architecture, compared to to something like linear.

In TL draw, you the entire complexity stems from sort of like the the the rendering. It's it needs to be really really smooth, but the amount of data is probably much smaller.

In linear, in in a larger company use case, you might have thousands of issues. Each of them have, like, different comments, etcetera. So this is where it might have much more data, but the the frame rate might not matter as much. So all of this requires different trade offs.

So to build a local first app, particularly today, Wes not everything is like fully there's no rail for local first yet. So I need to Yeah. Build more stuff yourself. So this is why I pnpm pioneering this. But I think in a couple of years from now, this will actually be, in many cases, easier than to take, like, your traditional web app and try to make it linear grade.

So right now, it really you need to figure out you need to understand your trade offs and then make those trade offs decision, accordingly.

So typically, good rules of thumb JS, like, protect the main thread. So on the main thread, you have, like, whatever stuff you're doing might block it and that results in jank. So, like, you click on something and then you can't scroll or whatever. So if it's possible to do something on a worker, I think that's typically a good idea. However, that might also come not for free in terms of a sim simplified mental model. In a traditional React app Wes you might use something like React state, use state, etcetera, this is where if you click a button that causes a set state, it's right there in the next render.

However, if you would now do something that would cause some change on the Wes worker, then that needs to go through an async boundary. So at that point, you already have a distributed system. Stuff can get out of sync somehow, and stuff gets more tricky.

So you wanna find a good balance where the stuff that needs to happen synchronously happens on a main thread. The stuff that can happen out of band, you may be move it to a worker.

But typically, it really depends, like, on TL draw. You need to be really careful about, like, the the rendering, the canvas.

In other cases, you need to deal with more state management.

For the case of overtone, I found, like, this balance that I embed SQLite in the main thread. So I need to be careful about my frame budget, but SQLite is so incredible incredibly fast that I can, per frame, run dozens of queries, at the same time without blocking the main thread, and it can get that beautiful synchronous simplicity.

Mhmm. And then I do, like, other heavy lifting on other threads. So in overtone, I also need to deal with, like, cover art and, like, transcoding media files, etcetera. So I do all of that on worker threads.

Some of that code is, like, implemented in Rust and then compiled to WASM, Wow. But you could do that however you want. Wow. And the SQLite

library that you're using, is that just a library that looks like SQLite and and saves to, like, IndexedDB? Or is it like a WASM library? Or like, is it actually SQLite? How does that work? Like, how did you get SQLite running in the browser, I guess, is the question.

Yeah. So there there's a few different, pieces to this.

So if if you wanna learn more about this, we've written up, like, an essay on this on on riffle dot systems.

So this is where you can understand the details a little bit more. So there's a few ingredients for this. Number 1 is getting SQLite to run-in the browser in the 1st place.

There's been a bit of, like, a history to this. So there's been SQL dot JS, which I think back then has been pre WASM already been possible to run-in the browser, not as as performantly, but already. And that's quite a feat of engineering.

But at some point, it was also then possible to compile SQLite to WASM and run it there.

So there's, like, 2 parts to this. 1 is, like, running SQLite. SQLite can run-in memory, or it can also run, on top of, like, some persistence layer.

Persistence layer, let's say, use an electron app or let's say use a unlike an app on your Android or iPhone.

This is where SQLite is then persisted to your hard drive or in Electron also to your hard drive with, with, like, just a file somewhere. Yeah. And so in the browser, you either run it in memory Wes it's super fast, but it's not persisted to anything, or you then also persisted to something. In the browser, we have limited options to where to persist something to you in, like, in a reasonable performance way.

So, the traditional option has been indexed to be over the Scott past couple of years, but there JS Node also a new option called OPFS, original private file system.

That's what I'm using, And that's taking care of the persistence.

Oh, okay. For the case of Riffle, we've taken 1 step further. We wanted to get the best of both worlds of having super fast queries. You get the fastest queries by doing everything in memory. And you can get synchronous queries by doing it on the main thread. So we run both SQLite on the main thread. But then we also sync it to another SQLite version running on a worker, which is responsible Wow. For the persistence.

So you just need to know all of this if you care about it and if you are so crazy that you work on something like Riffle.

But from a user perspective, from an application developer perspective, it's basically as simple as using something like MobX.

And this is just stuff you can geek out on if you're curious.

So that's built your state management is the same thing as your database layer.

Exactly. Right? Oh, that's awesome. Exactly. This is basically gives you if you wanna have something like MobX, but a bit more power, and you also want persistence and so on, this is what we've been going for with Riffle, which Riffle's concluded as a research project. And I'm now developing it further under new name called Life Store. And I'm hoping to open source that at some point as well. Ah, sweet. Yeah.

One thing I had seen before was w a Wes q lite.

Is that in any sort of relation? Is it less featureful than what Riffle is trying to accomplish? So,

Riffle currently uses a SQLite wasn't built, that is by the official SQLite team. Mhmm. There's a couple of, like, other projects that also experiment with different trade offs, different optimizations to bring SQLite into the browser. Then most notable besides the official SQLite wasn't built JS the WA SQLite build. And that's been really innovating on a couple of, like, on a couple of fronts in terms of optimization, speed improvements, etcetera.

That's a really cool one.

And, yeah, I think it's a by by a person, Roy Hashimoto, I think.

Yeah. That's brilliant. But Riffle's JS like a layer on top of that. Cool. Okay. Yeah. Because I'd I'd first heard about DWA Escalate from Electric SQL.

It's a really cool project too. Fun fact, I'm actually integrating LifeStore and Electric SQL since they work really well together. So I'm I'm an adviser to those folks as well. And this this works really well together since LiveServer is all about the client side state management.

And Electric SQL is all about, oh, you have an existing Postgres database, and now you want to sync somehow to the clients. So this fits together like a glove.

Yeah. Awesome.

Let's talk about your TypeScript project as well. This has been pretty popular in the last, couple weeks. People have been talking about it. It's effect, which is I'll I'll let you give give us the pitch. What is effect for TypeScript?

So I should clarify. It's not my project. It's a project that I got involved with, like, 3 or 4 years ago And Okay. Since then have tried to contribute to and help other people better understand it. And still, overall, a a work in progress in both in terms of, making it easier for people to pick up and JS it's developed. But we've just recently hit our first production milestone given that we took over the effect package on NPM, which already had a 1 and 2 point o release.

Okay. Now we released it as 3 point o, which is technically our 1 point o.

But it's been already in development since, like, 6 years. And in JavaScript dog years, I think this makes the project quite old already and quite mature, and that speaks to the ambition and the the thought that has gone into it.

So I don't just rethink a little bit of, like, how I'm building overtone in terms of state management, in terms of following the the local first ideas.

But I also felt like, okay. The the way how I'm generally building the business logic with JavaScript, with TypeScript, there's been so many things that have been bothering me in terms of, like, how error handling works, etcetera. So when I set out to build this project, I landed on Local First. I landed on, like, those state management ideas.

But I still had all of those other things, that bothered me. So I've been looking around, like, how should I build this? May I Wes even considering building it all in Rust since I think Rust has a couple of, like, by now, like, better trade offs in terms of how the the actual program language works, etcetera. But I wasn't quite ready to slow down my app development speeds by having to to babysit the memory management all day. So it's still drawn to the affordances that JavaScript and TypeScript gives you. But I was looking for some of, like, those more modern ideas how to build an app and the better primitives.

So in the same way as, like, back in the days when I built some websites, some web apps with jQuery, and then React came along and just blew our minds by, like, this like the abstractions that provided. Like, the apps now we could think in terms of components and really freed up freed us up from having to think about moving DOM nodes around, etcetera. And we we could care about those components. They would fit together, etcetera.

And I feel like there's something like that missing in the JavaScript world. Sure. We have functions, etcetera, but it just tell you half the story. You can like you have some parameters that you can put into a function. You get something back.

But that's not all. Like so many of so much of your code depends maybe on a global thing somewhere or it depends on, like, a database connection already being established.

Or it's like you need to really be aware of like, oh, that thing could, like, throw with a user not authenticated error at any moment. Mhmm. But also with other errors that you don't you've already forgotten about that somewhere deep down in your program could go wrong. So there's Yeah. Like, there the stuff that's typically on our mind is like the happy path. But there's so much Node happy path that we just don't have any affordances in JavaScript and TypeScript to to help us with that. And this is where I've been shopping around for, like, the the right ideas.

And then ultimately found those right ideas in some other programming ecosystems, namely like Rust ecosystem, Swift ecosystem, Scala ecosystem. But I wanted to have those in JavaScript. So I was looking for, is there some project that takes those better ideas and implements them in JavaScript? This is how I found Effect, like, 3 or 4 years ago, Back then, still under different Node.

Very, very tricky to get into. It was all about functional programming back then. Since then, it had really deemphasized that part in the same way JS, like, React is all about function programming, but no one knows that really. So it's kind of been this Trojan horse, that turn, like, every React developer unknowingly in a function programmer. Same for effect.

It's basically takes some of those good ideas, but makes them more approachable.

And this has been the foundation for, like, how I'm building everything with TypeScript these days. This just gives you a little bit of, like, a motivational blurb at this point. Yeah. But I can get to the what is it.

Yeah. Yeah. Because someone's someone's listening right now to say, is it a framework? Is it a React alternative? Is it state management? Is it just a bunch of TypeScript helpers? What is it?

So I think in a way, like, that you could frame it from each of the angles that you've just mentioned And some and to 1 to go 1 step even further, as we've just launched the the 3 ESLint o a couple of weeks ago, on on Twitter, a couple of people, like, framed it as such like, oh, effect is almost like it's a new programming language, but within TypeScript.

So in a way, I would say this is right. Something like Rescript or, whatever those things were warp called before, they're they're all, like, trying to go take quite a big step away from JavaScript and say like, hey. JavaScript is is beyond repair. We need a new program language to bring bring along those great ideas, and then we compile to JavaScript.

Effect has like a similar ambition and say, like, hey. There's so many things that we could do better, but the world clearly isn't ready to move to another program language. JavaScript, TypeScript is here to stay. So let's try to see how can we bring as many of the great ideas into TypeScript with, like, all the good stuff that we have today. There's so much good good stuff that is happening today Wes, like, you have no new environments such as Bos, Deno, all of that trying to make things better. You have better testing stories with retest.

You can run JavaScript everywhere. You have Versus Node with TypeScript. All of that is fantastic.

But now we need to innovate a little bit more on the way how do we actually write our applications.

And so this is where effect similar to how React is all in JavaScript, all all in TypeScript, and gives you some new ways how you can structure your app, namely components, etcetera. And then you mount it somewhere, and then everything just works.

The effect is similar to that, but kind of for the complementary domain, which is not about view programming, but more about general business logic.

So let's say you have your, like, async functions where you call 1 async function from another and then in there you throw, etcetera.

Effect lets you structure those things as so called effects. There's actually like some academic research behind that that over probably decades build up those, like, new primitives.

Those are called effects. You can call 1 from another. Think about them as like kinda like a ESLint function with superpowers.

And the superpowers are that you have much better control over how errors happen. So you also on a TypeScript level, you know, oh, these errors can happen.

But you also get things such as, like, type safe context. You might be familiar with, like, the React context. In React context, in your components, you can access a context value anywhere in your in your React tree.

But you really need to make sure that somewhere outside you wrap all of that in, in a context provider. Otherwise, like, you get default value or it just blows up.

So you don't have any type safe affordances to like if you use a certain thing deep down, React doesn't tell you, oh, you still need to provide that. With the fact you get a type safe guarantee for that. So if somewhere in your program, you might use a, like, let's say, like, a s three thing that, like, you can access a file from s three, then effect on a type save level or rather TypeScript will force you like, hey, you can't yet run this entire program.

You first need to provide us like somehow with like a s three connection.

So if you want a deep like, sometimes that's also referred to as, like, dependency injection.

Dependency injection has a really bad rep because it's kind of been always, like, fingers crossed and, like, does weird type things with the fact that becomes really, really beautiful. And, like, all like, TypeScript has such, powerful type inference. That means, like, if you do something somewhere, the return type automatically becomes the right thing. And now Wes basically expand the same nice stuff that we use and love on daily Bos.

Not just for the return value, but also for errors, also for the context.

And so effect just gives you those, like, different layers of new superpowers.

Better concurrency management is also another one. Like, in TypeScript right now, what like, we're we're mostly Node deal with, like, promise Scott all. But if you want more over that, this is like where where things stop. Like, let's say you have, like, you wanna go through a 100 HTTP requests that you wanna go through. Do you say promise at all, like, a 100 URLs? No. Then you do, like, a 100 at the same time. So you might want to do, like, 10 at the same time.

But then, like, if you have done 9, do you need to wait for the last one? So you you want much better control over over, like, concurrency as well. You might then reach for some, like, PQ or PMAP.

And this is where we have, like We just we just finished a show on this. We recorded last last week. We were describing a whole bunch of ways to work with promises and queuing and concurrency and and synchronous, and it's true. Like, the whole business logic of working with multiple promises is kinda held up by Syndrais Orahas' packages for that that type of thing. And what I'm maybe you can tell me if I'm understanding this correctly. Like, a fact is just like when we got promises in JavaScript, things got really good because we were able to compose things and take take Node promise from a random package and another promise from another package, and they they work together because it's a standard primitive that we have in the browser. So my understanding of this effect is that if we have everything as an effect in our application, we'd be at better type safety, and then we're able to do the business logic of the asynchronous code, error handling, which is a big pain in in TypeScript, not being able to type your errors without doing a whole bunch of wrapper functions around it, retrying concurrency interruption.

I I think I understand it. Like, you have this, like, primitive, which is an effect.

And these common things that we all do, retrying, error handling, interrupting, composing, queuing, that's all built into this library called effect.

Exactly. And there's just like I could keep going on now about new or better superpowers, like, for example, observability or interruption, abort controller stuff. All of that, like, everything gets better with effect. Like, in the same way JS, like, everything got better with components.

Once you started embracing components, so much good stuff happened. Like, you could share them. You could compose them. You could, like, do so many things with that. And this is what effect enabled for, like, general purpose programming.

And, it it is quite a little Tolinski, it didn't take you just, like, 5 minutes to get started with React, like having, known something like jQuery before. You need to rewire your brain a little bit. Yeah. But once you get over that hump and by now it's default. But once you get over that hump, you've rewired your brain. Everything starts making sense and click into place. So there's a similar thing going from a to b now with effect as well. But given I've rewired my brain, like, 4 years ago, I can't imagine going back, and I'm, like, more productive than ever.

Yeah. It does feel like I mean, a lot of people have been calling for more Rust like, whether that's, like, match for error resulting in errors or things like that, features within JavaScript itself. And it seems like that's probably a long shot in terms of something that you probably don't wanna wait for if that were to even happen. But you could just, in the meantime, or even just instead use something like effect and get these higher language features within your user experience for projects. Yeah. I you know what? I found the docs to be really convincing too in terms of the docs do a really good job of showing you be the before and after kind of. So you can see, alright, this does look clean even if I am using some things that maybe feel a little bit different than what I'm used to. It looks clean. It reads clean, and the fact that it's easier and gives you all these features feels like a big win. What what's like a what's one of the cooler things about effect that often gets overlooked in the conversation around it?

So for for me, it's like all about like, it's all about composition, really. And, like, composition JS also similar to the beginning of our conversations. Like, oh, if someone has never tried out, like, superhuman, what makes it cool? Oh, you have to try it for yourself. Like, some things, but, like, composition, if you went from jQuery and to React, and you could then, like, see, oh, I could use, like, that thing over here can reuse Sanity here or, like, then I move it. I was like, you're like, everything just becomes so much smoother. Like, the groundwork is no longer there. So for effect, it's that where you can just use other people's things.

Like, for example, a there in effect, there's a concept called a schedule.

Like, when you have an error that's happens in your app, what do you wanna do about it? Show a nice error message, but maybe there's some something more actionable you can do. Let's say you you make an API call that fails.

You might want to try again. But how do you try again? Do you just, like, hammer the the the Vercel, like, 10 times in a row and, you might even get rate limited and JS even worse than before? So in effect, there's like a very easy way to retry something. There's like a a schedule. There's a concept of a schedule to retry something.

And that schedule can be defined in itself.

Like, someone else could define like, hey. This is like the exponential back off schedule that works best for GitHub, or here's the one that works best for OpenAI.

And then you can just like with one line of Node, import that and and use it. Wes it's like, if you wanted to implement something like that without effect, it would just, like, litter your code with, like, tons of layers of nesting. And if you wanna then do that concurrently somewhere else, it just it's just crazy. And the other thing next to composition, this was an example of composition, is just that in programming, we deal with so many very heterogeneous use cases.

And so Node use k and, like, different use cases require for, like, different approaches.

So for example, there's Node approach that might require, like, for a very concurrent handling.

Some other use case that requires for Vercel sequential handling or another use case that might where streaming is the best approach.

And for each of those, we need to go to very very different corners of our Npm world. Like, if you wanna go streaming, it's like, oh, like, now your Wes maximalist and, like, everything becomes a stream. But some things are a poor fit for a stream. And then, like, you it's this there's no reconciliation so far for, like, that very heterogeneous aspect of programming.

And in fact, everything, it feels like very homogeneous and everything. Like, if you've learned to use, like, 1 module, you already know how the others work. And this is, I think, one of the biggest blessings in the JavaScript community where we have such a wild west of different package flavors, different APIs. You learn one thing, and you don't know how to use the other one. In effect, everything fits like a glove when you once you've learned once.

Has has anyone dipped into building a web server with this? The first thing I saw when I looked at all this JS, oh, that would be nice for TypeScript middlewares, you know, where, like, if a middleware modifies something, the function down the road, Hano does a decent job at this, but it's sometimes hard to type it when you're not directly inside of it. Has anyone done that?

Yeah. So you're you're pointing at a really interesting question that also I kinda, like, came to this conclusion early on in my effect discoveries where I realized, okay. So this is this better way to build something, but hold on. Does it now mean we need to, like, wrap all the existing things? The effects Sanity would say, Wes. But even better, we're gonna rebuild everything. And that's, like, super audacious in one way, but it's crazy how to see how far we've already come. And so by now, like, there's almost like a effect native equivalent to most of the popular things in Npm today that's already part of the effect ecosystem.

So for example, there's like Zod. Effect has a effect native equivalent called effect schema or TRPC.

Effect has a thing called effect RPC.

Effect RPC is also the thing that would enable your use case. When you want to build a web server, you typically want to build, like, some sort of API ish thing. With effector PC, you can get that. Like, you can get, like, a open API server out of the box. There will also be, like, a GraphQL thing, etcetera.

It's almost like in Rust, you have such a vast standard library. You don't really need to reach for, like, some third party thing. It's just part of the language. Mhmm.

Just part of the standard library. And think about this the same way with an effect that it it's just part of your ecosystem already.

I see. Obviously, this is today in 2024.

We have not yet covered as much surface as we might have, like, in 2 years from now. But it's so surprising how far we've already gotten that, like, I have just in terms of the stuff that's in my package, Jason, that is not part of the effect namespace JS, like, shrinking.

Maybe I don't know. Maybe I have, like, 5 more things that are not part of the effect ecosystem.

And, they're they're, like, shrinking.

Wow. Do you think this will ever get bluebirded? Meaning, like like, Bluebird was the big promise library that everybody used before we got promises, and and now we have them in the language. And Node we see things like signals.

There's been a PIPE proposal out there for a while. Do you ever see the language evolving to a point where this is something that's built in?

So, I I love that comparison.

And in one way, I would say, boy, I would hope so, since I think it it would remove a little bit of the debate of, like, hey. What is this? Since, like, no one really questions too much of, like, a JavaScript feature. Oh, it's like part of the language. Mhmm. Guess I'll learn it. Guess I'll embrace it. And with f effect being a bit of like this weird new thing, now it's like, to to really rewire your brain takes a little bit. Yeah.

But on the other hand, I would say I would say there's probably 30% of effect that step by step will find its way into JavaScript.

That, effect can then embrace or say like, hey, this part we no longer need. But I think some other parts are just that's just the way how JavaScript is right now.

Deal with it. So for example, just look at the, TypeScript issues in regards to explorations to typed errors. When you like, the way how you do errors in TypeScript is through throws.

And there's been many many explorations of like, oh, like, what could be similar to Java? Could we have, like, checked exceptions? And we say like, oh, the signature also could show this and this and this.

But TypeScript being not really the language, JavaScript is the language. You can do anything in JavaScript.

So with effect, you have the benefit of saying like, hey, this stuff is in effect. So we can give you much better, guarantees for it. And Yeah. So this level of opinion, the effect ecosystem can afford.

But with JavaScript being the thing that we're stuck with and not TypeScript. It's not the TypeScript runtime. It's JavaScript runtime.

Yeah. So I think some things just can't really be be fixed.

Another example is like promises.

Promises is like a very interesting domain just to to better study.

In a way, the fastest code the fastest asynchronous code that you can write in JavaScript is still callbacks. JavaScript is sort of syntactic sugar that come with its own performance stong implications.

Whereas, for example, if you do an accidental await somewhere, the code that will run afterwards will come in the next tick. So this can become a performance issue that's very easy to fall into this trap. The other thing about promises is that they're eager by default.

So if you have a a promise, that ship has sailed. You can't say like, oh, the thing that was originally this promise, let's say an API call. Oh, do it again. Well, you can't tell a promise like run again. There's just like already a handle to the value. And you can't really you you don't know what has happened here.

And so this is fundamental.

Does the semantics of promises are defined by Node? Very different to what other languages have arrived at. And I think there there's been this interesting anecdote recently that one of the very instrumental examples that Wes, designed effect after JS a Scott project called Zio. And the author of, of Zio recently gave a talk at the Effect Conference Wes he said, like, actually, Effect's gonna have a much wider impact than Zio, had for the Scala community.

And the reason why it's here to stay is because JavaScript has made its design decision around promises.

JavaScript is here to say promises are cursed, and this is a better approach.

Interesting take. I tend to agree.

But but the future will show.

So now's the part of the show where we talk about sick picks and shameless plugs.

Johannes, did you come prepared with a sick pick today?

Yeah. So I've been thinking what JS, like, a a more broadly applicable sick pick that that works for or can provide some inspiration for for the wider developer audience that's also a bit in line with what we talked about today. And one encouragement that it would make for most app devs is think about observability more. Like, you probably the way how you debug is probably just like very console log driven.

Mhmm. But there's been actually some very serious innovations and improvements been make, that's, like, all summarized under the term of of observability.

And so Sanity is obviously a big leader in the space.

Yeah. But but I think there's like a there's just like some some overall ideas that are agnostic to to various vendors.

And I think a technology standard that that's, stands out there is open telemetry, which I think also works with Sentry. And this is partially also, like, a a thing that just works out of the box with effect, but also works, like, in other like, pretty much in all language stacks, all technology stacks. OpenTelemetry is already supported out of the box or is on the roadmap.

And I think this is a thing that's very normal in other program communities, other language communities, but not as common yet in the web community.

So I would just encourage, like, app developers generally to take a look at OpenTelemetry, get an understanding what it means to instrument your application, and then see whether it provides value to you over your typical console log driven development.

For me, it's really been, like, a such a game changer to better understand what is happening in my app that it really changes the way how I develop.

Wow. We need to have someone from Senturion to talk open telemetry. I'm sure there's a lot of smart brains around the Sentria office on that one. So, yeah, gotta get into that more for sure. Sick. What about shameless plugs? What would you like to plug?

So here is only one valid choice for me right now, which is, the Local First of Am podcast, which I've started recently.

Since there hasn't really been a good home for, like, everything local first where we can go deeper on a specific rabbit hole.

And so I started that that podcast a while ago and, since had a couple of really interesting, conversations there. For example, also had one of the innovators of the local first space there, James Long, but also had other really prominent persons there such as, like, Martin Klepman, Peter van Hardenberg, Rasmus Andersson.

So, yeah.

Lovely conversation so far. Going a bit deeper, it's going on down some rabbit holes there. So maybe not for everyone, but if someone, found the the local first stuff intriguing, this is a way to go deeper.

I gotta say, since you dropped the 1st episode, which is really where I even heard the term Local First for the first time, This has been the only development podcast that I I subscribe and listen to. So, it it's really hard for me to subscribe to dev podcast considering we do Wes 3 days a week here. And I just wanna say, big ups to that because it's fantastic, and I I've really learned a ton from this podcast. There's a lot of really great future tech stuff, so definitely give it a a subscribe.

Thank you. That means a lot coming from Veeam.

Yeah. Thank you. Yeah. We'll Wes good. Thank you so much for coming on. I appreciate all your time on this. It's really interesting to dip into it all, and we'll catch you later.

Thank you so much.