Creator of Swift, Tesla Autopilot & Tensorflow. New AI language Mojo with Chris Lattner

I sure hope you're hungry.

Oh, I'm starving.

Wash those hands, pull up a chair, and secure that feed bag, because it's time to listen to Scott Tolinski and Wes Bos attempt to use human language to converse with and pick the brains of other developers. I thought there was gonna be food, so buckle up and grab that old handle because this ride is going to get wild.

This is the Syntax supper club.

Welcome to Syntax, the podcast with the tastiest web development treats out there. Today, we have a really interesting one on, Today, we have Chris Lattner of a modular and of a whole bunch of stuff in the past. We'll talk about that in just a second to talk to us today about, A new Python programming language called Mojo, as well as a whole bunch of machine learning AI stuff behind that and pretty excited to have him on. So Chris is man, probably one of the most well decorated guests we've had on the podcast.

Among many things, inventor of the Swift programming language was senior director of TensorFlow at Google VP Autopilot Tesla.

All kinds of really cool stuff in your background.

Thank you so much for coming on, Chris. How are you doing? Awesome. Well, it's really exciting to be here. It's great great to chat with you and Talk with a bit of a different audience than I'm used to talking to. Yeah. Yeah. That's that's kind of why I I reached out to have you on the podcast because We talk a lot about on the podcast about, like like, normal developers. You know, like, me and Scott, we're just regular developers. We're not writing compilers. We're not Fussing about how long strings are gonna be and things like that. We're we're mostly JavaScript developers listening to this podcast. And When I saw the stuff around Mojo come out, it seemed like a a big part of it was You're going to take a lot of the hard harder lower level, and you can correct me if I'm wrong, harder stuff around machine learning and AI and make it easier and accessible to

the rest of us. Is that right? Yeah. Yeah. Actually, that's that's a good way of explaining it. You could say I'm abnormal that way. Right? But but a lot of A lot of what I've been doing for my career has been working at this hardware software boundary. Right? And, working on compilers and developer tools and programming languages and stuff like this.

You know, if if done well, I mean, there's you could do anything both for good or for evil, but I prefer to do things for good.

If done well, what you're doing is you're making It possible to achieve new things in a way that you don't have to micromanage all the details. So if you look at a high power language like JavaScript, right, You can get a tremendous amount done. You can compose onto a whole bunch of different libraries, and the ecosystem is so wonderful. All the work in the community, All this stuff is made possible by the fact that JavaScript's a really powerful high level language.

But then underneath the covers, there's a bunch of technology that makes it go fast.

Yeah. And so one of the exciting things about what's happening in the world of AI, but also in graphics and many other things, is you're getting new kinds of hardware.

And so it's not just about CPUs anymore. And so what that change in hardware is driving is that changing a need for higher level programming language changes.

And so in JavaScript, for example, it's hard to get you know, JavaScript's amazingly fast as you know. Right? And and and the the JIT compilers and the ecosystem has, like, a massively cool technology stack underneath it.

But running JavaScript on GPU is not really a thing. Yeah. Part of my quest is to help unlock this new generation of hardware And make it as easy to use as Python or JavaScript or other friendly languages that lots of people can learn and use. Beautiful. And the the idea is that you have this

Super powerful hardware that's coming from like NVIDIA and companies like that that is purpose built for

Training models? Is that the the what this hardware is? Yeah. Well, so the the actual the history on all this stuff is pretty interesting if you're a nerd like me. Yeah. Because what ended up happening is that NVIDIA really came out of the graphics world.

Right? And so their their origins were back building, gaming GPU kinds of cards to accelerate graphics. And the thing that makes graphics go fast is that you're pushing millions of pixels onto the screen, And each of those pixels get processed independently.

And so what this led to is a very parallel architecture where the hardware is able to do many pixels at a time.

And so that parallelism is a very important part of graphics initially, But then they stepped into physics, and they went took a weird left turn through crypto and did some other things. And They're kind of on this hunt for, anything that had that similar parallel structure where you could do a lot of work in parallel at the same time To build and solve for a bigger problem. And so AI, turns out, is actually quite parallel. There's a lot of things that happen, and these machine learning models have A tremendous amount of compute, and NVIDIA has been driving a lot of this invention for many years. But these days, it's not just NVIDIA. There's lots of other hardware out there, new kinds of chips special built for AI, special built for, networking, and all kinds of other things that are going on, media. So for for example, like, things like YouTube have chips that just transcode video from one format to another. And so Oh, wow. What's happening is that there's this explosion of new kinds of hardware, and, there's no really good way to program any of that. And so what we're trying to do is we're trying to say, okay. Well, Let's take Python, which is a very well loved, widely used programming language. Maybe not as cool as JavaScript, Then, you know, we we could have that debate perhaps. But Yeah. Let let's take this and lift it to the next level so that it can now run and scale across lots of different new kinds of hardware. Nice. I it's really interesting the,

background on the GPU processor thing because I I've always understood that, you know, Especially coming from a background of building PCs or any of that, you get your GPU to to have better graphics. Right? And then now we have this, I think my first experience personally with seeing this mass adoption of GPUs for other things was, in fact, when everybody was buying them up for crypto rigs, for mining rigs and stuff. So I thought it was a really interesting explanation about it being for parallelism.

So what exactly in machine learning

is a good fit for parallel processing there. Yeah. So so one of the interesting things about these machine learning systems.

And you hear about these deep deep neural networks, these LLMs, these things, and you find out they have billions of parameters. It seems like everybody's bragging about how many 1,000,000,000 parameters they have these days. And is it, you know, 3,000,000,000, or is it a 100,000,000,000, or is it could be a trillion someday and, like, All these things. And so what that means is that each of those parameters is a number.

And then when you're sending in a sentence, what ends up happening is that sentence gets Pass through billions of parameters through this complicated stage of processing.

There's different words, but layers are one way that you see this. So there's multiple layers of operations that get get applied to this. Each of these layers then has lots of math inside of it.

And these math operations get done to these big matrices. They're called tensors. And these tensors are basically big arrays of numbers.

And most of these operations are all parallel. And so you could say, okay, let's do a matrix multiplication.

We can do all the rows in parallel with each other and things like this. Right? And so because you're dealing with so much number crunching and you're dealing with so much data, it really does lead to a different kind of architecture. And these days, super if if you Dive deep into it. Like, instead of having a PC with graphics card in it, now you get these servers that have 8 GPUs in them. And they're they're more GPU than CPU in some cases. And so And this this world is evolving, and it's becoming more specialized for this one kind of compute.

Now there's other kinds of compute, but AI is getting a lot of attention because it's impacting our lives and our products and things like this. Let's talk a little bit about, like,

compiled languages versus interpreted. I think people listening to this podcast are Maybe interested in the difference between that. We get we get the question all the time, like, what's the difference between a programming language and a scripting language? And In the JavaScript community, we're starting to see Rust sort of eek its way in. And we're a lot of our tooling is is being rewritten in Rust because it's faster and it's It's more low level. So, like, can you explain to our audience what the difference between, like, a compiled language and an interpreted language

Yeah. Well so, great question.

One of the things I'd I'd suggest thinking about is separating the language From the implementation.

And so as you know in JavaScript, there's V eight. There's the WebKit engine. There's, simple interpreters. There's, like, all kinds of stuff All the time coming up in the ecosystem.

Yeah. But the lang language stays constant. I mean, the language evolves, but it it kind of evolves separately and you get your transpilers and, like, all the stuff going on. Right? And so, when you think about whether something's interpreted versus compiled, you're talking about how one implementation is built.

Now if you take a language like MOJO, you take a language like Swift, what you'll end up seeing is you get actually many different options. And so MOJO itself can be Statically compiled. You can get an 8 dot out like you would use Rust to compile a tiny little binary, and we can make, You know, hello world into a 40 kilobyte binary kind of a thing. Right? Yeah. That that that's cool. We then have JIT compilers.

And so you can type Mojo on the command line, and you get a little redeville print loop. And so you get an interpreter experience. And so it's actually It's working like an interpreter, but it's using a compiler at runtime.

This is how v eight or something like that works where you're using You're using compilers in the loop to give you high performance.

It also and something like v eight will also use phases of interpretation.

And so these are all different ways of implementing language, and they have different trade offs in terms of, startup time, like how much time you wanna spend upfront versus how much time how much Performance you get out the other side. And so these trade offs can, impact the implementation, but, hopefully, they're invisible to the user. Right? And so if Yeah. So one one of the thing I've been working on compilers for a long time. One of the things that's that I've learned is that most people don't wanna know about the compiler.

If you don't have the compiler, you're doing something Yeah. Right. Yeah. Right? Either Yeah. Yeah. Yeah. Okay. Your code was wrong, and so the compiler's complaining about it, or the compiler has bug or the compiler's really slow.

These are reasons people find out about the compiler. But, usually, you don't wanna know about it. And so a good compiler will generally try to stay out of the way. That's beautiful. So, Wes, do you want to get into to Mojo or did you have a follow-up there? Yeah. Well, I just had a little

hilarious thing that happened to me a couple of months ago on Twitter is Like Node. Js, has the ability to, like, compile, I'm using air quotes here, it into a single executable runtime. And And I I did a hello world and posted it up, and it's it's 42 megs. Right? Because you have to give the entire JavaScript interpreter along with it, and people crucified me for using the word compiled, which is like like, I get it. And and that's why it's so large is that, like, JavaScript can't you can't just take JavaScript and Throw it on a chip. Well, well, there there is some projects around that as well. But Well, it's also funny. I mean, you're talking about people shaking their fist at you on the Internet. Right? Yeah.

That that that word transpile is also super polarizing where people will be, you know, oh my god. It's not a transpiler. A compiler and this and that and the other thing, and transpiler's a made up word. Who cares? Yeah. Who cares? Who cares? If you know what it means, it's descriptive. It sends the right connotation. Like, you know, the the English language is not a static thing. Like, let's go mutate this thing and make it more useful. Yeah. In that regard,

is CSS and HTML a programming language? Also one of those thing. Who cares? Right? Who cares? So many better things. So Mojo is

what you've created.

Is Mojo a programming language? Is it Python? Where does that live? Yeah. Great question. So Mojo is a brand new programming language, but it's built in the Python family.

And so if if you jump one one Friendly community over. You get to the Python world. Python has existed for 30 years. It's gone through its own transitions from Python 2 to Python 3. That was A fairly fraught transition. Right? But I'm still battling that yet. Yep. But it also has other friends in its own ecosystem. It has Jython. It has a number of other things in the ecosystem as well for different communities. And so Mojo is a friend and new participant in this community.

And so we're building out Mojo to solve problems that Python has never been super great at before, but we want it to be compatible with the ecosystem because we don't want to do another Python 2 to Python 3 kinda transition. And so Mojo is still young, but its major focus is On fixing all the problems that force you into c. And so if you look at the way people typically use Python these days, Python's amazing as the glue layer, as the integration layer, as the scripting and the hacking layer, and it's really good at the dynamic programming and just kind of playing around with things. But if you build a big Python library, often, you'll get to the point where you want to care about performance.

And right now, really, the only I mean, there's a whole bunch of different things in the ecosystem, but the the default thing that people reach for is go write the thing in c or c plus plus.

Yeah. And And then use the CPython APIs to bridge into Python and build this this, like, integration layer. And so many libraries in the Python ecosystem are these hybrid c, c plus plus, Python things, and they're incredibly complicated. You don't have a debugger that works across both boundaries.

Moving some class from one side to the other is just a huge pain. You have to know c and c plus plus. And Yeah. You know, a lot of the point of Python was you didn't have to know Right? And so if you have to straddle these worlds, you end up kind of in this worst of both worlds kind of situation.

What Mojo is doing is is Mojo Is it saying, okay. Well, let's keep everything that's good about Python. Let's keep the high level abstractions, the classes, the dynamic stuff. But then let's also add the ability to drop down to To get better performance by adding types, for example, and adding other lower level concepts like including pointers and things like this. So you can reach down to the right, you know, c like constructs when you need them without having to switch to a different language.

And we think that this is a good approach because this means that Python programmers can start out and keep everything they know. Because if you've been working in the Python community, you've learned so many important things. You One of the libraries, the tools, the ecosystem, syntax, the you know, how the underbar underbar ad operator works and all all this stuff. Right? And so why throw that away? What Mojo is allowing you to do is take everything you already know and then grow instead of having to reset into a different world to learn a new ecosystem. And And so we think this is pretty exciting and interesting for folks. Yeah. So in in that regard, are like, is general Python, like, libraries, Are they compatible with Mojo? Yeah. That's one of the major focuses is you can import an arbitrary Python library and just use it. And so that's that's huge and really, really important. We don't wanna fragment Unity. Oh, that's beautiful.

And

why did you choose Python and not our beloved JavaScript or or TypeScript for this type of thing? Oh, what a fair question. Yeah. Well, so the the history on this is that Modular, which is the company that I work for, is, really focused on building AI stuff And, specifically, AI infrastructure so that people can use AI in their products. So we're not building a large language model. We're not building a chatbot. We're We're helping other people do that. K? And so as as a consequence of that, we have to build a whole bunch of this fancy compiler stuff. We have to talk to all the chips And these accelerators and GPUs and all all that lower level technology that, again, most people don't wanna know about.

Wanna worry about the the machine learning stuff. They don't wanna know about the GPUs. And, as a consequence of that, we had to build this really cool technology platform. And we got to the point where we said, okay. Well, we need syntax for this. Yep. And as you mentioned earlier, I've built the Swift programming language from scratch. Right? That was a Hobby project of mine that kind of got a life of its own and kind of took off. And so building new language is very familiar to me. I've been there, done that, have built actually 4 widely used languages.

But, but the, but the Python community is or the the machine learning community is all about Python. And so it's very natural for us to say, okay. Well, since we're building AI tech, go to where all the AI people are, and that's really heavily dominated by Python. That's a good lead into a question I had about, You know, why is the AI community so Python focused? Yeah. That's a that's a good question. I I'm maybe not the best person to answer that.

My understanding is that it's a combination of luck and a combination of skill as often is the case. Right? And so it's lucky in that, some of the early systems like TensorFlow bet on pie Python.

And that was kind of because Google built TensorFlow. They were a c plus plus shop. They were all also a Python shop. And so, you know, I don't know exactly how that decision was made, but, that that I think had a big piece of it.

But Python also has an amazing data science ecosystem with Pandas and NumPy and, like, all this stuff for a very, very, very long time. And so I think that also really contributed.

But, also, I mean, Python is super easy to learn. And so I think that's a that's a huge feature, and I you know, I think that's a big piece that can't be Neglected.

Also, I think that Python's strength as a glue language is also, really important here. And I think that while it's super amazingly painful to glue c Plus plus and c code into Python.

That feature is really quite critical because a lot of these AI systems really depend on that, and that's been somewhat more challenging on the the JavaScript side of things, or it was for some time at least. Yeah. As as I've been doing A lot of AI stuff over the last year,

I found myself being like, wow, this is all Python. I find myself trying to translate things over.

And it's It's I think a lot of it comes from, like, the academic background as well. Like Python is massive in academia, massive in data science.

I used to have a book, Sublime Text on Sublime Text, and that was huge. That book was huge in the, like, data science, Graphing, world. And it's funny now as all the JavaScript and web developers are getting into AI, we're we're starting to be like, oh, wow. Like, Where's the JavaScript libs for this type of stuff? No?

Yeah. Well, in in the JavaScript community, it has tons of really cool AI tech that's being built into it.

Where a lot of the mind share is. And so, you know, our goal is to meet people where they are instead of forcing them to switch over.

Again, I built SWIFT from scratch. Right. I mean, I I'm pretty heavily invested in this thing. Right? So it would have been very natural to say, let's go make this a swift flavored problem, but,

doing that would not have been user centric. Do do you think it's something that the average web developer should be investing time into, getting into Python and Python based Communities, people building web applications. Is that like a is that an area that we shall Spend a little bit more time. And if if so far we're focused on PHP and JavaScript Also, a great question.

You're asking effectively for career advice.

Yes.

Possibly. Yes. So so so let me answer this in 2 different ways. Right? So one way is, what is the best tool for the job? And so if you wanna go build an Ios app, then learning Swift is a good thing. Right? That that that that is a good thing to do. And so learning Swift is, you know, a a way to solve a problem, which is you wanna build an Ios app. If you wanna go build machine learning tech into your products, you wanna go train a model, Learning Python, absolutely, you should totally do that. Trying to keep that working in JavaScript can be done, but it's gonna be an uphill battle. You're gonna Fighting a lot of fights, so you don't wanna fight. And it will distract you from the mission. Similarly, if you wanna go write a bootloader or something, then write again in c or c plus plus is Or rust or something else is probably a good way to go. Right? So my number one answer is use the right tool for the job. And Unless you want to go build the system itself, stay with the herd because I think that's generally a good way to go.

But the other way to interpret your question is career advice. Right? And so, I can't give really good generalized advice. I don't know. You know? You have so many listeners.

AI is really important, but it's also not the only thing. I think that one of the things that AI people get ahead of themselves with, and there's a lot of hype in the AI space about it reinvents all software and all these things. Right? It turns out that building an app, building a product is way more than writing some code. It's right way more than a neural net. Have to understand the product and user interface and the like, some your customers, you have to understand so much more of what you're building. And a lot of that is very technology agnostic.

And so, again, if you are in a space where AI can be a big powerful part of the product you're trying to build, Then, yeah, absolutely go figure this stuff out. But if it doesn't help solve the problem that you have, then, you know, you could look at it as know, it's like investing in any technology. Right? And so, you can look at it from multiple different ways. And the idea with this MOJO is, Like, where is it gonna run? Is it is it running on big data centers? Is it running on a feature phone

on a can it be put on a chip that I could just pop onto a breadboard and And start to work with that? Right. So one one of the cool things about,

Mojo is it's giving Python superpowers.

Yeah. So first answer is runs everywhere Python does. And so Python can be run everywhere. You can run it on an embedded device and things like this, but it's really most happy on a Linux box or on Something PC or server esque, I would say.

Now the reason for that is that it has this massive library ecosystem.

It has all these Kind of loosely coupled packages. You've talked about the challenges that Python has with its packaging in your previous episodes.

All that I think is is true.

It also depends on all the c and c plus plus code in practice in all of its libraries. And so porting and moving around this massive set of libraries is somewhat challenging.

But Python runs almost everywhere. Right? Now the places it doesn't run is it doesn't run on a GPU.

It doesn't run on a microcontroller, really. Right? And so one of the things that Mojo does is it allows you to scale down to smaller systems and scale out of all these dependencies.

And so Mojo can run for example, if you think about a graphics card, it's effectively a different computer It does not look like a PC. A graphics card, one way to think about it is it's like a, embedded system. Mhmm. Right? And so the software stack has to be extremely cut down. So you're using exactly what needs to run on it. Just like if you were running on a 16 bit microcontroller or something like that, you'd run exactly the code that you wanna run on that chip and nothing more. Right. And so Mojo can scale all the way down to that kind of a system, and that's super important for all the AI stuff. But it turns out it's also really useful if you're building an Internet And nobody's built a, Django and Mojo yet.

But if if and and part of that is because you can just import Django and use all the Python stuff as is. Right? Yeah. But if you decide to move something into Mojo and take advantage of of its features, you can get better performance, you can get lower dependence on playability, and all these other features incrementally.

And so we think that's gonna be super exciting as Mojo matures and the community builds out to see the stuff. Oh, so you should be able to just, like,

Take your existing Python app that's costing you $10 a month to run, and eventually you'll be able to just

Mojo run it, and it you could cut your server bill down? Yeah. Absolutely. Well so that's one of the reasons that people are super excited about Mojo is that, How should I say this without being mean about it? JavaScript has way better general implementations people use than Python does.

So Python is not a fast language. It doesn't really or threads well. It it it has certain ecosystem challenges that I think are pretty well known. And so, Mojo being a first principles from scratch Compiler runtime language, full stack implementation.

As you say, this is hard mode. This is not the easy thing. It means, in some cases, we're not just 10 or 20 20 x faster than Python. We're, like, a 1000 time or 10,000 times faster than Python.

Wow. And and so that's a big deal.

Now if you're sitting there and you have an Internet server that's just waiting for socket all the time, well, it doesn't matter that much. But if you're at scale, if you're doing some a little bit of number crunch crunching along with the integration stuff, if you're doing math for sure, then it can be a really big difference.

Something a little lighter I want to ask you about is the extension of Emoji files is literally the flame emoji. What's the decision behind that? And did that not cause A whole bunch of pain to implement because we had to stop putting emojis in our podcast titles because it was screwing up some car infotainment systems.

That's pretty funny.

So, take a step back. So Mojo, that's brand new programming language, so you have to figure out what goes after the dot.

Yeah.

Right? And so we could've done anything.

We could've picked dotm, for example, but that turns out it's taken by objective c and Maybe MATLAB and many, many, many other things. And so you say, okay. Well, we could go out to dotmojo.

So we did. So you can use dotmojo after all your file names, and that just Totally works. And so, you know, that that's the obvious answer.

But then you start to say, wait wait a second. We're in the modern world. Emoji is a thing.

Unicode is a thing. Yeah. English is not the only language. Right? The the the world should be prepared for this. And if it's not, then maybe we can nudge it just a little bit. And so What we allow people to use is the fire fire emoji symbol after the dot, and that is the proper, you know, Mojo file extension.

And, you know, we're not gonna be too pushy about it. You can use dotmojojo if you want. That's totally fine. But if you're progressive and you want a very pretty thing that takes less Screen real estate, when you're looking at it, then the emoji is pretty awesome. Do you have any sense of, like,

community pushback From from using an emoji, I I know people are, like, really oddly sensitive to those types of of things where you've been you know? On the website, in the marketing material, It shows the fire emoji. I can I can just, like, picture the Hacker News comments,

you know, being concerned about that? Asking which which Are you asking, did heads explode, or what percentage of heads exploded? Yes. Yes. Perhaps both. Yes. Yeah. I I I I have seen the same Hacker News you have. Yeah.

Well, Well, but but, see, this is the thing. Right? I mean, just because we're working on something really difficult and hard, and we're trying to move the world forward and move the state of the art of programming languages Compilers and all the hard tech stuff we're doing.

We could do that and have fun. Yeah. Yeah. And also, it shows me I mean, it shows me that you care about the implementation details that you went out and did that because I know putting sometimes putting a dash in file names on windows It's hard to deal with, you know, so putting an emoji in the file extension is something else. So it shows me that, okay, they really do care about developer here? Yeah. Well and so this is this is another thing where,

you know, it's a blessing and a curse that when you start from scratch so I built a personally, I built a whole bunch of projects from scratch, like That LLVM compiler that underlies Rust or, the Swift programming language from scratch or the Clang c plus plus compiler. Like, I built many, many, many things from scratch. Right? And so the benefit and curse of doing that is you can first principles every decision you make.

And so you have to decide where you're gonna spend your energy. What what what do you want to reinvent, rethink, redo versus what do you just wanna fast follow the The standard practice. Right? But but this is the kind of thing where it's like, okay. Well, again, you look at the file suffix.

Well, on a Mac, for example, the default is to not show the suffix in finder.

Right? And so this is kind of a legacy technology almost.

And so you think about it. Okay. If I'm looking at it in, you know, LS in a terminal or something, right, does it need to take up space? If I'm in, I I personally use Versus Code as my editor. And so, you know, it shows all the file extensions.

Turns out it looks beautiful in Versus Code. Right? And why? Well, it's because people need to use, maybe not emojis, but Unicode and their file names and things like this for lots of different reasons. And so this capability has always been there.

But I don't know. People just didn't think about it or nobody's taken that step, but I I suspect that Mojo will not be the last use of an emoji file extension. I think this will be know, you look back 10 years from now, and there'll be a lot of other uses. And then you can say the mojo started this whole trend. There you go. There you go. This this might be a bit of a silly question, but I think A lot of people who are just getting into AI,

wanna know this. How does machine learning work? And you You're probably the best person I have ever talked to that could maybe describe this. Not a silly question at all, and I can take much sillier questions if you'd like.

Okay. Good.

So, yeah, we can we can go there.

The best way to think about it is that, if you reduce it all the way down, what A machine learning model is is it's a function. And just like you would normally write a function in JavaScript, you'd write things with if statements and for loops and things like this. Right? You have lots of behavior.

If you boil all the way down, ML model takes a string or takes an array of numbers, and it returns an array of numbers.

Now they use funny words like tensor and things like this, but all a tensor is is just an array of numbers. K? Mhmm. And so if you've got that so if you got chat gpt, for example, you type in a bunch of text, what it's doing is it's sending in the bytes or the tokens for the text into this model. And then it gets out another array of numbers, and then it prints back out. Now how does it work? Well, this is a really complicated function. Turns out that writing in JavaScript or any other language, a cat detector, would be really hard. Right? That's actually something that, you could spend a lot of time on and not get very far. Right? And so the cool thing about machine learning is it's it's A function where what what are called the weights, these are the implementation details of the function, they're learned instead of being programmed.

And so the way it works in when you're defining a model is that you define what's called the model architecture, which you can think of as, like, the building blocks of how the thing will run. But then you have these parameters that get learned from a lot of examples. And so this is the thing that makes machine learning really cool is that you can say, okay. Well, I don't know how to write for loops To find out, is it a cat versus a dog? But I can produce a lot of images of cats, and I can produce a lot of images of dogs. And then what'll end up happening is The these machine learning systems through fancy tech, will learn the right values for all those magic numbers.

And then it's like you had Written all this super complicated millions of magic numbers that all kinda line up together to figure out if something's a cat or a dog, And you don't have to do it manually. So that's one of the reasons why machine learning is really cool. That that really made sense to me when I got into

embeddings In AI, what I did was I fed it a bunch of tweets. I asked people on Twitter, said, what did you do today? And I got 100 tweets, and I converted those all to embeddings. And the embeddings is just an array of 1024 numbers or something like that.

And then you can use different algorithms to figure out how close those embeddings are to each other.

And it was really Cool. Because, like, the the words didn't necessarily overlap, but because the machine learning model was trained on those things, It knew that those 2 things were were close together. Right? Like, I could say truck and car and red, and it would know truck and car are Close to each other. So obviously, that's a bit of a simplistic example. But as soon as I saw that, oh, this is all just numbers. It just knows how to convert.

A cat picture to numbers somehow.

Well, in in I mean, the other thing that's really cool about embeddings is that what what the model is doing is it's Turning it's building a representation of the knowledge that is built and embedding that into this, hyperdimensional vector.

And so if you really want to twist your noodle, take some of those vectors. And people have shown where you can say, okay, take the vector that represents, king.

Right? Now take the embedding that represents queen Mhmm. And subtract these 2 things.

And what and what do you get? You get man or woman. Okey.

K.

And so now and so now what you're doing is you're capturing not just the thing. You're not just it's not like assigning index In a SQL database, an ID in a SQL database or something for a concept, it's about building a hyperdimensional representation, Which can then be manipulated.

And this is how the models can make these leaps in judgment about how this stuff works because it's not just Classification into buckets.

It's about figuring out the fact that this is actually a many dimensional space. And, You know, the the you know, a king is both high on royalty and high on male.

Right? And so if you have a queen, it's high on royalty and high on female. Right? And so now you can manipulate these relationships. And that's how that, like, mid journey Wow. Probably works as well then. It can take that image into many dimensions. Right. And so, again, there's many, many different layers of tech that get built into this. And so there's you probably don't wanna go deep into units and how how how all this stuff works. But But the this this is again, one of the cool things about this is you're combining, pure math and differential equations and the chain rule and all these different Apologies that I barely remember from my school days Mhmm. With, a lot of high performance computing, with really cool Product features with, infrastructure and how do you get the stuff to go fast, with programming languages, with evolving APIs, with Communities. And so this is why the AI space gets so interesting is that there's not just 1 piece to this puzzle. It's it it all dovetails and fits together in a very nice way. Wow. Beautiful. And you're you're also developing

an engine to train your own models as well. Right? Like, I know if, Like, you're developing the language that you can use, and somebody could take

TensorFlow or something like that and run it on Mojo. But you're also working on an engine to Yeah. Yeah. So AI if if you if you take a step back, what Modular is doing and what the reason we created Mojo in the 1st place is that we need to solve problems for AI people. Right. So all the stuff we just talked about, so many different use cases exist for AI, so many different ways to build AI into your products.

But the challenge with this is that it's actually really hard to do that. And the reason is is that AI and these technologies have evolved very rapidly over the last, Say 8 ish years. I mean, AI has been around for decades, but it really took off kind of the 2015 time frame. And as a consequence of that, a lot of stuff got built. And all this stuff ended up getting very messy. The research evolved. What AI is evolved.

8 years ago, LLMs did not exist. Right? Things like this. Gen AI was not a term.

And so what happened is that lots of weird stuff got built, and it doesn't really work well together. And so what modulars is doing is we're going All the way back to the drawing board and saying, let's first principles all this technology. And so that's why we have to build a programming language. Turns out that's This fragmentation between c, c plus plus, Python, CUDA, SYCL, like, all these different weird things. Some of them run on GPUs. Some of them run on CPUs. Some of them User centric, this complexity is really slowing everything down.

The same thing is true when you talk about TensorFlow and PyTorch.

Like, you know, TensorFlow and PyTorch are built by 2 different mega corps, Google and Meta.

They're all lovely people. They're all super smart, but they're kinda like the dogs and cats. They don't get along. So I love dogs. I love cats. I love all animals. And so what we're trying to do is we're trying to say, Let's let's actually build this platform that all this stuff can can actually work together because at the end of the day, people building AI into their products, they don't wanna have to pick. They They wanna use the right technology for their job, and they wanna be able to move as their product moves into new kinds of use cases.

And when they do that, they don't wanna have to switch to different technology that they then have to relearn has different tools and its its own fragmented mess. And so if you look at what modular is doing with Mojo, we're meeting Python developers where they are So they don't have to retrain in order to use our stuff.

What we're doing with the AI engine is exactly the same for TensorFlow and PyTorch developers.

And in the case of Mojo, we're giving Python programmers superpowers so you can have higher performance, you can go on GPUs, all this kind of stuff. In In the case of the AI engine, we're giving machine learning deployment engineers, in particular, superpowers because right now, they have to use Twenty or 30 different tools, these converters and translators, these things that are built by research team are not really production quality. And performance is bad. The hackability, extensibility is problematic, and there's always pain points. And so we're meeting people where they are to help lift This community. Wow. And so the Mojo SDK

is you can download it. You can run it on your machine.

Is there It will this be open source eventually, or is this what the the company is? Yeah. So so Mojo so what Mojo is producing is 2 things. It's building a technology, which is Mojo is building a product, which is the AI engine. Okay. And so we see Mojo as a technology, and It's not currently open source. By the way, Mojo is a very young programming language. We just launched the SDK last week, and so that's the 0.2 release. So that's not 2.0.

That's 0.2.

And so Mojo is still evolving very rapidly, and I think we have over a 150,000 users now, but it's Still very, very young.

And, you know, my North Star is if we're gonna build something that's this big, complicated, expensive, hard tech, Then we better build it the right way. And so what I wanna do is I wanna make sure that we build the world's best thing.

And, you know, I've built Swift, and I built The Clang compiler and other things like that before. And so Yeah. I built many of these things. And so what I wanna do is take the next step forward. And so, I love open source. I've built 20 plus years of open source technology with millions of lines of code on GitHub. You can follow me there if you care.

But building one of these super narrow hard tech things with a committee is not super productive. Now I'm getting what we're doing is we're starting with a very tight knit team that's super focused, and we have very specific design points. And then We'll open source things over time as as it matures.

So, yes, we've we've committed to starting the open source process even before the end of the year, so we're we're committed to doing that. This so this might be a question you're uniquely,

equipped to answer.

What really goes into creating A new programming language like this, where where do you even where do you even start? Do you are you do you think about the syntax when you're first starting, or is it thinking about Guiding principles for this whole thing. Or what is it?

Well, so I cannot give you the right answer, but I can give you Several examples that I've been through. So you have to start with what is your goal. So, many years ago, I helped create the OpenCL programming language. In In OpenCL, the goal was allow a c like programming language to run on GPUs.

And so if you if that's the goal, then you say, let's keep it as similar as we can to c, Adding the specific things you need, in that case, vectors and the ability to reason about x, y, z coordinates and things like that, Add just those features to an existing thing because you're trying to appease the existing community, and you want that to go fast. Right? In the case of Swift, very different goal. Right? So Swift started as a hobby project that was, you know, kind of a a nights and weekends project because I I and the team had just finished building, c plus plus and a full production quality implementation of c plus plus and I was burnt out and ready for something new. And so in at the time, this I started switching 2010.

Objective c was heavily entrenched in the Ios developer community. The iPhone was really big, but objective c had been used for decades in Apple's history going all the way back to the next computer company.

And so The challenge with that is that while objective c made Apple successful and made the iPhone possible, it wasn't widely loved by external developers. Right? And so There, the goal was really build a new syntax, build new language that could be useful for Apple's use cases And could scale in. And so that became, as you say, a syntax design process.

And so you have to decide, okay, cool. Are we gonna use the func Keyword or the f n keyword. Are we gonna use function? Or are we gonna you know, all those kinds of things, but also deeper questions about, like, how the how the language works.

In in the case of Swift, it was highly bound by compatibility with objective c. And so it was really important that you could subclass an objective class and integrate with all the existing APIs and ecosystem and go from there.

In the case of Mojo, again, what is the goal? The goal is to give Python folks superpowers Enable a new kind of fancy hardware stuff. Right? And so what we did was we said, let's start by building that fancy Compiler stuff for all the hardware.

And then we get to syntax, and we say, okay, well, guess what? Python has set all the syntax for us. We don't have to invent what classes look like. We don't have to invent how operator overloading works. We don't have to invent, like, these concepts. What we have to do is we have to faithfully implement them While implementing the other language features that Python usually delegates to c.

And so, you Have been talking recently about rust on your show, for example. And so and so when I go out and I say, okay, let's go design An amazing language to do some things. You go look around into the world, and you say, what what other languages exist? What are the good ideas? Let's not just, invent things because we like novelty, but let's learn from the best ideas out in the world. And so Mojo has, for example, lifetimes. Become Very very inspired by Rust, but have learned a few things. And one of the challenges of with with Rust is it's not always the easiest language to learn Because of certain of the decisions they made. And so what Mojo does and what Swift did before that is say, okay. Let's go look at the good ideas They're out there in the world, and let's reinterpret them and integrate them in a novel way so you can get something that's really beautiful and solves the problem. In the case of Mojo, it's b b Python, but b Python plus plus. Mhmm. So b you know, solve those problems and allow Python to scale into the places it wasn't able to reach before. What do you think about

the future of both being a developer, as well as just jobs in general with this AI stuff Coming forward, is are are jobs gone or is this are we gonna make cooler stuff?

So great question.

I can't read the future, but my pen is The jobs are not gone.

And so I I look at this as, you know, a lot of people jump to, oh, well, AI is making so much progress. Therefore, AGI is happening in 2 years.

Right? Well, if you go back to, I don't know, 2017, People are saying, well, autonomous robotaxis are gonna displace all cab drivers by 2020.

Yeah.

Right? And, a Tesla's gonna drive itself coast to coast by the end of 2017. Right? Well, turns out that technology is way more complicated than people understand, and My bet on AGI is it's quite a bit farther out than than many people think.

Now I think that a lot of the coding assistants out there, so the copilots and things like this, are super awesome.

And, again, what they what they're doing is they're giving programmers superpowers.

They're automating away a lot of the boilerplate and a lot of the other stuff that goes into coding and practice. And so I think that that will give, a massive leg up, particularly for the monotonous repetitive tasks that, you know, go into coding.

But again, like I said before, you don't get out of having to design a product, having to understand your customer, having to having to manage the UI and the design and all the other things go into it. It's funny you say UI as there was a

vzero.dev came out the other day from folks at Vercel, and it's It's literally like a prompt to UI, and it's it's surprisingly good, man. It's and I didn't think that we would get there so quickly on,

Finishing out the UI parts. It's impressive. Yeah. Yeah. So, I mean, a lot of the stuff moves very quickly. And so, again, I look at, You know, how how best do we use human ingenuity and how do like, what what place do we play in the space? I look at, know, these AI copilot and graphic design tools and stuff like this is you know, you can kinda look at it as, like, hiring a new member to your team. Yeah. Right. They'll they don't replace all the existing stuff, but they'll make you go way faster. And

I think that's that's pretty cool. What about, like, in life in general? Are there any industries or areas that you are excited to see AI Push forward or help solve specific problems. Oh, well, that that's pretty cool.

I'll I'll give you the joking answer, then I'll give you the Okay. Good. I was gonna say you can give us a petty answer And, like, a Right. Like, a optimistic for the future answer.

So I was really disappointed that room temperature superconductors are not here. Yeah. Yeah. Right. So we need to get all the AI people on room temperature superconductors because I was, you know, led to believe that this had been solved, and I was all Excited, and then my hopes were dashed. Right? So, so anyways, that that aside, yes, pure sciences and AI do go together. But The thing that I'm really excited about with AI and the reason that I care about it so much is that we, as humans, live in the natural world, and we interact with each other. Mhmm. Right. The natural world is not a binary world. Right? I mean, you're dealing with these modalities that include vision and audio and, like, you know, thing LiDAR sometimes and these sonar, like, these these sensor inputs that are not you know, you can't write a cat detector in With 4 loops, like, it's not a thing. Right? And so what I see AI for and similarly, Wes, like, you're lovely, but you you do not speak binary.

Yeah. Last I noticed, that's maybe not your natural interface. To me, AI is like the best way to interact with other people, with the natural world, interface and understand what we want, And have this computing technology fit into our lives in a way that's both more personal but also more useful.

And so that's why where I get excited about this is that That that world outside the computer is a big place, and it's super important. And so if we can make computers able to help us out there, then I think that we I mean, we're already seeing the ways that that AI can be useful Yeah. And also room temperature superconductors.

Like, come on. You know, beautiful. I want a floating car. Yeah. Yeah. Right.

We have these questions that we call the supper club, which is they're kind of questions that we ask everybody, but I I came up with at least a couple here that I I Head specifically for you, and one of which is based on, like, pure joy alone, not considering The product, what is your favorite language to write in programming? Pure joy. Yeah.

Like, what what makes you feel productive?

Makes you feel happy? Okay. Well, that so diff different things. Alright. So for the kind of code that I like, unfortunately, c plus plus is Still the most productive.

C plus plus is the opposite of joy, so these are very different questions.

Yeah.

I really like the Haskells or the O'Camels or these more esoteric Academic languages because they're super pure and opinionated and true to themselves in a way that I think that More widely used programming languages are typically not. And so I'd have to throw my my hat into maybe the OCaml camp For just pure joy when I'm not focused on building a product and I just want to play around with things. Mhmm. You said you're a v s code user.

What about

Shell and terminal, what are you using? I've gone through a number of things, including back in the day, Ed and Pico and, I mean, on the mail side, like Elm and things like this, these days, I'm a VI or VIM user primarily in the console. And So I don't know if you're talking editors or general

tools. Yeah. Well, editor and, like like, are you using, Like, just the a Bash Shell, ZSH,

Phish, anything like that? I was a Seashall user for a long time, and then, Apple so I've been using Apple platforms since 2005.

They got rid of seashell or or did something really weird to it, and so they forced me to switch not to bash, but to seashell, of all things. Yeah. And so it's like it's like a bash alike. I don't really understand what the heck is happening and have never really had time to to do this, but that that forced me into the bash world. Okay. Awesome.

That's great. Yeah. That's the the default in OSX now is Z Shaw. I was surprised at that.

I got switched on. And and, also, it's funny. I I'm I'm split between I've I had used for years Seashell and turbo Seashell and things like that as my interactive show, but then always wrote all the scripts in bash. So I guess maybe there maybe I'm being told that I was doing it wrong, so I I can handle that.

Do you have any tips for people who perhaps write 1 language and are interested in branching out to multiple languages? Yeah. So I think that's a really good idea.

I think it's a it's a really good thing to get diversity of perspective. I mean, you had some recent shows where you're talking about Rust, and you're talking about Mhmm. The trade offs between cargo and npm and things like this. And so seeing how other smart people do things, I think, is a really good thing to expand your horizons.

The challenge I've always had with that is that if you just go, like, read a book, I mean, for me at least, I can't learn that way. And so what I would recommend doing is Start a project.

Even if it's a toy project, go go set yourself a goal, a reasonable goal, and then go build a thing. And then force yourself to go end to end. Like, if you wanna play with AI, there's all these, endpoints and APIs you can talk to. Go build a thing and actually wire it up end to end. Build something that, you know, doesn't have to be novel, but just build something cool that that you can force yourself to go through all the things. And that will give you the, hey. I did it moment at the end, and it will kind of direct your search through how do I use the tools or language or syntax or whatever. And you can kind of Stub your toes in a productive way versus the purely academic. Let me go read a spec and then see if it makes sense. Yeah. Yeah. We're we're often

Advocating for that type of thing too just because the motivation's different. Right? You're you're motivated by factors of wanting to complete a a project rather than Wanting to just memorize the syntax or any of that. That's exactly right. And and some similarly, if you,

you know, a new JavaScript feature comes out or something like this, often the best way to learn that is Build it into a product. Like, actually experiment with it and and see how it works, see how it doesn't. And sometimes what you find out is the common wisdom is not always a 100% right. And so getting a feel for something, kind of that hard to define sense of, how it works for yourself is quite different than just kind of reading that blog post about it. Do you have any? I know you said you've built lots of projects with Mojo just to Get a feel for building stuff from scratch. Do you have any, like, go to in JavaScript? Everyone always says that to do list, and I hate Coding to do list. So I'm always looking for new ideas of things you could build. Do you have any, like, go to for something when you're learning a new language? I build a lot of command line tools Because they end up not having tons of dependencies. Mhmm. And so much of learning a new language ends up being learning the like, how does strings work? Like, how how does the library work? How does How does the paction ecosystem work? And so, you know, you can throw yourself in so in the Python ecosystem, go build a Django app or something like that. That can be cool, and that that can be really awesome. But you spend a lot of time figuring out The weird Django idioms or or, like, if you're in the Ruby community, the typical advice is go build a Rails app. Right? But Rails is It's not it's not Ruby. It's like using all the super powerful crazy meta features of Ruby. And so it's almost a a weird introduction into it where you're learning the patterns more than you're learning the language. Right? And so I think that picking something where you can focus your energy on the part of the problem that you wanna learn instead of The accidental complexity of the framework that you're building into can be good. Excellent.

All right. So the last thing we have on the podcast is Two things. We have a sick pick, which is something you pick that is sick can be literally anything in your life, a chocolate bar, an app on your phone, An experience you've had, and as well as the second thing is a shameless plug, anything you'd like to plug to the audience. Yeah. So, I mean, how about sick pick exercise?

Like, getting out of your house and doing stuff is actually super awesome.

My my, I, my, my secret hack is that every morning, my dogs look at me and say, Are you gonna take me for a walk? Yeah. And so and so having having a guilt trip loop process that encourages you to actually do the thing that you wanna do is actually super powerful. And so,

regular exercise is a really good way to stay, both focused and healthy and, like, all these things, and I think they're off. It gets overlooked. Yeah. Dogs make that really, really nice too. Our our one of our dogs has developed the habit of when he really wants to go on a walk, which is just about Every afternoon, right, he'll just walk to the back door. He'll sit nicely, and the tail will just start wagging. You're like, man, this is this is way too adorable. Okay. We're going on the walk. Alright. Let's go. It's it's it's too bad animals can't communicate. Right? Yeah. Yeah.

You know exactly what they want. Yeah. Well, Not not far off. I think I'll I'll whip up a quick mojo script to, to do that. Yeah. Right. Yeah. Yeah. So in terms of plugs, like, I'd really love people to check out what we're doing at Modular. You can go to our webpage, modular.com.

You can download Mojo. Play play with it locally.

There's a whole community of people on Discord. I think we have over 20,000 people that are all chatting away and and building projects and doing cool stuff, building LMs and, like, all the things. And so if you're interested in the space, like, that that's a really cool place to dive in, and there's lots of super helpful and, enthusiastic folks that are All learning right now. And so one of the cool things about new technology is that everybody's kind of at the same playing field. There aren't the, You know, the sure surely, you'll get the job ad for the 5 years of Mojo experience required.

Nobody has that. And so What that means is that it's kinda greenfields in terms of building libraries and ecosystem things, and it's it's just such a super exciting time. Well, thank you so much for All of your time and insights, this is really exciting, and I appreciate you coming on. Yeah. It's wonderful. Thanks thanks for having me. Thanks so much, Chris.

Head on over to syntax.fm for a full archive of all of our shows.

And don't forget to subscribe in your podcast player Or drop a review if you like this show.