514: Running Local LLMs in VS Code

James 00:35

It's been a week. I was in Guatemala for a wedding and it was absolutely fantastic. It's good to be home. ah But yeah, Guatemala, absolutely awesome. And the wedding was the craziest wedding i've ever been to. So there was stilt dancers.

Frank 00:48

Oh, that's quite a skill. I didn't even know that was a skill to have, but somewhat envious now.

James 00:55

There were pyrotechnics.

Frank 00:59

See, the stilt dancers I get because like I've been doing housework and I kind of wanted stilts the whole time. So like that's a skill. i Don't really need the pyrotechnics. Don't need to set the plates on fire.

James 01:09

There were really cool like LED blinking lights or sunglasses that you got and like cool foam hats.

Frank 01:10

Yeah. yeah

James 01:14

It was wild.

Frank 01:15

Are you sure you didn't go to a rave?

James 01:16

It was.

Frank 01:17

Are you misremembering what just happened in the last week?

James 01:20

There was a lot of dancing. There was also a bunch of live bands and also like a live DJ set and it was kind of absolutely amazing. and

Frank 01:30

Cool.

James 01:31

a couple that we went or, know, we just became friends with friends with them in the last year. And, uh, yeah, there's like so cool and the wedding was awesome. So it was really, I'm glad we made the trip down there. It was pretty It wasn't too bad of a flight actually, you know, Guatemala actually isn't that far from us.

Frank 01:44

Oh, yeah.

James 01:45

It's not that far.

Frank 01:47

It turns out the Americas tend to stick together. um And it turns out you can just go up and down. i don't know. I should do it more often than myself.

James 01:54

Yeah.

Frank 01:55

But I do love. I got to see more of the Americas myself. You're making me a little jelly.

James 02:01

Yeah, we really want to go um over there. There's like Belize and a few other countries that we want to go.

Frank 02:07

Mm-hmm.

James 02:07

Once you're over there, well, maybe we'll go some other countries. So anyways, if we have any listeners in Guatemala, thanks for being a listener and a great country and excited to come back one day.

Frank 02:13

Mm-hmm.

James 02:17

i do There's lots of volcanoes, lots of hiking volcanoes. We didn't get to do that, but they're active.

Frank 02:22

Right.

James 02:22

they're It's happening.

Frank 02:25

Okay, again with the pyrotechnics, man. I don't i don't need that. i don't need I need the calm water, like an oasis. Just nice yeah nice and calm.

James 02:33

Yeah.

Frank 02:35

Maybe a waterfall.

James 02:36

Yeah.

Frank 02:37

That's as spicy as I get.

James 02:39

Well, it has been a week. um There's a whole beautiful list of things that we're going to about today. ah Is there any Internet controversy that you want to talk about or should we skip over any Internet controversy?

Frank 02:53

Oh, God, have I not been keeping up? I can't even think, like, you know, the controversy has come and go in waves. I think there's a trial going on. I'm trying not to pay too much attention to it, but it's hard not to watch a little a bit of high profile trials with ah some companies that start with an O and end with an AI. It's kind of fun to watch that stuff go.

James 03:13

I have seen those. common

Frank 03:15

Yeah.

James 03:15

That commentary has been interesting. I've been trying to watch like the the dailies of that. Very fascinating. I have missed the first few, but it it does seem that ah there's lots of distillation happening and people are confessing to things and all this stuff, which is fascinating. Yeah, that's of interest.

Frank 03:30

Yeah. But no, I don't think there's any like blue and black, gold and yellow, white dresses happening or anything like that.

James 03:32

That's for sure.

Frank 03:39

I think, yeah, the the controversies aren't too crazy. um The AI world has completely moved on. You know, everything you knew last week is irrelevant. It's all changed.

James 03:50

As always, I mean, that's a new week. There's new skills to be learned. I think someone maybe asked us like what skills we're using. I don't look up, but we'll do a listener bag. Maybe if you leave comments on any of our videos on YouTube, we'll do a listener bag.

Frank 04:00

Yeah. Okay.

James 04:02

We have some good questions and good tweets coming in, I think.

Frank 04:04

Yeah.

James 04:04

ah But someone did ask us a little bit based off our episode that we just did.

Frank 04:05

I think.

James 04:08

were talking about some of the ah pricing changes around AI.

Frank 04:12

Mm-hmm.

James 04:12

And one of the things that I had mentioned um was that one of the kind of you know cool benefits of some of these harnesses is not only the ability to use frontier models through the service that you're paying for, ah that are readily available. So all of them have them, right? Whether you're using Codex or GitHub Copilot or Cloud Code or OpenCode or Cursor or whatever, they all have them.

Frank 04:36

Kimi!

James 04:38

Yeah, and many are the same. Obviously, some are also multi-providers, right? With GitHub Copilot, you get a bunch of different stuff from the different providers that are out there and custom models as well. But one of the really fascinating things seems to be that you can do something called bring your own key when you're using this service.

Frank 04:57

yeah. Yeah.

James 04:58

And bring your own key BYOK as we often call it is the ability to say, i would like to use this other service over here to to route all my queries through. And that may be specifically, maybe an API key with Anthropic or with OpenAI or maybe Microsoft Foundry.

Frank 05:20

Yeah.

James 05:22

And ah you would be paying those providers for the token usage, however their billing is, right? And you would just use GitHub Copilot. You wouldn't use the models from from from Microsoft, right? That are available through GitHub Copal, you use them through these other services. And you might be saying, well, those all three sound like models that I might already have. So why would I do that? Well, the real reason I think is also to use a bunch of other models out there that maybe you don't necessarily access to.

And a very popular one is called Open Router. And Open Router allows you to navigate through and have a whole slew of things. Now, obviously with bring your own key, you know

Frank 05:58

Yeah.

James 06:04

the tuning, I'm very interested in your usage of the next thing I'll talk about, but I'm just gonna i'm gonna go long here and then we'll get into the thing, which is the the the biggest thing.

Frank 06:11

All right. Preach, preach.

James 06:13

preach. um But the system prompts, right, the prompts that are being sent, like the VS Code team, the Copal CLI teams are finely tuning those and the tools and stuff like that, right, as much as they can. They're open, you see them. Obviously, there's going to be a general system prompt for these other providers. They don't know exactly what you're picking necessarily. But then there's this other option, which ah Frank Kruger, I did a video on i'm gonna I'm gonna look it up here.

Let me go through my video history. Let's see Copilot CLI, this, that. I gotta to scroll for a while. I'm scrolling, I'm scrolling, I'm scrolling.

Frank 06:50

Produces so many videos, everyone. who Who keeps up?

James 06:52

um

Frank 06:52

Raise your hands if you keep up with James.

James 06:55

Ah, here it is. One year ago, it was called Bring Your Own Key and Models to Get Up Copilot and Visual Studio Code.

Frank 06:57

Wow.

James 07:03

And I talked about bring your own key, but also, running local models with Ollama or now you could also use Foundry Local.

Frank 07:11

um Okay.

James 07:13

So running local agents and local models on your machine, and instead which you wouldn't have to pay for anything because it's on your machine.

Frank 07:22

Yeah. um I've actually I've gone down every single one of these rabbit holes now with the great pricing apocalypse.

James 07:29

Hmm.

Frank 07:32

ive I just wanted to feel out the world and see like what kind of pricing can I expect out of different services and all that stuff. I did want to make one joke, though, about open router. Open router so bring your own key that within open router, you can bring your own key. So you can bring your own open router key, but with an open router, you can bring your own key in there too. It is such a weird messed up world because open router actually merges all these different providers together. It's it's very funky, to be honest.

um I've been rocking the DeepSeek model right now. Because it is stupidly cheap. They're running this promotion where it's like 75% off the price. And the price is already cheap. And so 75% off that. I was just curious what I was doing. So I did like a full day of work on it. And it cost me like $2. was like...

James 08:23

Wow. Nice.

Frank 08:23

That's a pretty good deal. pretty That was like 60 or 80 million tokens total. i was I was burning through the tokens and they have a good context length and it was just a couple bucks. So I was trying to bring your own key thing. So I still rock everything through vs Code. chat, AI chat, agent chat. i don't know what the feature's called anymore because I used to call it copilot chat. Is it copilot within VS Code? I can't keep all this straight, James, because it's just chat.

James 08:50

It's just chat. Just chat.

Frank 08:52

Just chatting, man.

James 08:53

Just chat.

Frank 08:53

ah

James 08:54

Just chatting.

Frank 08:55

Yeah, because Copilot's GitHub's thing. VS Code is more generic. VS Code has an interesting um implementation, though, where you, in general, you have to get an extension if you want to add models to it. So it's a little bit funny because you were mentioning like generic prompts and such. All the model providers out there have kind of adopted the same API. They all just copied OpenAI's chat completions API.

But oddly enough, VS Code doesn't allow you to just plug into that. They really want you to get extensions and do funky stuff because they assume that every model wants to be tailored. It's all a bit of a joke, but whatever. So, yeah.

James 09:34

Are you sure, Frank? Because you can go into model, manage, manage, manage model, model, manage, manage, man manage your models.

Frank 09:38

Model manager, head model, amp. and Yeah, manage your model models. so

James 09:46

Now, that being said, I will say that extensions can bring their own models into the mix as well.

Frank 09:51

Yes, yeah. So hit add model and there's going to be a fixed list.

James 09:52

So you go in. you have your you have your language models and it has all of them in there. So you have your your opuses, your sonnets, your GPTs, but there is a add models, which has a dropdown for Anthropic, XAI, Google, Open Router, OpenAI, i OLAMA, OpenAI compatible and Azure.

Frank 10:03

Yeah. Yeah. Yeah. Sure. Yep. oh Oh, maybe that's an insider one. I'll have to check out the OpenAI compatible one. I hadn't noticed that. So that's good. Thank you. Thank you, VS Code people, for doing that because yeah.

James 10:28

So my assumption is if you install an extension from a provider provider like Cerebris or something like that, then they would just basically pre-configure with that.

Frank 10:39

Yeah, and honestly it was getting a little bit annoying because we're we're to the point where there are so many AI shops out there offering models as services too. They're they're providing the full everything um and they all have adopted the open a API. So um yeah, I guess like they're they're all still a little bit weird around how they do thinking and stuff like that. But in general, they all kind of provide the same API and you can just plug into them. So yeah, I've been trying out all the models, man. Like I have put $10 credits into like every person out there.

James 11:15

Hmm.

Frank 11:15

i probably should be doing everything through Open Router, but I wanted to get a feel for all the different providers and see how they work. um And I've been pretty pleased. I got to be honest. um I think I'm still falling back to like the codex and the sonnets of the world. um But I've really been enjoying just taking taking a break from the norm and going out and trying the more wild side of everything. But BYOQ has been fun for me.

James 11:44

Okay, so when you go in and you add a ah model, let's say you're adding this, don't know, how'd you do DeepSeek? Was that an extension or something?

Frank 11:52

Yeah, so DeepSeek has their own extension, um but it's just it's a plugin for the chat window, basically.

James 12:00

Oh, that's what I was going to ask.

Frank 12:00

That's,

James 12:02

So it shows up and it just shows up in the model selection.

Frank 12:06

Yeah, it it shows up ah it shows up too many models in the model selection, is to be thoroughly honest, because DeepSeq decided be to be a proper provider and offer many kind of models, not just the DeepSeq models.

James 12:20

Oh.

Frank 12:20

So honestly, ah the UI needs a bit of work because DeepSeq just threw in like a billion models into the list. And you can just scroll and scroll and scroll and scroll to find all the stuff that you want. But yeah, that's the general idea.

James 12:31

Interesting.

Frank 12:32

These extensions can just add models to that list. And i don't fully understand these rules, but sometimes models added to that list are hidden by default. Other times they're visible by default. I don't fully get it. So if you add an extension looking for a model, you don't find it, go to the model manager and you can make it visible. Or if you have too many lists there, go make some invisible because there can easily be too many.

James 12:57

Got that makes sense. So I think that's what the important factor is, is that in general, like the chat window in VS Code is a chat window, like all up, and it has all these integrations and ecosystem into it. I still think you need a GitHub Copilot subscription even to use that.

Frank 13:16

Perhaps. It's hard to say. I don't know because I do have a GitHub Copilot subscription. So it's hard for me to say, but I will, I do watch my GitHub Copilot and the number is not going up. Like the requests aren't going up, um but I don't know.

James 13:26

Correct.

Frank 13:29

um Like sub agents are always kind of a weird thing. Like I don't know. If it always uses the model I chose for subagents or does it ever use something else as subagents? You know, I wonder about little details like that that honestly, I just don't know myself yet. um i think that I think VS Code being VS Code, there's a million settings you can choose. I just don't know how the defaults work exactly, to be honest.

James 13:55

Let's see, Copilot Search says, yes, bring your own key in GitHub Copilot VS Code, only available to to certain tiers.

Frank 14:04

Interesting. Okay.

James 14:07

Yeah, but you connect to your model provider and you're built through them. You do not consume your GitHub Copilot quota.

Frank 14:12

Yeah. Yeah.

James 14:15

Perfect, gotcha.

Frank 14:16

Fascinating.

James 14:16

Okay, so yeah.

Frank 14:16

Yeah. And this is how a lot of the other um apps work, like OpenCode and CloudCode and all all the, I'm not sure, can you do it in Codex? I'm not sure if that's available. ah

James 14:26

I don't know.

Frank 14:27

Yeah, me neither. Sorry. Sorry, everyone. I'm still living the VS Code life. i i I love VS Code. I'm sticking around in there. um But VS Code is more than just the chat window. It is what everyone's calling the harness these days. I hate that term. So I tend to just call it agent because it took me a while to come around to agent. And I prefer calling things agent. And what agent is basically is a bunch of system prompts and a bunch of tools for editing files and ah but other little basic things like memories and plans, modes for plans and things like that.

So that's the agent to me. But a lot of people call that the harness because the word agent got overused a bit. So what's interesting is when you do choose like a model, and I'll just keep saying deep seek just because my brain's um stuck there for some reason. um You're still using the VS Code coding model. harness it's still their system prompts it's their tools and all that stuff so my little mcp tools that i've added to my like settings file they they all work just fine so the models really have been commoditized quite a bit

James 15:42

That's pretty neat. Yeah, I think that the ability to just go in and then do this and then even run things like we'll talk about locally is very fascinating. Because obviously, you know you decide to um get build in one way or another or ideally even not get build at all. I actually just installed the Foundry Toolkit for VS Code, which is for Microsoft Foundry.

Frank 16:11

Right.

James 16:11

And if, so this is one of those like deep seek seek that has like a bajillion models.

Frank 16:16

Yeah. right

James 16:20

So I'm looking here, I'm looking, we have a code stroll, cohere command, deep seek R1 V3 V3, 3.2 V4 flash llama for metal llama, Mr.

Frank 16:20

yeah Yeah.

James 16:33

Large, all the GPT models.

Frank 16:35

Yeah.

James 16:36

So I would could get build like through this stuff, basically, I assume.

Frank 16:39

Yeah.

James 16:40

And then there's those are get up models via the toolkit. And then there's Microsoft Foundry models, which are like Haiku and Opus and Sonnet and Codex and more. There's like, oh, my gosh, there's a rock and llama. I was actually like there's a bajillion there. Yeah, there's tons. That's crazy. So they're all in there.

Frank 16:56

Yeah, and and if it's a bit overwhelming, I agree with you.

James 16:58

Wow.

Frank 17:01

It's a bit overwhelming. So um you can go to multiple websites out there, have kind of like shootouts, or they try to rank these models for different tasks. Obviously, the only task I care about is coding. um so and And the normal ones generally win. But, you know, there's some upstarts doing pretty good. The the Kimi gets a lot of good reviews, MinMax, Memo. all sorts of throw a consonant and a vowel together. And there's probably a model out there with that name on it and you can try it out. um But yeah, those those are all still paying for stuff. You're still paying for a service. You're going over the internet. um You said something I got interested in was like, well, what if I don't want to pay anything?

And what if I just want to turn off my internet? Can all this stuff still work? And it can, James. It can.

James 17:52

It can. Yeah, okay. I'm interested because i did just also have that in my video, but I feel like I made that video a year ago and i my assumption here is that the models that I can run on my machine have dramatically changed between then and now. So i'm really fascinated to see how that goes ah because I also literally just installed like again that foundry toolkit.

Frank 18:15

Yeah.

James 18:22

I feel like I installed it a while ago but I didn't go into the model selector.

Frank 18:22

Yeah.

James 18:25

Now it makes sense as a model provider and

Frank 18:25

Right.

James 18:29

there is a bunch of groupings and one is called Foundry Local via AI Toolkit. And there is all the DeepSeq models, GPT-OSS, Mistral, Phi Silica, all the Phi models, Quen, all the way up to three.

Frank 18:35

Cool. Yep. Quan, my buddy Quan.

James 18:44

yeah So, and um and it shows you if there are capabilities is of vision or tools or, you know, what their context window is.

Frank 18:45

if Yeah.

James 18:51

So um help us, Frank Kruger, understand,

Frank 18:54

Okay.

James 18:56

what you needed to do and what that experience was and what type of hardware you're running it on. Because right now I'm on my Windows laptop, which is running something, I don't even know. um i can I can see what it's running. Let me go to the task manager and it is currently running a, a what is that, CPU, an AMD Ryzen 7 Pro 350 with

Frank 19:13

h I don't know my Radeons very well. I apologize.

James 19:29

I do have an NP, i have an NPU.

Frank 19:31

so

James 19:31

Yeah,

Frank 19:33

NPU.

James 19:34

yeah neural processing unit, yeah.

Frank 19:34

Motion Processing Unit. Oh, NPU. Yeah. Okay. So, um I mean, in some ways, what you probably did a year ago is not too different from today because in the world of open source models, we all still kind of upload our open source models to Hugging Face, and there's billions upon billions of them up there. all with the big names and everything. i would say the biggest change, James, is that um we've gotten, the models have just gotten better, like to a scary level.

So I remember just a few years ago, All I could run on my kind of Macs were the 7 billion parameter models. And even those were kind of slow because the software wasn't tuned for them. People were happy to just get them working. They're like, oh my God, look, a local large language model running on my computer.

James 20:26

Yeah. Hmm.

Frank 20:27

Isn't that cool? Yeah. Well, this past year, people have just been obsessed with the engineering aspect of just making bigger models run faster. And it's glorious. I've been waiting for this time to happen for so long because like, yeah, all you got to do is throw some engineering prowess against these things. They're just big calculators, you know, just people love writing calculator code, just make better calculator code people. And so um what has happened is the models that are available are better.

you And the things that can run them can run bigger models and in a more sophisticated fashion. So, um and I was a bit out of the loop also. So I wanted to take this last week and try some things out and just see how they feel. So I was living in the Opus world, just for context. I spent all of April, checking what month it is. I spent all of April living that Opus life. um It's my buddy. It's my friend. We get along great. um so But then I spent a few days running the DeepSeq, and now I've spent few days running QWEN.

QWEN, Q-W-E-N. It's a 27 billion parameter model. The ones that people like are QWEN version 3.5 and QWEN version 3.6. It's well understood these are not the greatest coding models out there. But on my hardware, which I'll get to, you can run 256k context windows, which is huge.

James 22:02

and That's pretty good.

Frank 22:03

Because even with Opus, um VS Code was limiting to me to 192k before it did like compaction and that kind of stuff.

James 22:03

Yeah.

Frank 22:12

So a 256k window locally is really impressive. like It blows my mind. James, when I, you know, I work on that cuneiform project where I'm training large language models. You know what the largest context window I could put onto that was? Just guess.

James 22:28

ah Maybe like 20 twenty k

Frank 22:32

512.5 K's.

James 22:34

now No, no.

Frank 22:37

That was the biggest I could get on there. That's well, because that was 32 bit floating point math and it was doing all sorts of complicated things. And the engineering has gotten to the point where I can do 262 It's just impressive.

James 22:51

That's crazy.

Frank 22:51

So um we both have Mac minis, kind of overpowered Mac minis. So we can talk about those, but I'll talk about what I've, yeah, there she is. Isn't she cute? Yeah. um What I've actually been running mine on is I've had an RTX 3090. Anyone who's ever listened to me talk about things, it's it's my favorite GPU ever.

James 23:06

right.

Frank 23:09

I've had it for a few years now. I love using it for everything. ah But truth is, it's just been sitting there idling, doing absolutely nothing for the past few months just because I haven't been training neural networks or anything. So I'm like... The reason I like the 3090 is it has 24 gigabytes of VRAM, RAM, whatever you want to call it, fast RAM built into it. And 24 gigabytes can actually run these 31 billion parameter models if you use 4-bit quantization. So that is, we are using 4 bits for every parameter in the model.

So you need roughly 15 gigabytes in order to run these models. And then you need the other nine gigabytes for your context and your output and all that kind of stuff. So I went through the process of installing llama.c++. It's one of my favorite runners out there. You've already mentioned OLAMA is a good choice. um Kind of the industry standard is another one called VLLM. In the Apple world, there's MLX now. you know there's There's runners everywhere. People have gotten the religion of optimizing these things and installing them. So I went through the process and I wrote a little blog entry about it. Everyone can follow along if they want to go to my blog, preclorum.org.

And I give all the setup instructions. It was easy, Jane. Download the model, pass a few command line arguments, be amazed at how port forwarding still doesn't work in the year 2020. Yeah.

James 24:46

and

Frank 24:47

Try to get your stupid network into good shape. Try to figure out what incantations vs Code wants. And all of a sudden, i had I chose Quinn to start with. um It was there. It was just VS Code, acting like an agent, running sub-agents, doing planning mode, running with autopilot because life's too short to care about permissions and all that kind of stuff. And i had it analyzing my code right away. and I want to talk about like my experience with it, but that's pretty cool that I was pretty easily able to just get this thing up and running. This software has really advanced, I think.

James 25:26

Yeah, no, I think when the the team first started adding that feature, you know, it seems like an advanced feature because you were talking about hardware requirements and GPUs, you know, the 3090s on cheap ah nowadays, maybe back in the days of yore.

Frank 25:37

Yeah. No.

James 25:41

But... ah ah but you know, being able to plug in an API key or plug in a thing, you know, that's relatively straightforward, and but it's still running remote remotely. But having something running on your machine that you'd be on a plane, you'd be disconnected, it's just sitting there ready to go is really magical. That being said, there is still some edumacation because like you said, you have to install a thing and to get this thing and then do some port forwarding stuff. But that being said,

as far as the steps go to get that required and then it just kind of works is a magical feeling at the end of the day. It's kind of like when you get like an Android emulator up and running, you're like whoa, look at that thing.

Frank 26:24

Yeah.

James 26:24

And then you're like, wow, this i run it's like, wow, that only took a thousand steps and a bunch of pieces of software installed and bunch of SDKs to download. Then boom, there it is, great. I did it, right?

Frank 26:32

Yeah.

James 26:33

But that's really neat that one, it's built in, it's just ready to go.

Frank 26:33

Yeah.

James 26:39

And then that you're able to figure it out. It seems relatively relatively straightforward.

Frank 26:44

Yeah, I think what made it so comfortable is I didn't have to switch out of my normal workflow because over the past whatever months, I've gotten very used to using the agent chat and everything in VS code. I know how it works. I know what to expect out of it. And it was so weird to just see a model that I knew was running because, you know how I knew it was running, because I could hear the jet engine on the other side of the room as every fan on that RTX spun up.

James 27:05

Yeah. yeah

Frank 27:09

You know what you know what optimizing a model runner really means? They are... pushing that video card as hard as it can possibly be pushed. And that means power usage. And that means fans, thermal regulation. So like I do a lot of network training and I hear the fans on that thing. I have never heard it screaming the way it screams when I ask a VS code at chat thing to go analyze every single file in my repo and report back every bug. Fine.

ah It just starts howling. but um i So that was just such a satisfactory feeling knowing and I even did that funny trick of I disconnected from the internet. Obviously, my network was still up, but I disconnected from the internet just to prove that everything was happening locally. And it was, it was beautiful. So model quality, though, I think this is kind of the big one. um We are running compromised models here. For one, they're much smaller than even like Sonnet. I don't know how big Sonnet is, but it's probably five to 10 times larger than the Quinn 27 billion parameters. It's probably like 300 billion, something out there.

Um, And on top of that, I'm running a compromised version of the model. I'm using a quantized version of the model. So four bits per parameter, when usually you have 16 bits per parameter. And then I'm compromising it even more. The context that it's keeping and the output buffer it's using are also quantized. Those should all be 16-bit. They ain't. They're 4-bit also.

James 28:50

Mm.

Frank 28:51

Yeah. That's how you get the big context out of it. You quantize everything. yeah You really compress it down. um But at the same time, James, I can't really tell the difference between Opus, DeepSeq, and the stupid 27 billion parameter model running on my other machine. And I don't know what it is because if you asked me a month ago, I'd say opus forever. Don't you take the opus from me ever. And now I'm just like, can I even tell the difference between any of these models? I'm starting to get very suspicious of myself.

and So honestly, for the past day, I've just been like running the same prompt on like six different models to just see what happens with all of them. So I do want to say, OK, it's slower for sure. I'm getting like 40 tokens a second. I think Deep Seek, you get like 60 tokens a second. Opus, you get three tokens per second. Just kidding. Opus is fast when Anthropic wants it to be fast. Other times it's horrendously slow.

Um, it's a big jet engine over there and it does make mistakes once in a while, but you know what? Opus makes mistakes once in a while. Um, it's sometimes bad at file merging. This is like, you can tell where the VS code, um, harness and prompts aren't perfect because like it keeps messing up how it uses the file read tools like oh the file read tool needs line numbers i guess i'll pass it line numbers like yeah okay maybe you've just done that in the beginning like there's these funny little things but like it figures it out at if at just every session it has to figure it out every time but it figures it out

James 30:21

Oh, yeah.

Frank 30:33

And I just, I don't know what's wrong with me or the world or what's going on, but I have been so satisfied with these tiny little models. And I'm pretty sure I'm not deluding myself, but like, I think it's just the systems that we've developed, like good agents files, doing planning, asking questions, and then the coding part just becomes kind of rote once you've kind of approved all the plans and everything.

James 30:50

Hmm.

Frank 30:58

So yeah. Aside from the slowness, I honestly can't tell the difference between my little local model and a sonnet or something like that out there or a small codex. And I've also been trying, um Google has Gemma or Gemma, I'm not sure how you're supposed to pronounce it. ah Gamma 4. And that's a great little model, too. It's hard for me to tell the difference between QN and Gamma. You go to all the websites that benchmark these things. They all use that stupid SWE bench.

Benchmarks are all a joke, everyone. Just put that out there. But... um

James 31:34

That's

Frank 31:36

you know, the graph says this one's way better than the other one. But then in practice, you're like, I don't know. They're all making my job better and my life happier. And aside from the fans roaring, it's really hard to tell the difference. So I've i've been really excited by all of this. It's, it's, yeah.

James 31:53

pretty... Yeah, it's pretty neat. I mean, I think the biggest thing that I had a while ago was the speed of it, but I was also, I forget what hardware, maybe i was running it on my Mac Mini, I'm not positive.

Frank 32:00

Yeah.

James 32:04

But mostly the speed of it, you know, I use a lot of GPT models, which are quite fast, especially in CopileX, they're hosted in Azure, right? So they've got speediness to them. and agency and then or some of the smaller models like a like a sonnet or like a haiku or quite quick quick right compared to an opus so i i'd use a lot of those those models all the time and to me the speed is one factor but also correctness and being able to read and just the the answer at the end of the day is like is it correct or is it not i've been doing this

Frank 32:30

Sure.

James 32:36

trial ah since the beginning of the month. I posted about it, which is I'm not picking any model manually by hand anymore.

Frank 32:43

Oh, God.

James 32:44

I'm only using auto model, which auto model. Well, well, it seems to be different for for each tool, um but On the trip to Guatemala, I only have my phone. And as you know, on flights and long layovers, now that I have Starlink on Alaska, I do a crap ton of coding.

Frank 33:02

yeah

James 33:04

And I do it all with the cloud agent, the GitHub coding agent, the cloud agent.

Frank 33:08

yeah yeah

James 33:09

And the default is auto. So I just let it YOLO. And then when I got home, I said, I'm only going to use... I mean, when I say only, I mean like 95% of the time, unless I need to switch something because something is just not clicking. But inside of VS Code and now inside the CLI, you can just end VS, you can just hit auto. And based on like, it's not based on your prompt, it's a based on like availability and a few other factors of the model, it'll pick a model. Often this is like a Sonnet model or a GPT model, but it really just depends on time of day XYZ and they give you a little bit of a discount, 10% off for using auto.

And what I like about auto is that if it's based on availability, that means it's probably pretty quick and it's moving really quick too, because it's not being stressed.

Frank 33:50

Right. Yeah.

James 33:53

And um I'm trying to figure out, and the reason for this experiment is, How much does the model matter? And this is exactly what you're doing, right?

Frank 34:02

Yeah.

James 34:04

And I do believe that the models do matter in some instances.

Frank 34:04

Yeah.

James 34:07

So what I'm trying to comprehend in my mind is when does the model matter? When does the reasoning matter?

Frank 34:13

Right.

James 34:15

Because it's so easy to default into models that we think are the thing that's good for this thing.

Frank 34:19

Mm-hmm.

James 34:20

compared to just saying, maybe they're all pretty good. And then just kind of YOLO it on the auto model selection. Now, when I mean auto, I don't mean autopilot, I don't mean bypass. I mean like there is a dropdown that says auto and just go and it will pick from the available models.

Frank 34:33

0.9x, baby.

James 34:34

0.9X and just a little as a rip, right? So you get that little sweet discount and go to town there. So I'm trying to do almost the same experiment but obviously running you know normalm you know foundation um models in in the cloud, but but kind of the same thing.

Frank 34:41

Mm-hmm.

James 34:48

Does the Gemma, does the Quinn, does the DeepSeq model, how much of it does it make a difference? There's probably, I'm interested in this, like was there a point, i know it's only been like a week or so, but has there been a point where you did switch back to Opus or Sonic? Because you said in the beginning, like you were still favor of these models. Favoring is not the same as using or having to use.

Frank 35:06

Right.

James 35:11

them? Has there been a point so far where you're like, you know what, I'm actually going to switch to this other model because my house is getting too hot.

Frank 35:21

You know, I kind of like hearing it spin up. I keep making fun of the fan noise, but like, you feel like you're a power user. You're like, hey, ah rename this variable and then jet engine start off. You're like, oh boy, that's getting really renamed over there. um First, I want to say I really appreciate like the auto thing because i i think that's the biggest lesson I want to take out of this. I don't even want to promote like... um local models are the way to go necessarily. More I'm just realizing how important your processes and systems and the the the information we give these models, just how important that all is.

But to answer your question, I've had zero interest in going back to the big provider models. um Every so often you have like a little Tweety bird in the back of your head saying like, I wonder if Opus would have solved this already.

James 36:14

Mm-hmm.

Frank 36:17

you know Every time it introduces like a little bug or something like that, you wonder would Opus have made that mistake or would Sonnet or would Codex have made that mistake?

James 36:24

yeah

Frank 36:27

But in the end, like, who cares? It's free. I just tell it to go fix the mistake and it goes and fixes the mistake. You know, it's it's the same UI and everything. So I think maybe it's maybe too soon to know whether I'll switch back and probably I'll switch back because I'm going to keep paying for a co-pilot and I'll use some credits there. You know where I'm really going to use co-pilot is the cloud stuff, like you mentioned.

Because what I found is when I'm being truly productive, I'm using issues on GitHub and PRs on GitHub and using the cloud models. That's when I'm getting like my six things are happening at once and I'm being ultra productive. The model I'm using on my dev computer, it's okay because I'm usually thinking through a problem or I'm in a greenfield application and I'm doing design work. You know, I'm not trying to implement a feature or fix a bug.

James 37:19

yeah

Frank 37:22

That's that's all happening and on the web.

James 37:25

yeah

Frank 37:25

That's just happening in the background. this is though I'm having a discussion with the AI and it's okay if it takes a second or two to have its little discussion and I don't mind that. So I think that that's actually where I'm going to kind of settle. Like for my dev machine, I might just keep cranking on the 3090 and just enjoying it from enjoying trying different models to just getting out of the hegemony of anthropic and open AI. It's fun to use these open source models. They're all different. They have different capabilities. um You know what they'll allow you to do and that kind of stuff.

ah So, i yeah, I'll put it this way. We are at the 5th of May, not too much behind the scenes on how our ah recording schedule. And I have used 0.6% of my co-pilot credits because I've just been rocking these local models and it's been fine. And I have felt very little need to go to the bigger models. Yeah. That said, when I use the cloud-based models, that's when I'll probably be digging into my co-pilot credits or tokens, whatever they're going to call them after June.

James 38:43

Usage.

Frank 38:45

Usage.

James 38:46

Usage.

Frank 38:46

Fat.

James 38:47

So yeah, that's really, that kind of gets back to that ecosystem play, right? And yeah we're doing those GitHub Copilot dev days and I kind of talk about that, like the ability to, tap into the ecosystem, tap into other model providers, tap into other, you know, um, harnesses, if you will, and being able to use things on the plane, on your computer, in the terminal, you know, on your phone, you know, all these different areas, assign issues, do these things.

Frank 39:12

Yeah. So,

James 39:14

And that's how I use it. So to me, it's not about one tool. it's about the combination of all the tools. Now what's fascinating is like, we're going to use those tools and then we're going to, um, use different models in different instances based on where those tools are running. Right. Because If you're using a cloud agent, it's gonna be running and needs to talk to cloud stuff, right? So that's really, um really, really interesting in general to think about.

um I, yeah.

Frank 39:40

i want I want to talk about performance a little. Sorry, didn't mean interrupt.

James 39:43

Yeah, yeah. I was gonna ask, like did you have a specific thing around models, performance, um outputs that you were leaning towards?

Frank 39:45

Okay, so, yeah.

James 39:52

Because you talked about the different ones and that you're wanting to explore with more, but yeah.

Frank 39:57

Yeah, and i I've been doing AIs forever, and people keep talking about tokens per second, and I honestly don't know, like... What's a good number? You know, obviously bigger is better.

James 40:08

Yeah.

Frank 40:09

That much I know. But is 40 tokens per second tolerable? And what I've discovered is it very much is. You definitely get annoyed with the thinking loops that a lot of these reasoning models can get into.

James 40:22

yeah

Frank 40:24

um There's a lot of people who are just turning reasoning off because like it's faster for it to make a mistake and you correct that mistake than watch it think in loops forever. It seems these things really seem to get into cyclic loops and all that kind of stuff. So I think that there there are benefits to turn that off. But 40 tokens per second, it's still faster than I can read. So when it's editing code, it's plenty fast. When it's pumping out text for me to read, it's faster than I can read. So it's fine. um Now, that 40 is, again, with the the model quantized and all that running on today, I think, the 3090s, a 1,500-hour GPU.

ah But we both have Macs. And you can take that exact model, same exact model, same quantization, everything, run it on the Mac. And you're probably going to get like 17 tokens per second. Which, again, is fine for text. But when you have like sub-agents going and reading your entire repository, too slow. When you have it thinking in loops, too slow. um So the 17 is really bad. The good news is, James, in the last week, all of this has really been improving. um There's a new technology out there, MTP, and I'm trying to remember what the stupid thing stands for. Multi-token magic, MTM. I forget what the P stands for.

um it's It's this like predictive, oh, maybe it stands for predictive, where there's actually these... models have smaller models that are just kind of dumber. And they're just guessing what the next few tokens are going to be. And then the big model just checks whether the next few tokens should have been those things. And it turns out this cooperative way of um doing inference is faster. So you have a little stupid model doing a bunch of predictions, and then the bigger model is actually um validating those predictions. And both Quen and gamma slash Gemma ah support this.

And the new software out there, um there's on Mac, there's MTPLX, which uses the, oh my God, it's too many libraries.

James 42:47

Oh, cool.

Frank 42:58

And this is the MLX library, i like a fork of it to do this, where it does the intelligent thing, where it has a stupid model running ahead and doing stupid predictions and then the bigger model validating. And then um this is something Google invented, by the way.

James 43:13

Cleaning it up.

Frank 43:16

So everyone just like, oh, that's a good idea. Let's take that. And so ah Google just released that feature for Gemma, which already was a powerful model, but now it's fast. So now you can run these models, these 30-ish billion parameter models, at 60 tokens per second on our max, James. So I'm telling you, that's plenty fast because even a lot of the service providers out there for the big, big models, they're only giving you about 60 tokens per second between network bandwidth and them being overloaded by DOS attacks and all that kind of terrible stuff that they have to deal with.

60 tokens per second is plenty fast enough. And it's really cool that just in the just in the last month, I would say, this new technology has come out and is really revolutionizing inference speed.

James 44:12

That's crazy. Well, I do want to ask one thing because not everyone has a 3090 sitting around Frank Kruger.

Frank 44:17

Nice.

James 44:18

And the real question becomes like, what if I just have like my Mac mini or my Mac mini or more realistically, maybe a MacBook air that's sitting over here. Like, I think I almost want to follow up next week, which is like, what can you do on your MacBook air inside of a coffee shop?

Frank 44:30

Yeah.

James 44:36

Right.

Frank 44:37

Yeah.

James 44:37

What could you do on your Mac mini that's over here? Cause I don't think realistically, that going out and asking people to buy a 3090 to run these models is going to be realistic, because then I would just spend that on toke token usage maybe, right?

Frank 44:48

Right. Yeah. Yeah.

James 44:55

um More realistically, what I'm really fascinated about is maybe running smaller, tiny models as sub-agents automatically so they don't use tokens.

Frank 45:04

Right. Yes.

James 45:05

Or maybe there is a, hey, this one works really well on the Mac Mini. And so I'd love to see a follow-up article, which is, hey, you have this, but here is how you can use an optimized model based on the hardware that you have. Cause some people like us have our beefy, what, how many gigs of randomness thing have like 64, 128,

Frank 45:25

I went with 64. I can't remember what you went with. I think you went the same.

James 45:28

64 then, whatever you got.

Frank 45:28

Yeah, 64. Yeah, cool.

James 45:29

And then I got the two terabyte, you got the four terabyte.

Frank 45:30

yeah cool

James 45:32

So we got this, right? So this isn't even realistic because many people aren't going to buy this configuration.

Frank 45:34

Yeah. Yeah.

James 45:37

They're going to buy the base model, which is 16 gigs of RAM, which means it's more realistic on our M1. know it's a different M4 Pro, but M1, which is a 16 gigs of RAM on our little MacBook Air. So the question is like, what could be a good optimized model? How would you run in here locally versus running it on this thing here? Because I think that is a more realistic scenario.

Frank 46:03

Yeah, 100%. And I even make that point in my blog because I price out what would it cost to build a 3090 machine these days? And it's about $2,000. And $2,000 gives you 10 months of the highest tier of Claude code, you know, that that are, yeah, don't know.

James 46:13

Oof. Yeah.

Frank 46:24

What's the most expensive copilot, like $40 a month. So that's, that's,

James 46:29

A lot. Yeah.

Frank 46:30

months ah so yeah you can i'm not even going out there and saying go build a machine or anything like this but what i do say is if you have a machine like that you'll probably be using and then yeah yeah

James 46:32

Yeah, yeah there you go. But I'm also interested in, I do have small, I have a bunch of machines. I don't have that machine, but I do have like little, I mean, I have to imagine a Mac mini, it's gotta be doing something. Did you run any models on the Mac mini directly?

Frank 46:54

yeah 100%. Yeah, I have been. um So I have mixed feelings about it because our Macs do a great job running it, but it is definitely eating at the memory and it's eating at the GPU. So you're slowing down basically everything else on the computer. And the fan turns on on my Mac mini. Every time the fan turns on in the Mac mini, I'm like, oh, I'm sorry. I'm sorry. I don't mean to hurt you. Like, I don't care.

James 47:19

Yeah.

Frank 47:20

The RTX can start a fire back there. I don't care. You know, burn, baby, burn. But... My little Mac mini, I'm like, oh, I'm sorry, are you getting too hot over there? So um that's probably stupid of me. But probably the biggest, weirdest change that I've seen um in AI since I started is in the beginning, we were all compute obsessed. How fast is your computer? How many flops can you do?

James 47:44

Mm.

Frank 47:45

Today, that's almost irrelevant. All anyone cares about is how much RAM do you have and how fast is that RAM? Because these models are so big. The reason the 3090s, 4090s, are nice is because they have 24 gigabytes of the RAM. And therefore, these small models are small. 30 billion parameters can fit on them. Um, because we have 64 jig lights, you can fit some even bigger models, but it turns out like the model makers out there, they, at there the, the small models are about 30 billion parameters.

And then the next step up is 300 billion. There's not really an in-between ground there.

James 48:25

Wow.

Frank 48:27

So, um, you definitely can take like the Quen 27 billion, run it on your, um, Mac and it's going to run great, but it is going to use all the resources and you are depleting your dev machine and you'll notice hiccups here and there. Like I had YouTube music playing in the background and every so often you'd get like some static because the poor little processor is cranking away. um and you And when you go to activity monitor, you see the GPU usage at 90%. You're like, ooh, okay, we are burning this puppy. um And then it becomes a thermal game.

Like everything else, it becomes a thermal game. So yeah, it's it's a mixed bag running these things yourself.

James 49:07

I think, yeah, I think that the bummer part here may be, I could be wrong, but the Mac mini does not support eGPUs. So you can't just like plug in an external GPU into your Mac mini.

Frank 49:29

ah Yeah, I agree. And, you know, I've actually run eGPUs with Macs in the past, and that's even a bit of a mess, to be thoroughly honest.

James 49:38

Hmm.

Frank 49:39

um Thunderbolt is fast, but you're sending so much data so fast to make these models work. It's... a little bit weird. One of the reasons MTP is such a good technology is it actually keeps a lot more of the data on the video card before it ever has to get back to the CPU. um They try to do like their sampling on the video card.

James 49:59

Hmm.

Frank 50:03

People don't think about it, but these are probabilistic models. For every vocabulary word out there, it's giving a probability that it's the next word. And something has to take that list and sort that list and take the top K and do the top P off of that top K. And that's not, that is a serial operation. That is not efficient on a GPU. GPUs are parallel devices. And that is a giant serial operation that has to happen. And so for unoptimized models, there is a very huge cost we all have to pay if you're running these things yourself, where lots of memory has to be copied down from the video buffers into CPU buffers.

Unless you have a unified memory architecture like on a Mac. and then

James 50:51

Yeah.

Frank 50:52

Life is good. I think some of the Windows ARM computers have unified memory architectures too. I just know less about them than M's.

James 50:58

Interesting. All right.

Frank 51:05

All right.

James 51:05

I am going to try something on my Mac, that's for sure, because i do have it just sitting here ready to go.

Frank 51:06

Well.

James 51:13

I think it's fascinating and do want that little little fan to spin up on occasion.

Frank 51:17

Yeah.

James 51:18

But I do also want to point out that you can actually do this in the CLI now too.

Frank 51:18

and

James 51:22

You can actually configure it.

Frank 51:23

Hmm.

James 51:23

don't think it's as simple, but I know Kayla Cinnamon on hers went out and um has a whole thing about how to configure OLAMA and run it as the the select the models that are running there inside the CLI as well if you want to run in the CLI. So it could be cool too to give it a try.

Frank 51:42

Yeah, and I think it's just one of those. here's Here's the real deal, James. I never want to go back to writing code in the old way anymore.

James 51:53

Mm. Yeah.

Frank 51:54

But I don't like the idea that my ability to code is dependent upon paying someone a service fee every month. I don't love that.

James 52:03

Mm.

Frank 52:05

So what I like about all of this, and it's really more of a mental thing than anything else, to be thoroughly honest, is I feel like I'm back to I myself in my little office without an internet connection can still code again in the way I prefer to code these days as coding has gone through a transformation in this past year. And I don't want to go back. It's just, I'm over it. I want to stay in this world.

James 52:30

Yeah, me too.

Frank 52:31

And ah it's it's a little bit of a liberation to run these models yourself and to see, okay, they're good enough. I can still keep doing my planning, keep doing all my, keep doing my agentic workflows. Feels good.

James 52:45

I'm glad that we changed you over. Took a little bit what we got there, Frank. And yeah, I'm excited for the year ahead. I mean, it's only May and who knows? I mean, for me, I think with the next next steps here as we wrap up is it's more feasible than ever to use these models locally and combine them with the ones that you're using on the cloud and the different services of where you're doing your coding and how you're coding.

And what I'd love to see even progress even more is for all of these bits and pieces to come together. And I think they have come a little bit together in the form of extensions, but i would love to see that ecosystem unify even more. like I feel like we have the capability of building some sort of GUI or some sort of thing that really streamlines these processes and makes it easy to configure and connect all these different harnesses and tools that we're using out there to make it easier than ever. like That's what I would kind of expect because that would then enable more individuals to do that or maybe just build it into the tooling. like I do think that there is

something good and something bad about what Apple did on their machines. And I guess technically even on my Windows machine, there's there's models that are running locally for all the AI stuff. But it is like, do I want to install this thing? And one thing that might be fascinating is, is there a future that I see, which is streamlining this process to say, okay, download this model, like an easier configuration wizard is what I'm saying.

Like there's all these libraries and all these things, like how do we streamline that bit and piece?

Frank 54:05

Yeah. Unified, I'll just take because I'll be honest, I have like six different model runners on this computer and they all store the models in different directories. And I have models everywhere on this hard drive and it's just eating all my gigabytes and I want my gigabytes back.

James 54:22

Yeah.

Frank 54:27

But I'm not going to go organize these.

James 54:27

Yeah.

Frank 54:29

I have to run my grand perspective to go find all these stupid models. So I'm hoping at this WWDC, Apple comes up with some unification for um having like background services and good things that can

James 54:33

That's great. Yeah.

Frank 54:47

it's It's tough because, sorry, i we are trying to wrap up, but like we're still in the wild, wild west here. you know Innovation is happening. So you don't want to unify, prematurely unify and prematurely standardize this stuff.

James 54:59

yeah

Frank 55:01

It's fun having everyone ah competing against each other for speed and all that. And I don't want to lose that competitive spirit. But it is a little bit exhausting. So hopefully within a couple of years, it'll all be unified and standardized.

James 55:13

I'd like to see it. Well, let us know if you're running a local model, where and how you're running it. Maybe you have a 3090, maybe you don't like me. And then what are you running it on? Let us know. Go head over to our YouTube, youtube.com forward slash at Merge Conflict FM. And that's a great way to leave a comment on this episode or any of our past comments. And we'll do a listener bag episode. There's quite a few chimed up, so we'll get it going. So let us know. um But ah that's it for your local edition of Merge Conflict. until next time, I'm James Montemag now.

Frank 55:41

And I'm Frank Kruger. thanks for watching and listening

James 55:44

Peace.