534: Why Developers Are Obsessed with Budget LLMs (Jev, Luna, & more)

Frank 00:31

but

James 00:32

ah Yes, I was at NDC Azo. I recorded the last one from. Oslo if I'm pretty sure I did at least. Right?

Frank 00:38

Yeah, sure.

James 00:39

Or London? not I don't know. Wherever I was.

Frank 00:40

We'll go with that.

James 00:41

I was in London and then I was in Oslo and then I was in Germany. Now I'm in Austria. We went to Budapest today and then we'll be back to... London and then I'll be home. in oregon by sunday night at midnight and then fresh

Frank 00:52

Amazing.

James 00:55

to work at 7 a .m Very exciting.

Frank 00:57

Well, I'm glad you get to be our foreign correspondent yet again.

James 00:57

So. uh

Frank 01:01

I can't wait to hear about what's happening in Vienna. i'm I'm sure this is just a travel podcast now as far as I know.

James 01:09

you know uh vienna is very beautiful we're just talking about very very beautiful budapest also very beautiful we were in innsbruck doing in leo gang we're in salzburg all very very pretty vienna just we were walking around even though we'd already been here once i'm like man so It's just beautiful. just Everything is so regal and big.

Frank 01:24

Yeah.

James 01:27

and just beautiful and just like

Frank 01:28

Seed of an empire. Yeah.

James 01:30

ah It's... ah It's crazy. If you've never been to Vienna, I definitely recommend it. It's very, very lovely. And yeah, so excited. uh to be here on the road we took this this week off between ndc oslo and heather as an event here in vienna so that's why we're here now but we went did a bunch of sauna stuff in between and we took

Frank 01:48

Oh, okay. Fine.

James 01:51

In five days, five aerial tramways lifts up to mountains.

Frank 01:54

Wow.

James 01:55

Five of them.

Frank 01:56

Phenoculars

James 01:57

funiculars

Frank 01:57

cool that's your favorite topic i can't wait to see what ai is going to generate about these ones this time

James 01:58

Yeah. It was good. I'll tell you this. Okay. So before we get into some new models, so there's. A few things that...

Frank 02:08

Wait. Okay.

James 02:10

I think that... I wanted to talk about because I just we had a little bit of fun. On this trip we took... um We drove a bunch. We took some trains. We took so a bunch of buses.

Frank 02:21

Mm hmm.

James 02:21

and one thing that we forgot to bring was a keller had a as a is ebook you know uh I have my Switch 2.

Frank 02:30

Good.

James 02:30

But we sort of forgot to bring something that we could do together. and we saw some people playing cards and stuff

Frank 02:33

Right. Right?

James 02:36

uh and one thing heather and i both read like we're like family feud but i was like wheel of fortune heather's really good at it So I built. uh a two two or three player wheel of fortune game it's called wheel of wisdom because i don't want to get sued

Frank 02:44

Oh my God. Okay.

James 02:50

It's at wheelofwisdom .app.

Frank 02:51

All hallucinated trivia. Sorry.

James 02:53

ah Yeah.

Frank 02:54

Oh, say the URL again, please.

James 02:55

wheelofwisdom .app

Frank 02:58

Wow.

James 02:59

um it's a full progressive web app you can download it offline works there's 600 puzzles stores your history, stores your games. It's got a bunch of different categories. ah It's got trips and mysteries and all sorts of stuff. and There's going to be a... i'm I'm just getting ready right now to start a ah full fresh build. uh for an ios application too it's going to be a dotnet maui hybrid app so i'm going to reuse this

Frank 03:26

Oh yeah, cool.

James 03:27

eight HTML and JavaScript, it turned out really good. I built this but baby with baby Luna ah that we're going to talk a little bit about this week.

Frank 03:32

so okay

James 03:35

And a bunch of other models too. I used a bunch of models on it based on the task at hand. But it's really straightforward.

Frank 03:39

Okay.

James 03:40

It's just HTML and JavaScript, nothing fancy. But it's just like, I built it from my phone, Frank. Like this was the thing I wanted to talk about.

Frank 03:45

Uh-huh.

James 03:46

I didn't even, I have my computer with me. I'm here at the beautiful Moxie Hotel. um If you've never seen it on Moxie, it's like very... funky and you know exactly what you're getting, which is a tiny room for not a big price.

Frank 03:57

the

James 03:57

um And they're everywhere. And it worked out for us for our location we needed to be. And but. as we were going via bus via whatever it in my laptop and i coded this 100 from the github mobile app Uh...

Frank 04:11

Pretty cool.

James 04:12

i even bought the domain name set up the dns everything 100 from my phone ah using just the cloud. you know, agents basically, which is it's absolutely. ridiculous. And we've been playing it on every single thing that we did. So I just want to kind of throw that out there.

Frank 04:30

Fantastic. i Honestly, I think you're just feeding into your domain registration addiction. I think you're just looking for excuses to buy new domains. But um this is pretty cool. I haven't actually made that leap of doing it all from a phone yet. um So once again, you're ahead of me. I'll give it a shot. I'll write some trivial game as a web app on the phone. And and then this is ah there's a fun, trivial game because already there's dollar signs at the top. So I can imagine you two trading money already. I hope there's gambling rules and drinking rules with this game.

James 05:00

Oh yeah.

Frank 05:01

um But. it's pretty fun i'm excited to hear what your baby luna experiences are because um I don't know if you've been getting the news over there in the Europe. But um over here in the Utsa. ah We're not making models anymore. We're slowing down. We're we're slowing progress down. no No more model releases. And by no more model releases, we mean... both major model providers are going to release a whole new suite of models in the last week I don't know if you've gotten that news, but we're we're both slowing down and speeding up over here. How is Uruk doing?

James 05:35

ah er yeah The interesting part about Europe... is There's actually more AI kind of everywhere than...

Frank 05:43

Oh.

James 05:42

you would you would think so. Oh, definitely. like There's a few small towns we go on the Oregon coast where we like are on the Facebook. It's the only time we go to Facebook because usually each community has a Facebook feed.

Frank 05:52

Yeah.

James 05:54

and almost all the graphics for like the the restaurants are all ai generated for 100 percent

Frank 05:58

Hmm.

James 05:59

Here, what's really fascinating is they're really into AI video. like stuff.

Frank 06:04

Scary? Okay.

James 06:05

ah a advertisements, and they all say AI in the bottom right corner, but there's there's a lot of them.

Frank 06:07

Right.

James 06:10

Uh, quite a bit in train stations and things like that which i haven't really seen in the states maybe in san francisco or something

Frank 06:17

Mm hmm.

James 06:17

uh more i haven't really seen it in seattle but there seems to be more of that

Frank 06:19

Wow.

James 06:23

around and definitely more seeing But I'm not seeing Like big ads, obviously, even here in Vienna for like Claude and, you know.

Frank 06:32

Hmph. Yeah.

James 06:32

open ai and chat gpt and all this stuff so i don't really know how that's going necessarily But I will say. Even though I'm here in Europe. My Twitter feed is still filled of a whole bunch of stuff. And funnily enough, Frank. um Both. Burke and I have been tweeting a bunch about Luna. And I tweeted...

Frank 06:49

Yeah. It's caught on dude.

James 06:50

A day before. A day before GPT-6, Luna came out, and I said, Luna Max.

Frank 06:55

Yeah.

James 06:56

And i sweared I swear to God, I had no idea that this model was coming out. at all. But GBT 6.

Frank 07:05

Yeah.

James 07:05

Luna and GPT-6 Sol. No more Terra.

Frank 07:08

So.

James 07:08

Terra's gone, apparently. I guess.

Frank 07:10

Who needs Earth? Earth overrated.

James 07:13

Rep.

Frank 07:14

That's what I hear.

James 07:15

Those.

Frank 07:15

planets. Blah.

James 07:16

Both landed at like half the price or whatever.

Frank 07:20

Yeah, it's really weird because both companies are playing the game of like our token counts are changing dramatically and we're modifying our prices. So it's just it's I take it as the money people just messing around so that people can get roughly the same amount of work done for roughly the same price. Technically, they lower their prices, but. If you look at how token hungry, especially on Macs, some of these models can be, it's made up for with those new trainings. It's a little bit all over the place. Honestly, it's so hard to keep up with this stuff now. I'm surprised we're even trying to keep up. Thankfully, I have you. You you keep me in the flow.

um But also. I noticed with um GPT, like... They flat out say, like, come middle of October, we're throwing out the old models. So, like... update or gtfo or you know you got to keep up with us and so i used to make an argument like oh we we should keep with the old models for a long time you get to know them um you get to know their Um... pros and their cons. But no, that's not the world we live in. We're moving in a fast-paced world where these models change every month or two. um Yeah, and you... you I think it's our podcast, but Luna's really caught on.

ah For good reason, it's cheap. especially luna max is performing just as well as like

James 08:39

Super cheap.

Frank 08:43

I forget the exact numbers, but GPT-5, 6, GPT-6, Sol at like lower medium thinking level. Luna Max is crushing it. So you're such a trendsetter, James. like luna Luna's out there. i I haven't been using it too much. I'm an idiot. I've been using Terra. What a dummy over here. but um it seems like the rest of the world's caught on to your little game cheap luna for everyone

James 09:09

Jeep Luna and like, you know, you have to. there's trade-offs right when you put on a an extra higher or max reasoning and While it is crazy fast, it is going to be... chit-chatty a lot more it's going to be burning more tokens That being said, I did a demo at NDC London where I built an interactive i've done this a bunch of times i did this at um Co-pilot day, but I built I built and deployed In real time, a full...

interactive. Oslo. day tour which brought in a bunch of coffee shops, sites and saunas.

Frank 09:48

Hmph.

James 09:48

And then you could plan your day or do a whole map. And it was like, it was an absolutely, it used Impeccable. used It used this beautiful website. I set it to Luna. if This was 5 .6 Luna at the time. Max. And I said, then you know publish the GitHub repo. set up to get a pages verify everything like click around do the thing It did all of it. It cooked the whole thing. for about six minutes. you That's including the CI, CD pipeline, six or six or seven minutes.

Frank 10:14

Okay.

James 10:16

18 cents.

Frank 10:17

Yeah.

James 10:19

just And it was it was.

Frank 10:19

That's pretty cool.

James 10:20

Gorgeous, in fact, let me say I can find it for you because I think you know if you go to my github and i i'll post the the link here for you but if you go then just type in uh pi oslo i bet that'll pop up i don't know what it was it called let me see what i call it here it's called Uh...

Frank 10:38

We can all see what baby Luna is outputting for CSS. I have CSS opinions because I have been fighting um some web development with it. And it loves to bounce between coordinate systems. Like sometimes it uses M's, sometimes relative M's, sometimes PX.

James 10:54

Oh yeah.

Frank 10:55

It seems like these models just... um They love CSS. They love generating CSS and they love being inconsistent with CSS. So I'm excited to see a little baby Luna's CSS.

James 11:05

Yeah. And this one, honestly, is just HTML, JavaScript, CSS. It's just like, here it is.

Frank 11:11

Yeah.

James 11:12

And I've run this demo a bunch of times. And sometimes it does pick a frame or something. And this was on autopilot. It asked no questions, did no things.

Frank 11:19

Hmm.

James 11:19

And obviously I had very little input, but the whole goal is like. here it is you know it's not perfect there's ah a little bit of overlapping text here and there but it has a grid as a coordinate system it has like you know all the different places you scroll down you have like different you know different beautiful pictures you know as it's you know it's it's pretty good you can copy to a link you can like filter i mean for a one shot no big deal like it's a pretty

Frank 11:43

Yeah.

James 11:44

Nice looking app. Uh,

Frank 11:46

Well. you know I know it's like it even did mobile pretty well. um It has, like you said, a little defects around mobile. but What I've really learned, and i've I've had to change my whole attitude this last week, I was getting frustrated with models creating bad UI and bad UX, as I always am. That's my perennial topic on this show. But I had a recent epiphany. I'm like, these models are never going to create the perfect UI for you. because I have opinions and all that stuff. So I just need to accept that it's going to do a rough draft first and then we'll refine that draft and then we'll refine and we'll refine and we'll refine.

So I used to get upset and I'm like, you big dummy. Why are you making that text bold? That text is obviously. secondary it doesn't matter hide that away make this other thing bold you totally forgot to do this thing i told you i used to get frustrated and then i had a moment i'm like what's happening here frank you're you're getting frustrated at a bunch of a bunch of matrix multiplies. Like it is not helping the situation by you getting emotionally involved with this model producing bad code.

So I had to do a forceful mental shift this week of. It's going to do a crappy job the first time. And I'm just going to joyfully point out all its mistakes, ideally with dictation, so I don't have to type. I'm curious how you were um interfacing with your phone if you were doing dictation or not. um And then just, you know. If it takes 10 iterations, it takes 10 iterations. That's just what it's going to be. um

James 13:19

yeah

Frank 13:21

So one-shotting is cool. It definitely makes for good tweets, one-shotting and all that stuff. But I've come around to... Using a slightly dumber model is okay because we're not doing rocket science here. A lot of time, it's just human stuff that I want to change about these. And that's why I've been interested in the smaller models like the Lunas and the Terras, even though Terra is not cool anymore. But I totally get where you're coming from of using these smaller models, because if I'm going to be iterating, I might as well get. quick iterations and not be spending a fortune on it.

James 13:53

yeah it's hard like uh opening i was doing so many promotions with soul that like soul was at some point cheaper than terra which is kind of crazy and i am a big terra fan it was like one of my is basically my default model for a long time until i sort of really started experimenting with luna extra high and and max Um, But I agree with you on the UI bit. The thing that I've been doing the most. is like taking especially in my mobile apps taking so screenshots and then i mark it up And I save this, this, and this, and this, and you know, this here.

Frank 14:18

Mm hmm.

James 14:21

And that's really helpful. And then the one thing that I've been really interested in, it's like I use Impeccable on the web. It's kind of built into the Copilot app, but you can also just install it. It's just an NPM package, Impeccable. And it has a bunch of design guidelines and things, but it also helps you put design. best practices into sort of and dot impeccable folder. And then. with the impeccable skills it's like a whole bunch of skills that are like i'm doing design let me check and let me generate it's kind of keeping itself up to date kind of like your thing that you were talking about i think you had the styles at md or design

Frank 14:53

Yeah. UI .md. Well...

James 14:55

yeah ui .md yeah similar

Frank 14:57

It's a difference, and I want to be clear. Like, I don't like... style and design and user experience being mixed together.

James 15:04

That's a good point.

Frank 15:05

You know, data flow, information flow, what inputs go where, all that, that's user experience.

James 15:05

That's a good point.

Frank 15:10

That's UX. That is, how does the app feel?

James 15:12

Yep.

Frank 15:13

Apple used to call this human interface guidelines. Then it became UI, but now we don't say UI anymore. We say UX. to separate it from, I don't care what color it is. I really don't. um you know Maybe when I'm bored, I'll work on the design and style. like That's why I don't care too much about impeccable, because to me, CSS styling is completely secondary. I would. If I could convince these models, that they all refuse to, I'd say, don't generate any CSS. You're not allowed to do any CSS, no styling. Let's work purely on information flow and all that stuff.

first and then we'll get around to styling I just want to point that out because you're you're mixing the two and it just it's a little Peeve of mine. These things are terrible at UX.

James 15:54

Yep. You are correct, Frank Krueger. Correct. you know we talked about many times on this podcast thank you for calling me out and correcting me Now let's get this merge conflict back into the correct Commit state here is that... I was just having this conversation with Heather about AI and design. And then I said, you know the thing that I talk about with Frank all the time is like the actual user experience or the taste. like There is a design taste, but there's also a flow taste as well in the user experience of how people are touching, interactive. Does it fly out? Does it do this? Does it come up from the bottom? What's the swipe interaction with? What's the button press? Where's the placement? where is that like whats how How are we moving around the application? um that's like really important even more important to be honest with you and and an AI can see a thing and it can also obviously like.

Press buttons. Bye. it's not going to have that sense of is that a satisfying movement Touching the mic.

Frank 16:52

Yeah. Yeah. um Yeah, and and and I find that that's it's the information flow and such that they tend to get wrong. All these models are honestly good enough at the design. they They love their purple to blue gradients, as I like to say. They're they're really good at generating all those gradients and all that. And it's fine. I let them play on that all day. What I find is they're just bad at. information flow and prioritizing items and all that stuff. But like I said, um i've I've learned to not even be frustrated with them anymore. I just sit back and regard that as Yeah, iterative. we'll We'll work on the information flow. one bit at a time.

James 17:30

Yeah. ah Well, these new models are out. There's obviously the the Luna, the 5 Luna, and also the... Five Seoul. They are half the price of the previous.

Frank 17:39

Yep.

James 17:41

And also I will point out. that uh The default before for Luna was 200. Okay. on the context window now it's 272 it's gone up there so it's it's matching the other ones

Frank 17:51

okay Yeah.

James 17:56

But we also, like you said, got another we have more bottles, but we did get another model from our friends over at Anthropic. Uh, ah Claude. 5 .5 I know where that.

Frank 18:05

Yeah.

James 18:06

...from Nes...

Frank 18:06

What?

James 18:07

Reclaude Opus 5 .5, is it?

Frank 18:08

Opus. Opus. Yeah, it's almost a happy birthday.

James 18:09

Opus?

Frank 18:11

like can Do you remember Opus 4-5? I think that's when we all lost our jobs because that was the first model that actually did a good enough job where that was the big transformation last year, I think, from... ah definitely for me it was like Oh, here's here's some code I have in front of me. I don't feel like typing. Please modify the code to do this. to here's an app i want just write the stupid app for me and that was the big epiphany with opus four five so i think the five five was them doing a little bit of marketing and pushing and getting out um

James 18:34

Mm hmm.

Frank 18:44

this ah 5 .5 anniversary models, how I'm thinking of it. like We had to show that we've actually improved over. opus four or five because as we all know opus five is garbage put that put that thing down delete it we don't need that thing um So hopefully 5 .5 is their swan song. Oh, that's the wrong analogy. um Their apology letter to all of us for a bad opus. Because we've all been rocking Fable. Fable's been great, but... Opus 5 has been trash. um So.

James 19:15

Interesting.

Frank 19:15

i'm I'm excited to try maybe op oh Opus 5-5 now again.

James 19:20

I definitely haven't had that experience with Opus 5. I feel like when I'm doing UI work, it does still do a... a pretty really good job of of different things. That being said...

Frank 19:28

Okay.

James 19:29

It's still really expensive, right? It's definitely the most i mean this is the cheapest opus yet and obviously it's you know

Frank 19:31

Yeah.

James 19:35

25 % cheaper than the other one, which is 50 % of Fable, but still really expensive.

Frank 19:41

Yeah.

James 19:41

model at the end of the day, especially when you're thinking of, for example, you know luna or soul which is going to come in soul is going to come in uh let me see let me look at five soul here ah Yeah, it'd be cheaper, right, than than it. Or I guess maybe it's the same price. it's very so No, it's cheaper.

Frank 20:00

it's so hard to judge because it really depends on how many tokens they use and how

James 20:04

It is.

Frank 20:04

Our pricing models are absolutely ridiculous right now.

James 20:05

Okay. if If you were to use the default contacts on six sole, it's half the price of five.

Frank 20:14

Yeah.

James 20:14

um And that's not including like. you know the cash other stuff and stuff like that but yeah so there's a bunch of things in there And I bet that being sad. Um... Like... You got to remember also Fable is still a model that doesn't have that. I mean, I think they're now doing it with 5 .1. It doesn't have that zero data retention policy, which is also of interest.

Frank 20:33

Right, yeah.

James 20:34

you're kind of doing I think it now does in 5 .1. but i do think you're right like i think when they do the the dot five they're trying to make a statement right that okay we had five four five dot one or whatever that this is like the thing right that we're doing which is important

Frank 20:46

Mm hmm.

James 20:49

But the question is like, yeah, how often do you need these? like crazy super heavy models, right? That's the real question at the end of the day.

Frank 20:54

Yeah.

James 20:55

Especially if you're going to go on a higher reasoning.

Frank 20:55

And I do like.

James 20:56

I mean, those tokens are going to be crazy.

Frank 20:59

Yeah, and I like that we've We've gotten to a place where the benchmarks and the performance graphs that people are publishing, the x-axis has become dollars now, which I really appreciate because it used to be other arbitrary things like tokens or time and all that. And in the end, it kind of doesn't matter. What we care about is dollar bills. um How much does this task cost? And I like that the benchmarks are actually judging themselves like.

What was our success rate and how much did it cost? What was our success rate and how much did it cost? And I think it's taken us a year, but we finally settled on that's the graph everyone's interested in. Because when you make that graph, that's when you see your beloved baby Luna. far on the left side, away from everyone else being super cheap, but still going about halfway up that graph. vertically. And if you want to get the other vertical distance, then you got to move to the far right of the graph where everything becomes very expensive. So there's actually a weird kind of empty zone a little bit in the middle territory. And also, everyone, please pay attention to whether that x-axis is logarithmic or linear.

A lot of people cheat and you can get really confused with how much something is costing because a lot of times they show a logarithmic one. And that makes the very expensive models look like they're only a little bit more expensive than Luna, when in fact they're 10 times, 20 times more expensive and all that.

James 22:20

yeah

Frank 22:22

But I like that that's where we've gotten.

James 22:22

Yeah.

Frank 22:23

as an industry. like here here's Here's how much it's going to cost for you to get 50 % on this benchmark.

James 22:30

Yeah, that makes sense. I still think, you know. Pick the right model for the job. You know, I still use a lot of my code one dot one flash at the highest for bug fixes specifically.

Frank 22:36

It's hard.

James 22:39

I still use a lot of Luna. I still... do some deep dives into to soul and into to opus on occasion back and forth Um,

Frank 22:48

And.

James 22:49

But it's hard, man.

Frank 22:49

Maybe I'm a weird programmer, but I still like to chat with these things. So I still judge them a lot on their ability to explain and talk. So their conversational ability. And so when I was making fun of Opus, a lot of it was because of that. It was just a really bad writer. It would just it would introduce words you don't know. It would throw away concepts that just. bad english basically it was just bad at being a chatbot which is kind of what the basic skills should be so i've i've learned if i want to actually have a good conversation i need to be talking to Soul?

roughly something like that um some of these smarter models they might win on the benchmarks

James 23:24

Yeah.

Frank 23:28

But when you're trying to have it explained to you how a... bilateral filter works it's better to sometimes to have the slightly more medium models that are just better at explaining things and speaking english basically

James 23:41

And I also want to point out.

Frank 23:42

What's up?

James 23:43

don't sleep on these model routing um things that we talked about recently like hydrofusion i just I think it's out in VS Code Insiders now. You can use it there. I'm not sure if it's in the app yet. But I'll point out that I took this wheel of wisdom. That's probably the most complicated thing in the world. But I wrote this big prompt. I talked to the the CLI. I turned on Hydrofusion. I said, hey. And I pointed at the .NET MAUI.

hybrid web view. docs and i said hey listen

Frank 24:10

Yeah.

James 24:11

here's what i want you to do i want you to do blah blah blah blah blah this whole thing with the shop blah blah blah it went and it cooked dude it cooked

Frank 24:18

Hmm.

James 24:18

Or like... 20 minutes. 25 minutes. Came back.

Frank 24:22

Yeah.

James 24:23

one shot of the whole thing, like tested it, did it. It's like, I ran it. I verified it. I put it in the mobile camera, did all this stuff, blah, blah, blah. $4. the whole thing

Frank 24:31

Awesome. yeah

James 24:32

like it was

Frank 24:33

Yeah.

James 24:34

It was... It was amazing. And it was rubber ducking and back and forth. And like, that's the thing where I'm like, I'm doing a big.

Frank 24:40

Mm hmm.

James 24:40

a bigger thing, right? Let me do something that's going to really Hmm. duke itself out between these different models and and i thought it was so cool but frank there's a new model that isn't even a new a new model i don't have access to this model but i know that the i think that the internet may have gone i think the internet may have broke

Frank 24:53

I do. Yeah.

James 24:57

because while baby luna seems to be cheap and fast apparently there's the The cheapest... And fastest thing. in the entire world.

Frank 25:06

oh

James 25:07

I believe it's...

Frank 25:07

Hmm.

James 25:08

it's Bev? Is it Bev?

Frank 25:11

Bav? Jab?

James 25:12

Yeah. Stav?

Frank 25:15

No.

James 25:14

Jeb?

Frank 25:15

Now it's Jev, which is... a terrible name, but whatever. We're all getting used to terrible names. um yeah it's it's you know you know the fastest way to do something is to not do it. Then it's done instantly. So it is a large language model.

James 25:32

Checks out.

Frank 25:35

But um unlike all the ones we've just been talking about and we use every day and have proven themselves useful to us, we're not using them because of marketing. We're using them because they're actually useful and we all like them. Jev doesn't output. text. It doesn't output code. It doesn't. Chat? So it's kind of a funny, it's still a large language model because you still input to it with unstructured natural language.

But it's output. is not unstructured natural language. Its output is... A category, basically. It's hot or not for prompts. That's what I call it. um It's a new model out there and it took the the internet by storm. My God, there was... It was the only thing you could read about there for a couple days. and it is fascinating because it's a new modality for these large language models. There's a lot of things we can do with these models, other than turn them into chatbots. We've found a lot of success with chatbots and then agents. But someone thought, hey, what if I train an LLM to do something else? What if I train it to be?

Oh, really good. classifier? What about a really good general purpose classifier? And what if we make it kind of fast and train it really well to do that? And that broke the internet, this classifier.

James 26:52

Yeah, I watched the CJ over at the Syntax FM. ah youtube and podcast break it down and the thing that he kept saying over and over again was classifier classifier classifier classifier classifier he's like you know what's good at classifying he had some really good um demos you know ah you know he he was like hey imagine you're in a word processor and you're trying to analyze if this is

Frank 27:03

Yeah.

James 27:14

you know a certain tone a certain if it's ai generator or is this and you can like write these rule structures He also used it in their podcast metadata generator, which. everyone's got one apparently and and go currently syntax .fm does too but they were saying like they take

Frank 27:24

and sometimes

James 27:28

their recording, it goes through a a transcription model, then it goes into LLM to write all this stuff. and he said the biggest thing is like what you can do with jeff is basically write rules like it can answer questions like you give it the the data and you give it questions to answer and it can basically do a yes or no or like a categorization type of thing which allows it to check check itself right so Given, hey, the LLM output this, it can then classify. yes was it accurate was it not accurate xyz and and and then and then it it it goes down and...

is classifies correctness or needs human review then they they route it in a certain way which seems interesting uh or they were using it i saw a bunch of demos for like self driverless cars and like these decision trees and all this other stuff i don't know

Frank 28:13

Oh God.

James 28:15

Um, someone did like an air airplane tracking price thing or something like that. I don't know, but I couldn't turn off the internet and I still didn't. get it, I guess, ah in general.

Frank 28:24

Yeah.

James 28:25

but Because I didn't try it. I didn't have my computer. I didn't even know. I don't even know. How do you use it? how do What do I got to do? Do I got to go buy something?

Frank 28:31

So no, it's it's actually pretty easy to use. um Like I said, its input is still natural language. So you can give it all the information in the world. So you can you know tell it a story or ask it a question or something like that. But what you need to do is specify its output schema, kind of. It's not even a schema, but basically given a question or something, some information I want, here are the potential answers for that question.

James 28:49

And...

Frank 28:56

So you can't have it like calculate a number. But if there's 20 possibilities and you wanted to pick from one of those 20 possibilities, you just enumerate those 20 possibilities.

James 29:06

Yeah.

Frank 29:07

And then it will pick one of those possibilities and give you a confidence score. But that's a trivial thing for a tablet because all these networks are probabilistic. They all have probability scores. They just don't show us the probability score.

James 29:18

Yeah.

Frank 29:20

But they all have it. That's how that's how they work. Um. And it's it's clever because let's say I give it a whole bunch of text and I have 20 questions. Well. We all know there there's basically two parts to using these LLMs. First, we do what's... annoyingly called the pre-fill, where it's taking all the context and feeding all that information into the LLM. um And then the next half in a normal one is the decoding stage where it starts generating more text and all that stuff.

Well, it turns out when you're doing that input thing, it's already doing predictions. And so it's already predicting the next token. That's how all these things work. And so Jeff is just kind of clever in that it's always. Kind of predicting just the next token is going to be one of those. answers. It's a discrete set. it's one out of two answers that's your true false kind of stuff it's one out of 20 if you say here's 20 potential answers to this question pick which one of these is best um And it can be a rating even. So like, you know, um it's just not just hot or not, it's how hot. So is it cold, medium, hot, you know, so it could be three levels like that.

And because all these models are just always outputting the next token, this one is just always outputting the next best. guess. And if you have 20 questions for it... You don't have to do that pre-fill step. 20 times. It can just kind of. fork the model at that last little token and then um very cheaply answer the 20 questions. vaguely in parallel. um so that's why everyone likes it because of that inexpensive nature of it all you're paying for is those input tokens which as we all know are kind of the cheaper ones prayer.

reasons. it's doesn't actually make that much sense but that's just how the world has evolved the markets have evolved But the nice thing is, if I have 20 questions for one set of inputs... it's pretty dang cheap because it's only really generating 20ish tokens then as output tokens and it can do all those in parallel so it's also fast So that's like at a low level how it's working. Bye. Pretty cool. I mean, it's just a clever way to use these models that we've had for a long time.

James 31:40

So I wouldn't, for example... Like... Let's say I wrote a blog post.

Frank 31:46

Mm hmm.

James 31:46

I wouldn't take this blog post and then I wouldn't say, hey, go give me five spicy titles.

Frank 31:53

No.

James 31:53

that you would come up with them. But... If I said, hey, my blog has... 50 different categories. given this text. Go. figure out. what categories it falls into, it could do that.

Frank 32:07

Yes, absolutely. Or you can... Go for the hybrid approach. Have one network, Luna, generate 20 different titles for the blog posts and then have this pick between them. Because maybe it's a bigger, smarter model. Maybe it has better judgment. Maybe it has better taste because it's been refined to have better judgment, better taste. but it's not generative in the normal sense.

James 32:27

Oh.

Frank 32:28

So you could do a hybrid trick then.

James 32:28

And.

Frank 32:30

and

James 32:30

And then you could write specific questions about how you want your titles. to be? Is it click baity is it blah blah blah is it whatever and is it can answer those yes or no and i could go into this algorithm to get there now that's kind of cool and i think that's what cj maybe was saying where he's like okay this is where the lm is generating their podcast metadata and then we use this to like

Frank 32:36

mm-hmm Yeah.

James 32:50

And then they come up with the ideas.

Frank 32:51

Shoots

James 32:52

That's really neat, actually.

Frank 32:54

Yeah, because it's not spending all of its time generating text.

James 32:54

Um,

Frank 32:58

they can make the model. Really? big and powerful and smart. And. tuned to this capability of choosing between multiple options really well. Whereas the current models are tuned with with the Right. reinforcement learning, the human feedback reinforcement learning that we all use. They're tuned to be chatbots and agents and all that. They're not tuned to make decisions necessarily. they're They're obviously making plenty of decisions, but they're not tuned to that. This one can be tuned to that exact problem. And so it's just smarter. it's It's just a more efficient use of these models. And you're guaranteed the correct schema output because it's not like we're parsing JSON in the end, even though it's all presented as JSON or not. It's really just generating. one option out of. x number of options its efficiency.

James 33:46

That's cool.

Frank 33:47

Yeah, it's... You know, when when Transformers were first invented with the paper, um attention is all you need. the LLMs had. were two big models. There was the encoder and the decoder. And then a couple years later we decided, you know, we can simplify this whole and caboodle. to just the decoder. So now that's all we use today. That's happened with GPT. So we took the transformer model. We threw out a good half of it because we said, you know what? That part's actually just extra complication. We don't actually need that. And look at all this amazing stuff we can do with just the decoder side. I like Jev because it's saying,

Yeah, that's cool too, but you know, we don't actually need to be generative all day long. We can chop off um the the recurrent part of this whole thing and just output something. an answer. And so we're just we're refining. It's still the same thing. It's still a transformer. It's still an LLM. um It's using all the same architectural principles. People have made junky versions of this already. It's the easy enough.

James 34:49

this

Frank 34:49

Any idiot can train a network to do what this is doing. Hard to make a good network, obviously. Okay. um But I like it because it's just... Another. version of the same thing. Like this architecture, this transformer architecture that was invented. has way more potential than just what we're doing with it today. Chatbots became a thing. First predictive text became a thing. That was a little boring. Then we got chatbots. Those were great. Then we turned chatbots into agents. Fantastic. And now this is just like, hey, here's another thing we can do with these LLMs. And it's efficient. We're not paying for all these output tokens.

James 35:23

Yeah, it's smart. uh yeah that makes a lot of sense have you used it for anything yet have you tried it out

Frank 35:29

I definitely got access to it, thankfully. My little brain wasn't able to come up with any great scenarios for it, so I was asking it the dumbest questions. Of course, I fed it a bunch of stock market data, and I'm like, hey. Which one of these stocks should I buy? Which one should I sell? Which one should I hold? Do not use it for that. it's a trap um but you can you know because you know buy sell um whatever hold you know those are three options you can ask any network for those um Other people were doing a lot more clever things, but that's always my stress test. I like to just see what it has to say. So I would say like. Um,

uh i would make up update i'm like here is the microsoft stock is like crashing should i buy it and it's like 33 chance you should buy it okay it's crashing but they're like yeah but it's microsoft it can't say that because it's not generative but you know that's what it's thinking it's like it knows what microsoft is so even even though i gave it false data of microsoft crashing it's still like

James 36:18

Yeah. Yeah.

Frank 36:25

Maybe you should still think about holding it.

James 36:27

Yeah.

Frank 36:28

And so it was fun to just like do those kind of little stress tests and everything of it. But I think about other things like um people had it playing Doom, which is. really hard it's a fast model but it's not that fast because it still has to take input tokens so you still have to translate um images and visual things and other media into text because it's predominantly a text model other than people are working on multimedia versions is where you can give it a picture and have an answer. you know, multiple choice questions about a picture.

um It's... There's a lot. to play with it. And again, it's just it's just another use of a transformer. We've had classifier transformers in the past. um We've all used them, especially in the early days when we were using much smaller transformers. So this is kind of honestly just revisiting an older idea of these classifiers that we all used to use and just making. Hey, for the last... three years two three years we've really learned how to scale up these llms and make them huge and intelligent and we've learned this trick um Human feedback reinforcement learning. Let's apply that to... classification also. And I don't want to be reductive by keep calling it a classifier, but that is what it is. Fundamentally, at a machine learning level, it's just a big general purpose classifier.

But the neat thing is, it's a big smart model that's been tuned to be good at classification, and that's why it's useful.

James 37:55

specialization as I like to call it.

Frank 37:57

Yeah.

James 37:58

yeah

Frank 37:58

Exactly. Yeah.

James 37:59

Yeah.

Frank 37:59

So I don't know.

James 37:59

well Very cool.

Frank 38:00

Did it deserve to take over the internet by storm? No, that's the bubble. what Generative LLMs have proven themselves.

James 38:05

Yeah. Yeah.

Frank 38:09

They're fun to chat with. um They help us in our day to day work. They help you write. weird apps while you're on vacation. they've proven themselves This one is an old idea given a modern twist. um And it took over the internet because we're in a bubble. That's fine. but it is But that shouldn't detract from it either. It is a good idea. It is a more efficient use of all these things, and it is nice to see a modern version of that.

James 38:39

and maybe because of the the cost and the very specific use case there are applications today that might be doing some things like think of wordpress right how this can maybe be able to infuse a bunch of little things and give you all these recommendations and all these little classifications for your blogs or for your youtube for example like your videos and all these other things that are doing i think i saw someone

Frank 38:51

Mm hmm.

James 38:58

they They wrote a plug-in I think for Twitter that was like a classifier of all the tweets and it would automatically hide the tweets or whatever like in real time that are maybe biased or or spicy or whatever so stuff like that but i could imagine a different use case for even some of the apps I'm talking about, the podcast generator and some of these other things that maybe I'm using an LLM like you're saying, but I'm also using this to classify.

Frank 39:03

Yeah. Right. Yeah.

James 39:19

um in correlations that might be a good trial run to do so very cool i'm make excited to give it a try

Frank 39:22

Yeah, and you brought up... Yeah, and you brought up probably the most useful use case that is probably going to be used for at least in the immediate future is if you're using something like open router and you have a choice between 12 different models to use to actually answer the question I need a classifier. Choose a model for me.

James 39:38

Yep.

Frank 39:38

you know why Why pay for an expensive model to do that classification? Might as well pay for a cheap Jev. to choose. which model should actually do the dirty work.

James 39:48

Yeah, I'll be interested if Jeb takes over the router world. So very cool.

Frank 39:51

Yeah, basically. yeah

James 39:52

all right well i'm gonna go hang out with my wife thank you so much frank for Give me the lowdown on Jeff. I did see it all over the feed, but I said by the time I get back. there'll be something new but frank's gonna tell me all about it and then i got a bunch of new models to talk about so i was like wow just like crazy never stopped man um

Frank 40:05

yeah Yeah. No.

James 40:10

Alright, well go play with all those new toys and set that Luna to max and have just give it a rip it's it's it's a good time it's it's so cheap it's just it's it's awesome

Frank 40:16

Yeah.

James 40:19

Um,

Frank 40:19

Yeah.

James 40:20

all right that's gonna do for this week's merch conflict so until next week i'm james montemagno

Frank 40:24

And I'm Frank Krueger. Thanks for watching and listening.

James 40:27

Peace.