Skip to content
Episode 490

490: SwiftUI, SwiftData, Apple Intelligence, All In VS Code??!?!

November 24th, 2025

1 hr 4 mins 58 secs

Your Hosts

About this Episode

Dive into the dynamic world of SwiftUI, SwiftData, and Apple Intelligence in this episode, where we explore how these technologies are transforming development. Join us as we discuss Frank Kruger's innovative work on the Clean Room application, which showcases the elegance of macOS UI design. Discover how AI-driven tools like Apple Intelligence can enhance your Mac's capabilities, offering powerful APIs and translation features that simplify complex tasks. We also delve into the benefits and challenges of using VS Code for Swift development, sharing insights on optimizing Swift projects and leveraging AI for content creation. Perfect for developers and tech enthusiasts, this episode provides actionable takeaways and thought-provoking discussions that will inspire your next project. Tune in to uncover the future of development and productivity!

Follow Us

⭐⭐ Review Us ⭐⭐

Machine transcription available on http://mergeconflict.fm

Support Merge Conflict

Transcript

James 00:00

Welcome back, everyone, to Merge Conflict, your weekly developer podcast, talking about all things the world of software development. Today, we're talking about not AI, but we're to sprinkle a little AI in there, Frank, because um I want to bring to attention, Frank Kruger, my inspiration in the last week was you.

Frank 00:18

Oh, oh, i i love being an inspiration.

James 00:19

ah

Frank 00:21

I think that's why you still love me on this podcast. um do Do tell. how did I inspire you, James?

James 00:29

Well, as a um fan and friend of the show once said, Frank, your UI is an inspiration. Simple, clean. I have so much to learn from you. um

Frank 00:41

ah Do I get a bell?

Read Full Transcript Hide Full Transcript

James 00:42

Do we get a

Frank 00:43

do Bell? ah Thank that That was a, I just, i it's a humble brag, but like, I'm i'm taking it. So Miguel said something nice to me and I'm i'm ringing a bell. Ding. Hi, Miguel. Thanks for saying something nice about me.

James 00:57

a? Yes, ah that was pretty good. ah You came out with an application called Clean Room not too long ago, which is about managing virtual Macs. We talked about it on this podcast. And I think I was inspired.

Frank 01:08

Hmm.

James 01:09

ah It is very clean UI, actually. I've actually been thinking a lot about clean Mac and clean Windows ui and how kind of similar they are in a way. They have this like fly out, overlapping, navigation drawer-y thing on the left-hand side, which does a bunch of things.

Frank 01:26

Right.

James 01:26

That's been in Windows as well. It's kind of been in Mac as well. But with the Mac, are we on Tahoe? Is that what we're on now Mavericks?

Frank 01:34

Sure. i keep saying Tahoe. I hope that's right. 26. Yeah. Yeah.

James 01:37

ah matt Mac OS 26.

Frank 01:40

yeah

James 01:41

well, I don't like all the rounded corners. I do like this rounded corner ah specifically. Flyout navigation thingy.

Frank 01:48

but Yeah, the sidebar.

James 01:49

the The thing, the the first, the sidebar.

Frank 01:52

Yeah.

James 01:52

I like a good sidebar. and They're inspirational. They take up room, but they can go away when you don't need them. and And your application is very nice. It's very fluid. It's it's very pretty to look at. But I think I was inspired because you're like, I wanted and application, so I built an application. and you built it pretty quick. And I said, you know what? If Frank can build an application, I bet I can build an application too.

Frank 02:16

That's mean, kind of.

James 02:15

And I've... Yeah, i yeah.

Frank 02:19

If this idiot can do it, I certainly can do it.

James 02:22

Exactly. And I can do it with AI, Frank, that is for sure.

Frank 02:26

Oh, wow.

James 02:27

up

Frank 02:27

Okay.

James 02:28

I was inspired by this app.

Frank 02:28

i You're probably right.

James 02:30

I was inspired by this app, but then also inspired by the fact that we talked about cross intelligence, your Swift intelligence. We talked about Apple intelligence, all the intelligence on device. And I thought to myself,

Frank 02:41

Yeah.

James 02:42

I have a conundrum, which is there seems to be a bunch of power in my Mac that I dislike. Not that it's powerful, but I dislike my Mac. Like, well, what if I can make really powerful software for myself that I can just like use on my Mac?

Frank 02:52

oh

James 02:55

And maybe that'll make me love my Mac more.

Frank 02:58

Oh, okay. I didn't know you were having a falling out with your Mac, but um I do totally get the feeling of there's a lot of horsepower here and all I keep doing is scrolling Twitter. I don't feel like I'm using all the horsepower that this computer provides and ah feel a little bit bad, you know, like the people working on computational theory in the 40s and the fifty s I'm like... If only they knew that I was using all that power to just scroll through endless feeds of people's banal lives. How depressed would they be? So I totally get that feeling of wanting to use the power you bought, you paid for.

Sitting there. Might as well use it.

James 03:35

Yeah, and I was inspired by this, and there's a few other inspirations here that I think is of interest to kind of talk about is there are a lot of really good APIs in macOS and iOS um that are just freely available. So we talked first about Apple Intelligence, right? We did some podcasts on it where basically you have a little medium, small LLM sitting there, ready to go, multimodal,

Frank 04:00

Small. We'll call it small. um I think back in the day I would have called it medium, but these days i we'll call it small.

James 04:03

micro, nano,

Frank 04:07

I think it's about it's two two it's about two gigabytes, two to three gigabytes. I think it's about four or five billion parameters.

James 04:19

Yeah, not bad. i like that. That's pretty cool.

Frank 04:21

Yeah. Mm-hmm.

James 04:21

And there's a bunch of other APIs built in to the device that normally you'd have to pay for. like That's something you'd to pay for, right? If you want to you know do some analysis of some text or something like that. there's also built-in translation APIs. i don't know if you knew this, but you can download translation packs. You can query translation packs, download them for the user, or like prompt for the user to open their settings and download them.

And then if you have English, or another language, and you have another language pack, you can translate it to that other language pack and that other language. So that's pretty cool.

Frank 04:52

oh ah Let me stop you there.

James 04:55

It's just built right in.

Frank 04:55

is that Is that an API separate from Apple Intelligence? Because no, I didn't actually know about that.

James 05:00

hu

Frank 05:01

And I should be using it in my own apps. um Is that a separate... Apple loves creating kits. Is that a separate kit or a framework or something?

James 05:11

Well, it's in my app that you can go and look at, um, where good question trans translation, uh, import translation.

Frank 05:18

You didn't code this. Some AI did. Yeah. translation okay so it's a whole separate module it's a whole separate thing uh that's cool do you know if that's happening on device or off device maybe it's on device because you said you have to download a pack or something huh

James 05:33

it is happening. is right on your device, Frank Kruger. And yeah, you can just you can see if things are installed, you parse it you get a bunch of translated entries, you ask it to translate them from one language to another language.

Frank 05:41

fascinating

James 05:51

And bingo bango, it does it, which is pretty cool. So you can imagine, ah let's say you had iCircuit where users were submitting um content. ah You could detect what language it's in, and then you could have a translate button to the user's local language, right? Which is pretty nifty. and then you have to pay for that. That's really convenient. um Let's say you're a podcast. Let's say you have a podcast or you make videos for the internet and you wanted to, hypothetically, you wanted to translate a transcript.

Frank 06:16

Hypothetically.

James 06:21

You could rely on Google to do it for you, or you could translate it yourself and probably get better output than, than that, because the Apple models are pretty stinking good, to be honest with you.

Frank 06:21

Hmm? Maybe. Mm-hmm.

James 06:32

Um, yeah, it's pretty good.

Frank 06:32

Is it?

James 06:34

I hadn't had the review, uh, English to Spanish. It was pretty solid, uh, even in tech, our lingo. And there's another one, which is actually like built in walking directions.

Frank 06:47

yeah it's It's honestly impressive how many APIs are there. um They've had a directions API for a long time. um It used to be contingent, like you had to show a map in order to legally use that API. But I think they even removed all those requirements and all that. I think you can just use the API now.

James 07:06

Yeah.

Frank 07:06

it's I mean, getting to your overall point, it is kind of amazing how many APIs are in the Apple SDK these days. Like the every year, it seems like they add something.

James 07:15

Yeah. And I was most intrigued, obviously, about the power of the LLM being sitting there. Right now, we could talk about power and game power and 3D rendering and ray trace shaders and all these things, right? Everyone's talking about RT shaders. Everyone loves the ray tracing.

Frank 07:30

Love um

James 07:31

Um, but I thought because I was building feedback flow and doing sentiment analysis of it, I thought it'd be really cool if I could think of an idea that would be of use to us actually and save me time. So I thought of it like this is there are several things that I do every single week for this podcast.

Frank 07:44

them.

James 07:49

Um, so the first thing that I do is I create a thumbnail And I have a PSD file. And for the last 498 episodes, I've opened up, I've opened up photop.com, opened up the PSD, found it on my computer, opened it, found a photo on the internet, drug it in, edited the number, exported to PNG g um JPEG and uploaded it.

Frank 08:00

Amazing. Mm-hmm.

James 08:13

it It takes a little bit of time, five minutes, move it around, scale it. The other thing I do is I

Frank 08:22

And I should say, I really appreciate that. um If anyone didn't know this, James does all the production work. I'm just a talent. I just show up here spout stuff. He does all the real work. So I appreciate you wanting to automate your process.

James 08:33

And then I thought, i try well, then I thought we also have transcripts. And now Zencaster pumps out a transcript automatically. Um, which is really convenient. Um, but it does it in a text file, which is fine for fireside or host or podcast hosts, but I'd love to upload those as SRT transcript, file subtitles for YouTube. So I'd love to convert that. That's just a simple conversion. But then I thought, wouldn't it be so cool if we took the transcript, which we one shot this puppy and we created chapters for YouTube.

Frank 09:08

Oh, now we're getting advanced. Now we're getting into intelligent work. Yeah, because you don't want to write an algorithm yourself for that. I think I've tried in the past, and they come out with kind of terrible.

James 09:15

No, no. Some people would say, James, just put that in copilot, put it in chat GPT and ask it to do it. And the answer is yes. But then i wouldn't be writing my own custom program for myself, Frank Gruger. And what fun is that?

Frank 09:32

ah are Are you saying you wrote an agent? Did you use an agent to write an agent? that This is sounding a lot like an agent, and I think we're we're we're an agent writing territory now.

James 09:37

I used, so I cracked up my crap, cracked open my VS code, uh, as one does.

Frank 09:47

Uh-huh.

James 09:47

And that's where they write Swift applications. Uh, the very first thing I did, wrote a Swift app.

Frank 09:50

Oh, you wrote a Swift app. I love this. This is exciting.

James 09:55

Oh no, not Swift.

Frank 09:56

You are really out of your, you're writing an app for yourself.

James 09:56

There's a, yeah.

Frank 09:58

Usually you write apps for other people and you're writing it in Swift.

James 10:01

yeah

Frank 10:02

This is kind of, did you have the crazy pills one day or is this really just my influence? I i like to think it's my influence.

James 10:09

I thought to myself, well, Could I do all of this in Maui? And the answer is yes. You have a nice, beautiful cross-intelligence API.

Frank 10:18

Mm-hmm.

James 10:21

Maybe i should have. um But what fun is it to write in the programming language that you know, in a framework that you know, in an operating system that you know, and or even in IDE that you know? That is no fun at all, Frank. That is that is not how I YOLO, sir.

Frank 10:36

I mean, i think programmers are thrill seekers. And by thrill, I mean we like to learn stuff. We're we're just nerds. We're nerd thrill seekers. And stepping outside of your box is always it's it's always enticing. You ah you got you got to test if the grass is greener over there. You don't know if it is unless you walk over and try the grass out.

James 10:56

Yeah.

Frank 10:57

that

James 10:57

And I've written, it's 100% accurate.

Frank 10:57

does that and Does that work? Yeah.

James 11:00

it's It's all accurate. The grass is so green and so brown all the same time. um And sometimes you don't have grass at all. You know, XeroScape. um And then I thought to myself, you know, I've written tons of Objective-C and tons of Java code in my early days of cross-platform mobile development, doing bridges, doing custom work, XYZ.

Frank 11:19

yeah oh

James 11:19

And I haven't written any Swift code like at all. Maybe the tiniest demo for Bridge, but I was pretty hands-off, and it's really obviously stabilized, and I never touched SwiftUI either.

Frank 11:23

oh

James 11:32

So I'm really out of the loop, and what's the best way to learn but to have Copilot write out the code for you? Yeah.

Frank 11:39

Oh, dang. Okay, you were doing so well up to that point. What's the best way to learn? Is to do. What's the second best way to learn? Have someone else do or have an AI do? Okay, um fair enough. But honestly, ah we've been saying since the beginning, a huge part of these AIs is learning. um So even if you just glance at the code, it generates once in a while. I'm curious how much you glanced at the code or if you know nothing about it, because I know you're all about the vibes.

um i I'm curious to know, did you did you go 100% or we like vibe?

James 12:14

Well, my process here um was to set myself up for a little bit of success. So I decided to not use Xcode. So that was my first thing ah in general. That may not have been super successful to create the project, but I was like, you know what?

Frank 12:28

yeah

James 12:30

I'm going to go all in. I'm going to go straight vibes, which means VS code co-pilot and some MCP servers, the Xcode build MCP obviously have Xcode.

Frank 12:38

Okay.

James 12:40

I added a few, custom instruction files just around like Swift programming best practices that I found on the internet.

Frank 12:40

Neat.

James 12:47

Um, and then.

Frank 12:48

Oh. These you just randomly googled? You just pulled in random instruction files?

James 12:53

Well, I asked it to go find like modern conventions and then do it.

Frank 12:56

Okay. Even better. Okay, great. Yeah.

James 13:00

ah So, no, there's there's just the tiniest thing. ah What i when I ended up doing was... I had to generate the the files, but mostly about like the patterns that I'm using, which is like model view model, like observable object stuff. We're in the MVVM days. It's a beautiful thing, baby. Um, little builds, little things. So nothing mostly around swift data. I use core data and then Frank told me to rewrite and swift data. um which i did so i did like best practices and the cool part here is i installed both the xcode mcp and also the apple documentation mcp and i do this because i do very similar things writing dot net code and c sharp code which have the microsoft learn mcp or the aspire mcp which allows me to go and talk to community docs or builds or things like this at the same time of like what i'm running

Frank 13:35

Fascinating.

James 13:48

So this was really helpful. The Xcode build MCP helps you scaffold things, do best practice, launch emulators, simulators, build your app, and a whole lot more. And it did give me the code structure that you see in front of you, which you don't agree with, but it's using a what workspace or something.

Frank 14:03

Don't love it.

James 14:05

Not a fan.

Frank 14:05

it It very over-engineered it. So yeah, I've actually looked at your code. I'll put all pretenses away.

James 14:11

You liked it.

Frank 14:11

um It did a funny thing. It put like your app in a package. um Fine. Like it makes it reusable, but not really because it's Swift It really only has one place to deploy, which is Apple platforms. um It's just, it's a little over-engineered. I've never seen anyone actually bother to put their whole app in a package. And then for for that reason, i you can't even open your Xcode project file. You have to open a workspace file or else it can't find that package. So I find it a tiny bit annoying those choices that it made.

James 14:44

Yeah, solution.

Frank 14:49

um But I do want to say, like I do idolize you for your use of MCP servers. It doesn't even occur to me to go find like an Apple documentation. I just assume if the network doesn't know, the network doesn't know. Whatever, I'll just move on. um it's It's pretty cool that you know about those. So while I don't agree with the project structure, the Xcode build or whatever the MCP server was that thought up this terrible project structure, it's kind of cool.

It never occurred to me to put the app in a package. And I'm like, oh. right so Okay, talk about being outside the box a little bit. And now I know what it would look like, and I know I'll never do that because it's super annoying to put your app into a package.

James 15:32

But it's there. So yeah it did take me a little bit to find the code, but once I found the code, I was good to go.

Frank 15:33

It's there.

James 15:38

um Yeah. So this was pretty neat. I decided again, i was just going to use Swift UI so I can learn it. And the cool part about the AI is that you can just ask it questions, right? So what it generated, why it generated it and all these things.

Frank 15:48

Yeah.

James 15:49

And the Apple docs are really nice too, because in plan mode, you can have it search all these docs. So what's nice is that this can get all of the samples, um, WWDC content, um, search framework symbols, search Apple docs, resolve references.

Frank 15:59

Right. Nice. nice

James 16:05

You can ask it for specific years of content. It can find similar APIs, can do a bunch of stuff, which is really, really nice and pull that into the context window. Um, so I really liked that. And that was a big, could be a big benefit and

Frank 16:17

I hope you'll link to it in the show notes because it never... apple Apple's documentation has improved over time, but it really is spread out over multiple places. And you brought up the big one. It's those WWDC videos. It's surprising how often you have to go to a WWDC video to actually understand how something works.

James 16:34

Oh yeah. Oh yeah. So I just kind of got to work. I actually started this application very small. It was a single screen app with just like tabs and it only had two things.

Frank 16:42

Okay. Mm-hmm.

James 16:45

It was like, take the text file and make it an SRT file that took about five seconds to build. And then there was the image kind of creator and there was no database or anything.

Frank 16:55

Oh, right, yeah.

James 16:57

It was just like a very simple tab. I, the image creator, it did really good. I'm assuming with core graphics. And just like, it was really seamless and just fonts, custom fonts, loading custom fonts.

Frank 17:03

Yeah.

James 17:09

I now list all of the system fonts installed on your machine instead of just a few of them.

Frank 17:09

Cool, yeah.

James 17:13

And it can see everything and it was storing everything as well, just the system preferences and it scaffold that out. I think I had that working in like 30 minutes probably. And I was like, cool, I'm good.

Frank 17:22

So that's it. Okay, so we should say again, this is a production app for you, a post-production, a production app for, we've recorded the episode, but there's so much metadata, just like releasing an app. There's so much metadata you need to generate for all this stuff.

James 17:34

Mm-hmm.

Frank 17:35

I'm curious. i feel like the the icon generation can be a real rabbit hole. um I know i personally would over-engineer that UI. You know, I would have layer support and options for all the layers and import this kind of image format and all that. Did you have a hard time restraining yourself to what that UI would be? Because this started out, you said you were inspired by my beautiful UI, so I appreciate that. um But it's hard to whittle down a UI to be succinct and single purpose and all that stuff.

So is it just like you gave very vague directions to an AI and this is the first thing it spit out and you're like, good enough? Or did you actually have pretty concrete plans on what you wanted for the icon generator?

James 18:17

Yeah. Well, the thumbnail generator, yourator yeah. The thumbnail generator that one, cause I did generate the icon for the app with AI, but no, this one was pretty simple.

Frank 18:19

Thumbnail, I should say. Yeah.

James 18:25

I said, listen, I have a background image that I'm going to give you. and then I'm going to have an overlay image.

Frank 18:32

Yeah.

James 18:34

Um, that I'm going to give you, and that's going trend. and This is important. The overlay is going to have transparent properties for it. That's important because you need to preserve it. If you be like down sample it to JPEG or something. And then I said, i want to be able to scale those backgrounds. And then I want to be able to. So it's really like two layers, so layer, layer. And then I said, I want to be able to place text on top of it for the podcast episode, usually the episode number, but it's going custom text.

And I want to be able to adjust where to put it. And that was pretty much it, to be honest with you. um And I said it was a YouTube thumbnail, so it kind of had the context that it's 16 by nine.

Frank 19:10

Good. Yeah.

James 19:15

um It decided...

Frank 19:17

Like font.

James 19:18

to, yeah, it decided to do ah few things and then I refined it. So it actually one-shotted it, I would say, overall.

Frank 19:24

Okay, cool.

James 19:26

um It did decide to go like full 4K by default, which was which was fascinating. It did do a full preview. I've worked on the preview a little bit since then. But yeah, it pretty much was like, pick a file.

Frank 19:37

Mm-hmm.

James 19:39

It imported the correct, um, like the correct, um, pretty, know, pick file, picker permissions that it needs on top of it. And then it just sort of had a little preview at a scrolly I've been refined and said, oh, can you like do a fit to, can you scale it? Auto scale the background, but blah, blah, blah. Let me do background scaling, fill fit aspect ratio, blah, blah, blah. That got some things. It decided that it really wanted to put an outline on the font and I was like, give me an option. And then it didn't do any padding.

Frank 20:11

Okay.

James 20:12

So it only gave me position, which was like top left, top right, but bottom up center. And they said, let me adjust the padding all around.

Frank 20:16

Right, okay.

James 20:18

So I refined it quite a bit, but I would say honestly, that I have that, just the thumbnail generator. Now it's not generating images, you're choosing an image, right? It's not, no, I could, I guess, figure that out.

Frank 20:30

It's a compositor. Yeah, it's a, yeah.

James 20:32

it it It is a thumbnail compositor. um But for all intents and purposes, I was opening this Photoshop file, doing layers, dragging it around, and it's just like was unruly. Whereas like all I'm doing is changing 488 to 489 different background image go. um

Frank 20:49

Yeah, cool.

James 20:50

And then I also told that I want to be able to paste from my clipboard, and it did figure out to do that, which is kind of crazy. But it kind of one-shotted it, which is amazing.

Frank 20:55

It is actually. Yeah, i mean, I'll toot Apple's horn. They have great graphics APIs and their clipboard stuff is pretty easy. So if you know what you're doing, it's not the hardest code to write. I think it's ah all that graphic stuff is pretty well built in. So I think it's always just a matter of getting a good u user interface up there and making sure you can actually generate the kind of thumbnails that you want.

uh that's what i was curious when i saw that you had the thumbnail thing i'm like oh that mean all of our thumbnails are going to look the same from now on or were you able to ah you know still add a little bit of spice and creativity you know if we need a halloween theme one or something like that will you still be able to do that i think that's always the hard part when you start automating uh production stuff is you lose the variability But I'm happy that you're happy. It sounds like it's easy to modify, too, if you do need a Halloween theme or something like that in the future.

James 21:52

Yeah, I think that that was what made it pretty easy was to go in and say, okay, I'm going to just kind of do this and see how it turns out in general. um And it kind of just, yeah, was enough and easy one shot. And I think from like that point of view, if like I got, if I wanted to stop there, then I could have easily stopped.

Frank 22:10

Great.

James 22:15

But then of course, nobody just stops there. um Because the next thing is like, okay, well, I have a bunch of podcasts and a bunch of podcast episodes. Now we really need to create an entire podcast manager at this point, right?

Frank 22:25

Yeah.

James 22:28

Like, i just don't want to open it.

Frank 22:29

Right. Okay.

James 22:29

i want to save the history, right?

Frank 22:30

This was a single view.

James 22:31

I want to.

Frank 22:32

one Okay. Right. The UI was just a one-time kind of thing. Now you want a database. You want Microsoft Access of podcasts. Okay.

James 22:41

I truly do. So um I went into plan mode. I went into refactoring mode. I went into VS code. And I went into plan mode. And I was like, here's what I want to do. want to turn it into this. I was like this. actually pointed it at the X codes app source code. So X codes allows you to manage multiple X codes.

Frank 22:57

Oh, interesting.

James 22:59

It's open source.

Frank 22:59

Yeah.

James 23:00

And it looks really nice. And it has a nice fly out.

Frank 23:03

Yeah.

James 23:04

It has a cool thing. looks better than my app, but it's someone paid attention to it. And ah it looks very nice.

Frank 23:11

I don't like the new icon.

James 23:11

And I said,

Frank 23:12

Sorry, new icon people. They changed the icon. I hate change.

James 23:15

I also do not like the icon either. That is correct. um But i was like, hey, here, here's some reference code.

Frank 23:22

Cool.

James 23:22

I said, go find samples, do this stuff for split view, XYZ on it, blah, blah, blah, blah.

Frank 23:23

Yeah.

James 23:27

blah And I said, I want podcasts, I want episodes, I want to do bubble blah, blah, blah, blah in it. And I said, I want to use core data because all I knew is core data.

Frank 23:36

Yeah. Sure.

James 23:37

And, uh, actually like that pretty much one shot at pretty good. And I said, I want the existing functionality. i want these details. I want these things. And I want the transcript and the thumbnail and like details and store that data and that information. in it, cause I don't want to like reopen the transcript every time and that's our T file and export. I want to come back. i don't want to, I want to, said, I want defaults.

Frank 23:56

Yeah.

James 23:59

I said, every podcast should have defaults. Like here's a default, like overlay default, uh, text location, all these things. So that the, the podcast is going to store information about what the default for every podcast would be as well inside of there. And, um, that went really well.

Frank 24:16

you

James 24:16

And then somebody told me to use Swift data and that's where everything went to Rye.

Frank 24:22

Oh, that's me. That's me. um But it's I just wanted to say, like you you did the classic UI thing of you you you start with basically a script. Here's some inputs and it generates some outputs. It's a function, it's a script, however, I think. And you switch to an editor where it persists state and you can come back and work on it again and that kind of stuff.

James 24:38

Yeah. Yeah.

Frank 24:43

There's a big difference. um And it takes a lot more planning to build an editor versus just a scripty thing that takes inputs and spits out outputs. I apologize. um The core data thing, it's core data is fine. And obviously it works just fine in Swift UI. The thing is, boy, I think it's just like, didn't, I'm sorry, i I'm forgetting what Visual Studio used to have database designer files. And I used to live inside those designer files because I loved making my little boxes and my little relationship arrows and all that kind of stuff.

And that's the original core data schema designer built into Xcode. But I think in this year, 2025, we've all kind of moved on to kind of code first ah data modeling. And that's why i always feel like the old-fashioned core data stuff is a little fuddy-duddy, a little old-fashioned. And it's more fun to go code first with Swift data.

James 25:41

yeah yeah so i think that this was a good lesson and learning opportunity for me like the reason that this the core data stuff worked really good it's like corporate data has been around forever but i also went into a like full plan mode so don't sleep on plan agent mode inside of like vs code because it asked me tons of questions like backup

Frank 25:42

Yeah. OK.

James 26:04

Just ask you questions upon, question and you can just say what other questions? What other questions?

Frank 26:07

Nice.

James 26:08

Like really refine it. It's my new favorite thing for anything pretty big at all or medium size or even small. Just if it's not like a, just one, like one or two pages, like I'm adding a new piece of functionality. Let me think through all these things. And this is good because it asked me questions about the data type and this thing and the relations, like what this data and that data, blah, blah, blah. And it asked me a lot about iCloud sync and this thing. do you want to set it up now? Do what do you want to set up later? What doesn't need to ready? All these other things.

So it did really, really good. And the plan mode is just like a mode that you can change in VS code and it and it generates a full markdown file and you can rev on it. I think for the Swift data, it actually one-shotted it. Like it did the thing I asked it to do, which was more simple, which was like, hey, This is in core data, convert it to Swift data and go look at documentation to figure that out and um kind of go from there. So I think that like, that was really, really interesting to like, have it go down this rabbit hole in general.

Frank 27:13

ah That's awesome because um i sometimes i'm I'm always in agent mode because I'm too lazy to click a checkbox or switch modes or anything like that. And I'm always like, don't write any code, please. Let's just talk about this and ask me lots of questions. I want to get this right. um So that's actually cool. You'll have to tell me how how you actually switch into that mode um because it's definitely not something I'm taking advantage of right now.

James 27:38

Just go from agent plan. That's it.

Frank 27:42

Okay, fine.

James 27:42

That's it. It's just a mode.

Frank 27:44

You're gonna make me click on the drop-down box?

James 27:44

Yeah.

Frank 27:46

Fine. Okay, I guess I can do that.

James 27:46

You got to click. Oh, it, you know, it could be like, ideally like, you know, Clippy comes up and it's like, Hey, do you want to do some planning? You know what I mean? And I think that would be pretty good. Well, so let me talk a little bit in detail here for a little bit and spiral about what happened next in Swift data and what led me arrive. I'll go on a little tangent here, you know, if you will.

Frank 28:08

Cool. Sounds fun.

James 28:09

So yeah. Okay. So I went down this path where I was going to go off and start to do a bunch of work basically on the Swift data and and the conversion all went super good. So I think co-pilot and use the cloud agent. It was really spectacular for like the work that it did. Now, the interesting part of this is that as soon as I implemented Swift data and started using the application, I made a little YouTube video. I'll put it a link in in there. i noticed that started to be like a little sluggish.

that's so weird. Like the flyout navigation drawer, like isn't going. I was like, what is it? Now I did add a few things on the flyout, which was like, show me the podcast, show me the episodes, add searching and filtering for them. and I was like, maybe it's a layout system. Maybe it's that. And I went into the most complex deep dive that I have ever done using every model I could think of, of like, what is this problem? I rewrote that user interface.

five times over and over and over again, maybe 10 times, to be honest with you. And it kept optimizing layouts and optimizing this and optimizing that. And I was like, no, this is not it. So I said, I need to figure out, I'm going into my debug mode, and what is the thing that's actually causing the delay? Because i noticed in my thumbnail that when I would move a slider, it was also very sluggish. Everything was sluggish. i was like, what is going on? was in my machine, I rebooted, it's the same.

And I was like, okay, remove ah I was like, i want you to remove from the core piece. Don't delete any code from the UI elements, but strip back. i want to debug the performance, remove the UI. We're going to add it back piece by piece. So the agent was like, cool, let's do this. And it did, it kind of like, it kind of like, kind of like commented out the, the main UI view, the stack of it for all intensive purposes.

And said, okay, I want you to add back the header, open the app. I was like, cool. Performance is great. Let's add back the next section. Cool. Performance is great. Let's add it the next section. And then what, like what was the thing? That that the yeah the performance did any little back and forth on, back and forth on. And it was really when I put in the episodes. So like the podcast itself was fine. but when I put in the episodes, everything really slowed down. And I went through and I went through and I was like, okay, I know that this wasn't a problem with core data it has to be something with swift data.

So I asked codex, I think to do deep research, understanding everything. I said, what is the problem? I don't remember if codex figured it out. If you figured it out, Frank, which was like, you're doing like, are you saving a bunch of big data? And was like, well, I'm saving the image, but I'm saving it. You know, base 64 encoded. I don't know. It feels like it shouldn't be there. And then I figured if it was you I think it was you, I think you looked at my code and you're like, oh yeah, I had you review the code and you said, oh, you are storing everything as like one data piece basically, instead of like ah reference or something like that. yeah What's, what is that thing called? Like the fragment?

Frank 31:32

Well, in database stuff, it's just called first normal form. Like, you're you're just putting too much data into one table, basically. So whenever you're just trying to look up, say, like, a podcast name, you're also downloading the entire transcript, the entire image file, all that stuff, because... but Like any object relational manager, it makes assumptions. And the biggest assumption it makes is if you're going to access an object, you're going to want to touch every field.

James 31:54

Yeah.

Frank 31:58

And so it has to grab every field from the database.

James 31:59

Yeah.

Frank 32:02

um i I want to take credit, but I'm pretty sure you found that on your own. And then I merely made fun of you for your bad ba database design. I'm like, oh, James, come on, come on, rookie mistake.

James 32:12

Oh, yes. the the the so so so The first route that I went down, funnily enough, I was like, well, I can tell that the Swift data is clearly a problem here. and What's unique about Swift data is that everything automatically saves and updates. so When you change something in the UI, it automatically updates in the database. It's very magical and it just works and it'll sync across devices. So I said, well, that's not how normally like I code.

Like I don't normally code that. I usually am transactional. So I was like, let me strip this back. So actually, before I have fixed it correctly, which you told me like a better way of fixing it, um what I did is I created a shadow DOM basically. I said, create.

Frank 32:50

Right.

James 32:51

You have the database, get it from the database, put it locally. and then when I hit save, save it back to the database. And that worked, by the way, which was like how I use SQLite, right?

Frank 32:58

Yeah. Yeah.

James 33:01

That just is how I use it And it works magically. It fixed my problem because it's not, it's getting the data, but it's not saving the data every single time I change one slider or change one little thing or anything like that.

Frank 33:14

Yeah, and honestly, I couldn't make fun of you too much for that solution because it's a very frank solution. It's very much what I would do, to be honest. If an API isn't working the way you want, whatever. Life's too short. I'm just going to work around it. If it's querying too much, fine. I'll i'll i'll show you a fast query. Reading something from RAM, that's the fastest query you can do. Yeah.

I think I only took exception to it because you were losing a lot of built-in features in Swift data by going that route.

James 33:43

Yeah.

Frank 33:45

When you start fighting an API, that's when things go bad. um And not in a bad way, like you said, you got it working, but it was a whole bunch of code that you had write, I don't know, two, 500, maybe even 1,000 lines of code sitting there that just essentially, if you just change things a little bit, you won't need to write those. And worse, you were missing out on basically the replication features of Swift data.

James 34:14

Yeah.

Frank 34:14

One ah one of the best features of core data, and it's something I wish I could integrate into SQLite, is it has automatic um synchronization with CloudKit if you want it. So if someone has an iOS version of your app, which this doesn't fully apply to an app that you're just running for yourself, but I get that. But it's a major feature of if you have an iOS version of your app and a Mac version of your app, the two can stay in sync.

Or if you have two Macs and you add a podcast to one, it shows up in the other. It's a really nice feature, honestly. As someone who's done a lot of data syncing in their career, I don't want to write that code. It's so much better that someone else wrote that code and I can just take advantage of it. It's not the hardest code in the world to write, but it's full of bugs and it's difficult to get right.

So I just felt bad because you were doing work you didn't need to do and you were missing out on features. um But I can't even make fun of you too much because it's a totally valid solution. You had a problem, you solved it.

James 35:15

I solved it. And then you were like, you know there's like this way to do like relation and external storage and like query it on demand and do all this stuff. And I was like, I could go read some docs, but let me just ask the docs. So so I was like, hey, i said, I want you to, I'm like, I think I know the problem. And it it was like, oh yeah, here's the problem. It was like, once I said, i think the problem is that I'm storing a bunch of jads really big. i'm like It's like, yep, that's a problem. And it's all like, I was like, okay, well, how do I fix this problem? i'm Like in your vast knowledge search, go find all of the best practices for modern, more recent search, all the, every single, and start with the newest WWDC videos, the newest samples, the newest documentation, X, Y, Z, and figure out the solution.

Frank 35:55

Nice. Yeah. Yeah.

James 35:58

And then like, it literally implemented it. Now to go from my shadow Dom to like normal Dom, it like refactored it in like 20 minutes and it like all worked and now everything works perfect. It's all flawless and have no issues pretty much at all. The UI is still kind of a hot mess because i run in high DPI, but for me, it's perfect because it's, again, it's an app for me, solo person. That's all it is. And it's great. um And like the cool part is that it actually,

Did all the Swifty data stuff. I can ask it questions about it. It implemented light theme, dark theme automatically.

Frank 36:27

Yeah.

James 36:28

It's just using system defaults. I said, don't use any custom things.

Frank 36:29

Yeah.

James 36:30

Just use the stuff that's in the box. I don't if it actually struggled a lot with toolbars. That was the biggest struggle. And I don't know if I love my use of toolbars, but it struggled a lot.

Frank 36:37

yeah Me neither.

James 36:41

Yeah.

Frank 36:42

Well, honestly, because toolbars is one of those things that keeps changing a little bit. um And the Mac support especially kept changing over time. um But it's it's funny. I just want to say, like in general, um I happen to like SwiftUI. I think it's a great way to write apps. But it gets a lot of flack for being slow. And I think a lot of that slowness can be caused when you're just, you don't fully understand how it works and how ah invalidation system works and when it decides to re-render, what work it needs to do when it does a re-render.

you know it's It's nice that Apple developed an API, or stole it from React, however you want to say it, which makes writing user interfaces easy. But then it's another step that we as programmers or as AIs need to learn that system kind of in depth so that when we we don't paint ourselves into corners like this, or if there is a performance issue, we know how to solve them and what can cause them.

And i'm I'm very happy that AIs were able to figure it out. But I would also say, like, if this was your day job, if writing SwiftUI Mac apps was your day job, you would learn these basic rules pretty fast. And I think you would you would't know exactly this mistake.

James 37:57

Yeah.

Frank 37:59

This is one of those mistakes you make once and then you learn from it and you generally don't make it again.

James 38:05

Yeah. And I think now what's the cool part about this is like, i actually want to go learn about Swift data and I want to learn about Swift UI. want to learn these things more and I can do that by asking questions, but I can also just figure out what WWDC videos I should go watch as well for best practices or other conferences as well. Um, on it, which I think is neat.

Frank 38:23

And what I like, I got to see a fun behind the scenes version of all this too. You were writing blog entries too about all this experience. And I got to read these blog entries because the blog entries kept changing over time. but As you learned, like it was fun to like track your learning through all these blog entries.

James 38:38

Yeah.

Frank 38:43

I was having fun sitting on the sidelines.

James 38:45

ah i So here, if you go into the repo, which opens the show notes, it's called like app podcast assistant. But when I was doing stuff, Along each way, I was like, okay, I have this big chat and I did a lot and I could document stuff and it loves to document stuff, but I was like, i want to like go on a journey of what we did here.

Frank 39:01

Yeah. yeah

James 39:09

And the best way of doing that is like a written blog. So I put together like these big prompts, which is like write blog for this thing. So every time that I did something of major difference, I asked it to write a blog. So there's a blog folder and there's, um, the one was three swift data performance lessons from optimizing podcast assistant. So actually the first time I, did actual optimizations, which was like loading lists and doing stuff. I was like, okay, you fix a bunch of stuff, go write a blog post about it. And I had, it I said, go search Montemagno.com.

Look at all my blogs, use my, my, how I talk about stuff.

Frank 39:43

Yeah, sure.

James 39:45

Um, and then I did one, which was like refactoring from swift data chaos to POCO architecture, architecture, which is this. And then, ah there was one, the last one, which was from POCO hybrid back to pure swift data lessons from a friend. I'll read it a little bit. It says, ah week a few weeks ago, not that's not accurate. It was like two days ago. I wrote about, and it links to this thing, TLDR Swift data's query felt unpredictable.

Frank 40:08

Yeah.

James 40:13

Relationships were faulting at random times and the UI wasn't updating when it should. So I built an intermediate POCO, plain old class object layer with a podcast library store bridge to make everything just work. And it did work. Buttersmooth performance, predictable updates, about 1,500 lines of carefully crafted bridge code synchronizing POCOs and Swift data models. I was proud of it. Then I showed it to a friend who's been building Swift data apps since the framework was launched. Their reaction? Why did you do all that? Just use at query directly and when in doubt, create a new view.

Swift data reactively works great when you follow the patterns correctly. I copied and pasted like a text message you sent me. And then it says, I pushed back, but the UI updates were inconsistent. Relationships were faulting. Selection binding didn't work. They smiled. Did you add dot tag to your list rows? Did you make your models hashable? Did you use external story for haver ah heavy data? I had not. So I spent the weekend complete a weekend, like 20 minutes, um, going back to pure Swift.

it So it goes on and on. It's like a whole journey of our conversation.

Frank 41:18

That's funny. It's so dramatic.

James 41:20

So dramatic.

Frank 41:21

Put a few words in my mouth there, but that's fine.

James 41:22

Yeah.

Frank 41:23

That's fine.

James 41:24

Yeah.

Frank 41:25

I'll i'll be the, yeah. ah You know, I love this. um ah Two things. made me think, A, this is genius. um I really wish I did this for my other apps because they always taught us in engineering college to like take notes.

James 41:36

It's fun.

Frank 41:40

I don't take notes.

James 41:40

Yeah.

Frank 41:41

I've never have. ah I've done very large engineering projects and I've never taken notes. ah this is genius just be like okay I've got this long chat history here please summarize this into a blog so that like you can relearn it later and or we can have these fun moments together like this but secondarily it's making me think like geez why aren't our chats just saved as like a JSON or XML file in our repository like it it should be there Like, what why is it in some mysterious cache directory in the IDE?

James 41:57

yeah

Frank 42:14

um I feel like all these these chats are important. I think feel like they really should just be stored in the repo itself.

James 42:18

Yeah. Yeah, I think so. I mean, that's kind of like my vibe. handsome man. has been vibing some stuff recently. And he was like, I have a ongoing, like, he's like, I have a doc that's like, every time I learn something or change something, I'm like, document that thing. But like, he's like, you just go back in history of like, he can see every single, like kind of significant change and why, and like do this stuff on it.

Frank 42:42

yeah.

James 42:45

And i was like, that's kind of cool. But I don't, I don't know how long that lasts. Like, that's my thing is like that lasts for like, I don't know, A few days and I'm like, I got too many markdowns. Right. So I was thinking like significant things in my life of like, okay, how do I want to go and do this thing basically?

Frank 42:57

sure

James 43:02

But I think it's fun. It's a fun artifact. I think that blog is way more fancier than documentation, but if you look at feedback flow, it's like, here's all the documentation.

Frank 43:08

yeah

James 43:11

A good pro tip is, is like, um, i was talking to Toby on the getup team and he's like, oh, I have a cron job that runs that fire off an agent. That's like, Hey, look at all the changes from the day and then go update the documentation to make sure it's like up to date and recent and blah, blah, blah. Cause the documentation can get out of date.

Frank 43:28

Smart.

James 43:30

So that's the other part too.

Frank 43:30

Yeah.

James 43:31

now

Frank 43:32

Yeah. um I don't know. I'm sticking with it, though. I love your blog idea, and I really want the chat history just in the repo itself.

James 43:40

Yeah. Yeah. But yeah, put in there. There's nothing stopping you either. Like here it is.

Frank 43:44

Yeah.

James 43:44

And like the cool part is like, you can actually right click and BS code on the chat and just say copy all. And it'll just give you like the, the, the markdown just gives it to you.

Frank 43:52

Oh, OK.

James 43:53

So now right click save.

Frank 43:53

Yeah.

James 43:55

That'd be cool. So.

Frank 43:57

Yeah, or it could just be automatic. OK, so where are we? We have um built two fun features that you need for production.

James 44:06

Mm-hmm. Ah.

Frank 44:08

Then you took the big move. You went from a scripty app to an editor app. You went through a few arc ah architectural changes in that app because it was slow because of you, your bad data modeling, James.

James 44:21

Yes. Oh,

Frank 44:24

And now what? You're still working on this thing because you've you've already spent like a few hours on it, right Because the AI is running all the code.

James 44:31

yeah, pretty much. Yeah. I say at this point, I'd only really spent a few hours on it, to be honest with you. And then at this point it was, um, AI time, baby. It was eight to high time. And, um,

Frank 44:42

Right. Apple Intelligence time? Is that what you're saying?

James 44:44

I Apple intelligence, baby. I said, Hey, I got any, I got, I got three ideas.

Frank 44:47

Heck yeah.

James 44:48

I want take a transcript. Give me, give me podcast name ideas. And like, and then i was like,

Frank 44:54

ah Okay, okay. Oh, a show title generator. That's great, because we used to spend like 10 minutes after every episode arguing about that.

James 44:58

yeah.

Frank 45:01

I remember those days.

James 45:02

Yep.

Frank 45:02

Those were fun. Yeah.

James 45:03

Yep. I got the transcript. the The cool part is once you have the transcript,

Frank 45:08

Yeah.

James 45:09

that's, that's the wealth of data, right? And, uh, you can do stuff with it.

Frank 45:11

Yeah.

James 45:13

So I said, I want to be able to do the transcript. I want to be able to, um, generate a description, give me a short, medium, long description. so i have to write a description like an animal every single time. Um, I also said, I also, i also generate social media posts. So was like, generate social media posts for me.

Frank 45:32

Okay.

James 45:32

Cause why not? Uh, and then the more challenging one is i would like chapter markers on this puppy now.

Frank 45:38

Okay, okay. so So let me ask you, ah the name generator, i'm a lu well I'm a little worried about its creativity, but that's definitely an easy problem for it.

James 45:48

Hmm.

Frank 45:50

The summarize and the short, medium, long, it has to be pretty good at that. what's What's your experience so far? Are you actually running Apple Intelligence? Is it actually doing these things? Is it working?

James 46:00

All, and I said, I only want to use apple intelligence on device only let's go. And as I said find the document, was like, find it, do a thing.

Frank 46:04

Mm-hmm. Yeah.

James 46:08

um no, I have a services folder and I have a AI folder inside of it. And there's a, a title generation, service and it's.

Frank 46:15

Cool.

James 46:16

ah To do that is 60 lines of code.

Frank 46:19

Yeah, it's kind of fun.

James 46:20

And there most of it is documentation, ah comments. And ah for some reason, ah this gentleman, Mark, keeps showing up in my source code. i don't know what's going on there.

Frank 46:32

Oh, good old Mark.

James 46:33

but

Frank 46:34

Yeah.

James 46:34

oh

Frank 46:35

He's a god that all Swift developers pray to. And you just have to say his name every 40 or 50 lines in your code or else that your code won't compile. His name's Mark. Yeah.

James 46:47

Yeah. So Mark is there and Mark is telling us a lot about the code, but, um, there is a language model session. You give it instructions. I said, you are a creative pod. I didn't tell it. This says you are a creative podcast producer. That's me who writes engaging, concise episode titles.

Frank 47:02

Aww.

James 47:05

Um, and then the prompt is you are generating titles for a podcast episode based on this transcript, analyze a trend.

Frank 47:10

See, I thought you would have said clickbait. You're like, get trick people into playing this episode. Come on, man. Be dirty.

James 47:18

Yep. Oh, no. Well, it says, analyze a transcript, identify the topics that are discussed the most, focus primarily on the main topics. The title should reflect that. Create five, generate five creative, concise titles for this podcast. Keep the titles under 10 words each. Keep them, make them engaging, descriptive, and SEO friendly. Now I could adjust that and make it a little bit more like fun in this, but I would say the episode titles are pretty good. They're pretty solid.

Frank 47:44

Okay, cool.

James 47:44

Yeah.

Frank 47:46

Yeah, that was one of the prompt engineering.

James 47:46

Yeah.

Frank 47:48

Like, you got to start somewhere and just see if it's generating. I'm very happy to and put in, make sure every episode title has a pun in it, because then life could have just gone really bad.

James 47:56

Yeah.

Frank 47:59

um so So would you say your first system prompt was kind of good enough? You were pretty happy with the results there?

James 48:05

Easy, solid one shot done so simple to do.

Frank 48:07

Nice. Cool. Yeah.

James 48:10

Yeah. And then.

Frank 48:10

And I should say, sorry, what one more thing.

James 48:10

Hmm.

Frank 48:12

um Apple Intelligence does have a few parameters you can choose. um in In all these LLMs, there's a thing called temperature, which is how much randomness it applies when it's sampling from the probability set that it generates. And this isn't perfect correlation, but the more random you make it, the more creative it gets, obviously, because it's picking things that aren't very predictable or probable in this case.

ah So i'm that's another fun thing. yeah You should mostly focus on the system prompt, asking exactly what you want it to do. But it is fun to play with the temperature settings. And I find a lot of people mess around with a prompt all day, but they don't mess with the temperature nearly as much as they should be.

James 48:59

Is that in the session or in the prompt?

Frank 49:02

it's in the options for the session. When you create a session, um, you receive an options thing and even things like chat GPT and all those, they all have a temperature setting.

James 49:05

Ah. Yep.

Frank 49:12

No one touches it. Everyone uses the default of 0.7, 0.8 or 0.9.

James 49:16

Interesting. I like that.

Frank 49:18

Um, no one does one, uh, uh, no randomness because then the LLMs tend to break. But, uh, It's just a good thing, especially when you want something creative out of it. Bump up that temperature. Let it a little crazy. Mm-hmm. let let it go a little crazy

James 49:37

Yeah, i like that. I should do that on that one. That sounds really good. because And the same with the description too, because it's pretty much exactly the same. The social posts are pretty much the same. um i mean, it it just generated a different prompt based on like what the different things are and it just output them for different things, different, you know, social media posts. It it does a different one for different platforms, different lengths. It does one for Twitter, LinkedIn and threads.

different things and it outputs them. It's really cool. It just outputs, you know, the actual POCO as well, you know, so that's kind of nice. um I like that. And then the biggest problem was the chapter generation, know, because Frank, um the AI is actually pretty clever.

Frank 50:16

Yeah, okay.

James 50:19

So for those first three, it decided, well, ah this transcript could be pretty big. So what if I just cut off the first, you know

Frank 50:28

Oh, context links. The nemesis of all fun in AI. Context links.

James 50:34

Yeah, to because could for like our podcast, you actually probably don't need the entire podcast to actually, you know, the transcript to do it.

Frank 50:35

Yeah.

James 50:43

You probably actually only need like a little bit of it. So it was pretty clever. And cut it off. And then I ran this problem, which was yeah, the context window. I was like, here's the entire transcript. Now you can in the other ones, you don't need the time codes, but in this one, you need the time codes. It's important. I need you to basically find five to 10 important segments in this puppy and then do this. And it was just like, no, no, no.

And the problem was it's too big. Our podcast that I was generating off was even like 30 or 45 minutes. It's like, Nope, can't do it.

Frank 51:11

Especially recently, we've been doing one hour podcasts. We used to say our tight 30 minute podcast.

James 51:14

Yeah.

Frank 51:16

But yeah, we've we've been we've been talking a lot. And um actually, I remember when Apple Intelligence first came out, i they don't really document what the context window length is. But in one of the WWDC videos, because that's where all the good information is, one of the engineers actually said it, at least for this version one, it's about 4K, 4,000 token ah context window Which if you've never done LN program, that's your inputs to it, system prompt, ah history, everything, and it's the output.

So that all has to fit within 4,000 tokens. Ooh.

James 51:52

And yes, this became the next biggest problem that you and I decided to solve, which was the good old sliding window approach. And it' did this one makes sense.

Frank 52:00

oh

James 52:00

I kind of understood this, which was let's chunk it together, summarize those chunks, give it time codes, basically shrink it down. The whole idea was how do we make this as small as humanly possible? And the idea is if we talk for five minutes about this AI feature, That is like one time code. It's really, the only thing that's important is like the few time chunks that we're talking about this thing.

So instead of it being, you know, um you know, 10,000 characters, it can be 200 characters. It can just like summarize the thing smaller than being the entire transcript basically.

Frank 52:36

Yeah, um especially back in the day, like we used to have 512 token context windows. So like in the early days of LLMs, you got really good at tricks for compressing things down. um So if you had a long chat history, you'd say like, summarize this entire chat history into four sentences. And then you would use that as your reference.

James 52:57

yeah

Frank 52:59

And it's just a good trick. I mean, even if you have the ridiculous 1 million token context windows that a lot of these networks feature these days, you don't want to actually pay for those million tokens. So it still behooves you, especially if you're if you're doing any LLM development.

James 53:13

yeah

Frank 53:17

Yeah. Come up with a data summarization strategy pretty early on, um especially if you're working on editors where users can keep adding data and adding data and adding data. ah You can't just count on that always fitting into your context. So if you're going to keep long chat histories or you're going to generate long stories, you're going to fit large data models into it you need to come up with summarization strategies.

And I think I pitched to you just the most obvious basic one is just, yeah, chunk up the episode and ask it to summarize those chunks and then paste the chunks back together. And that's the new context. It's kind of idiot proof. I mean, there's other ways. There's a reg and all that kind of stuff, but i like the chunking and summarize the chunking approach. It's conceptually pretty simple and it generally works well in practice.

James 54:10

And the big difference too, is that there's a sliding window. So you're actually doing overlapping windows. So you are kind of ah doing a chunk and then the next chunk will have a little bit of the previous chunk and a little bit of the next chunk and then back, you know, kind of slide this window across basically over it and you're, you're overlapping these windows. Now I seem to not be able to get it to work really good. However, you decided to build a little application that works perfect. And then I copied all of your code into my application. actually copilot did and that ended up working well. So how did you actually solve it? Cause the mine wasn't perfect. Mine was maybe.

shoehorned into an existing thing, basically.

Frank 54:44

Well, if I can toot my own horn, it's because I actually coded mine and I didn't just ask an AI to do it. No, it's fine.

James 54:50

Ah, interesting.

Frank 54:51

It's fine. I'm just messing. um I think it's just a matter of it's it was faster. I had an idea of how I wanted this thing to work. And ah the so the the big things I wanted to happen were um the sliding windows and summaries, the sliding window with overlap so that it had some context from before where the window was starting. So it wouldn't just... The problem is context with all these things. If we just jump into you making a joke, it's not going to fully we understand that. So you need some kind of overlap so it has some context to build off of.

But the second big thing I really wanted you to take advantage of was structured inputs and structured outputs. Because I think that those are some of the most powerful ways to use these LLMs. So although they can all they all present themselves as these like chat interfaces where it's just freeform text in, freeform text out, the proper way as programmers to use these things is structured data in, structured data out.

So instead of saying, like, give me a transcript where the first line is a timecode, the second line is the speaker, the third line is the actual summary, say, no, here's an object. It has three fields. One is a timecode, one is the speaker, and one is a summary. And behind the scenes, what's happening is it's creating a little JSON schema. And it's telling these networks are now trained with an understanding of JSON schema. um And it'll do that.

And the really nice thing is when it's outputting stuff, it's guaranteed to generate an output that matches that schema. So then there's no ambiguity on exactly what data you want out of the thing. There's no guarantee it's going to generate good data for you, but it'll at least be structured in the way that you want.

James 56:39

Yeah. There is data.

Frank 56:43

Yeah.

James 56:43

Yeah.

Frank 56:44

And it's also a second opportunity beyond the system prompt to give it a little bit of metadata, describe what that data is. So you can say like... um ah This structure is a summarized part of a transcript. It's not just a chunk of a transcript. It's not just a section of a transcript. It is a summary of the transcript um with a starting timecode and summary text. So you're not saying timecode text. You're saying and starting timecode, summary text.

James 57:14

Yeah. Yeah.

Frank 57:15

And those those little clues matter a lot to these LLMs.

James 57:17

yeah

Frank 57:18

They pick up on little clues like that. The other advantage that structured data gives you is that you can give it properly formatted examples in a language it understands, which is JSON. All these things are just trained on JSON. And an important part of system prompts these days is giving examples. Because no one trains their networks anymore. Back in the day, we would fine tune this network so it knew exactly what kind of data would be receiving and what kind of data you expect out of it. But nowadays, we are all using that capital G general part of it and relying on it, just reading our minds and knowing exactly what data we want out.

But instead of that, you could just say, here's an example. If you see this kind of data, output this. If you see this kind of data, output that. And those really help the LLMs beyond just the system prompt. So I implemented the algorithm as I was describing it to you. I forced it to do structured data. with metadata, and so descriptions beyond the system prompt, and I gave it examples. And kind of I one-shotted it as a human. It kind of just worked the first time i wrote it, and I was pretty happy with that.

James 58:36

Yeah.

Frank 58:38

I meant to spend time fine-tuning the system prompt, but ah with those few tricks, it was already performing pretty well, in my opinion.

James 58:46

Yeah, and I had it pretty close, and I feel like this was the opportunity where like it was time for me to crack open a book, basically, and do some stuff. But you are my book. And then cracked open Frank Kruger, and I gave him a problem, and he wanted to solve it, and he did, which is good. And then it was really neat, because then I then told my coding agent. I literally went in to get a copilot coding agent. I was like, hey, go take this Repo, replace this thing and then update it. I was like, cool, like I can do that.

yeah these are They're like, whoa. I was i was like, oh go analyze this' like oh this. This approach is better. It's like, oh, thanks.

Frank 59:17

Oh good, it said that?

James 59:17

<unk> it' say oh so that this This approach is great.

Frank 59:18

It gave me props? Thanks buddy.

James 59:20

Whoever wrote this code is a genius. It's like, okay, great. um Anyways, it's really cool. it it even

Frank 59:25

Aw, thanks, AI.

James 59:25

idiot it It even copied your bad ah you vast debug shrinker like UI line by line. It's not great, but it did, and which is hilarious.

Frank 59:36

No, it's not.

James 59:38

It's very bad, but it's cool. It works, and and I worked on it. So now it's in there. The debug's in there. I'm now going to make it so I can save the sliding window so I can regenerate things. don't it every time. it's very The thing with Apple Intelligence is, actually, if you have a small context when it's pretty fast to generate some things, but to generate those sliding windows, it takes forever.

Frank 59:54

Yeah. Yeah. does.

James 59:57

It's like really slow because it's doing it. It's doing one at a time. But now don't know if you could paralyze it.

Frank 1:00:03

yeah No, because the whole point is you're using that GPU to its max.

James 1:00:06

No. Yeah.

Frank 1:00:10

So while theoretically the operating system technically allows it, you could paralyze them. In practice, there's there's one GPU there and it has this amount of memory.

James 1:00:20

yeah

Frank 1:00:22

There's a reason that context window is 4K. It's memory. It's not computation. It's nothing else. It's just memory.

James 1:00:29

Yeah.

Frank 1:00:30

And so you will just, that's always been the problem with running these LLMs on your machine is memory. It's so funny, in the early days of AI, we all thought we were going to be compute bound.

James 1:00:36

Memory.

Frank 1:00:40

Nope, turns out we're all memory bound these days.

James 1:00:43

Well, and how I...

Frank 1:00:43

And so, yeah, you can't run them in parallel.

James 1:00:46

Yeah, and how I run this out, by the way, people are interested. I almost run most of the time just from VS Code because I can just have it like build and do stuff and run the tests and everything from the Xcode MCP server, the build server. But I actually prefer to have Xcode open and then also run it from there too because you get the profiler and the debugger and everything like that, and that's really nifty.

Frank 1:01:07

Yeah. Sure.

James 1:01:08

especially to have it like output console logs and do things like that just like a normal id would do i think there's probably ways to maybe do it from fiesco but i'm not really sure maybe there's not it's got to be i assume

Frank 1:01:11

yeah sure No, if if you just run a Mac app from VS Code, it'll spit out all the debug info. And you can even have it bind to LLDB and get debugging information while it's running.

James 1:01:28

oh ah

Frank 1:01:30

You just got to set that stuff up in your VS Code launch, launcher task, launch.

James 1:01:34

yes

Frank 1:01:36

Yeah.

James 1:01:36

Got it. Okay. Yeah.

Frank 1:01:37

Yeah.

James 1:01:37

Cause what I did in VS code is I set up a new profile. So I'd recommend this. If you haven't messed around with profiles, I have my default profile, which is like my.NET development.

Frank 1:01:41

Hmm.

James 1:01:45

I have a swift and I have a new JS web dev one. The cool part about this, what the profiles is that.

Frank 1:01:49

Cool.

James 1:01:51

I copy over all of my settings for my default, and then i remove a bunch of extensions and then I install ones that I need.

Frank 1:01:57

Cool.

James 1:01:58

So for this one, for example, I removed all my.NET stuff. I don't need it for swift development. And you know, unless I was doing a backend or something with it. And then I installed the Swift language support, which is the official extension from the Swift team. And that also includes the LLDB dApp, which is the debugger built right in.

Frank 1:02:12

yeah

James 1:02:16

and I guess I just need to bind that up basically. Um, so that's cool. I have like a little tiny itty bitty, you know, um, profile and that every time, if you open a folder, you pick a profile, when you open that folder again, it'll pick that profile automatically.

Frank 1:02:22

Yeah. Nice.

James 1:02:31

So really cool.

Frank 1:02:32

Yeah, we we keep talking about profiles on this podcast and I still don't use them enough. I think tonight you might have finally broken me. Like, I'm like, I got to do that. You know, the problem is for me, I have a bunch of, I have so many extensions because I do all sorts of weird stuff. So the weirdest thing for me is when I like open some C sharp code and the C++ CMake thing starts complaining.

James 1:02:46

Yeah.

Frank 1:02:53

I'm like, go away CMake.

James 1:02:53

Yeah.

Frank 1:02:56

I'm just realizing I should really use profiles.

James 1:02:56

Well, totally. And you could have almost a profile.

Frank 1:02:58

Yeah.

James 1:02:59

could have a profile for every app if you wanted to. This is the profile for iCircuit, right?

Frank 1:03:02

Yeah.

James 1:03:03

then it just loads this stuff up. Anyways, okay. We're at an hour. i need to also wait in your car, another podcast.

Frank 1:03:07

We gotta to summarize this transcript, baby.

James 1:03:08

We're on a holiday. and Uh, now I'm going to try, well, I'm implementing a new feature right now as we do, as we talk. Well, everyone can check out the source code. I'm going to definitely try to check in all of my changes into main so can check out because it is working fairly well.

Frank 1:03:22

Okay.

James 1:03:24

But it's an app for me. You can at least check out the code. You can critique it and just let me know what you think about it. And then, yeah, ah it turned out pretty good. I'm pretty happy with it in general. I think it probably would just been easier for me to just call Microsoft Foundry or something like that and call an open AI like nano, you know, thing with or whatever. But this is much more fun.

So think we had whole podcast.

Frank 1:03:46

And you learned a lot. fun Fun and you learned.

James 1:03:47

Yeah, it good.

Frank 1:03:49

Yeah.

James 1:03:49

I learned a lot and I have a little Mac app and that's pretty cool. And it's the same thing. I was, you know, um, uh, there's this new thing about those, like this ring light app or whatever that they're putting into Mac OS or whatever, and then handsome and vibe code it for windows and WPF. And I've, I've coded that little typer thing and wind forms. It's like, apps on demand, man. That's like my jam.

Frank 1:04:09

Yeah.

James 1:04:09

Just like, and this one's actually pretty, it's a real app. So, and you, cause you kept saying, but you can sync it.

Frank 1:04:13

It is.

James 1:04:14

You can do stuff in iCloud. I'm like, Frank, this is an app for just me. Like nobody else is going to get this. Like no one cares about this app by me. They're like, Oh, you could put it on. like, no, I'm not gonna find the app store. Cause nobody wants this app. Like, you know,

Frank 1:04:25

I gotta be honest, though, now that you have the transcript summarizer and the the description generator, feel like you could put this one on the App Store. I really do feel like, although you wrote it for yourself, I think you would get some downloads.

James 1:04:40

I could do that transcript down. And then also the translator part of it too, I think would be neat.

Frank 1:04:44

Yeah, you have some cool features, honestly.

James 1:04:44

People would like that, man. Wow. Well, that made my day. I was going to this week's merge conflict. We'll end on that super high note. So until next time, I'm James Montemagno.

Frank 1:04:54

And I'm Frank Kruger. Thanks for watching and listening.

James 1:04:57

Peace.

Download Transcript

Episode Links

Episode Comments