Alan Richardson speaks to Jerimiah Ham about Software Testing with AI and his new Free AI enabled Mobile Automating and Testing Tool Quern.
Join Alan Richardson and guest Jerimiah Ham, developer of Quern, as they discuss testing tools, AI in software testing, and Jermiah’s new AI mobile tool designed with AI Agents as primary user.
Quern, an open-source, local API and MCP server for working with iOS and Android devices, simulators, and emulators. Quern gives an AI agent access to UI controls, screenshots, logs, and network traffic, turning mobile testing from a sequence of manual setup tasks into a much more observable workflow.
Jeremiah describes the approach as pairing with AI rather than handing over control, using the agent to augment human observation while keeping the tester in the driver’s seat. The episode also looks at turning exploratory interactions into repeatable scripts, the limits of current models, and why mobile testing may be entering a very different kind of workflow.
Quern - Jerimiah Ham
- Jerimiah on Linkedin
- Quern Web Site - quern.dev
- Jerimiah’s Medium Blog
- Quern Blog post
- Jerimiah on Twitter
- Github
Summary
In this episode, Jerimiah Ham introduces us to Quern, a breakthrough tool he developed to streamline mobile automated testing. Built originally to solve issues Jerimiah faced in his projects, Quern has evolved into an AI-integrated platform facilitating smoother interactions with mobile devices, both iOS and Android. The conversation journeys through the motivations behind Quern’s development, emphasizing its open-source nature and potential commercial pathways. Jerimiah describes how Quern empowers developers by automating tedious tasks and enhancing real-time insights into app performance.
He candidly shares his experiences and challenges working with AI-driven development, specifically using Claude as the primary coding agent. The discussion covers the technical nitty-gritty of the tool’s features such as UI automation, proxy integration, and knowledge base creation. This episode is a deep dive into Quern and a testament to how AI can revolutionize testing and development workflows, offering solutions that were once unimaginable.
Transcript
Alan Richardson [00:00:00]: And it has started recording, so we’re on. And welcome to another episode of the Evil Tester Show, and we have a special guest, Jeremiah Hamm, and we are going to be talking about the Quern— is that how you pronounce it?
Jerimiah Ham [00:00:15]: Yes.
Alan Richardson [00:00:16]: Quern.dev AI superpowered mobile automated testing debug and many other things tool. So hello, Jeremiah.
Jerimiah Ham [00:00:27]: Hello, thank you for having me on.
Alan Richardson [00:00:29]: That’s cool. I’m— we’ve been trying to do this for ages and it’s all my fault that it hasn’t happened for ages. So it’s good to have you on. So I guess we should start with what is Quern?
Jerimiah Ham [00:00:42]: Yeah, Quern is, well, technically it’s an API server that has an MCP interface and it is a tool that your coding agent such as Cloud Code can use to be able to interface with mobile devices, simulators, emulators. It supports both iOS and Android right now. It is macOS only. Kind of that’s a prereq for any iOS development. So I know that unless you are really truly Android-only mobile developer, you’re most likely using a Mac.
Alan Richardson [00:01:24]: When I look at the tool, it’s the kind of thing that makes me think, I wish I was doing mobile app development and automation again. And then I remember what that was like. And then I think, well, I’m glad I’m not doing that again. But yeah, so did you build this for work?
Jerimiah Ham [00:01:42]: I built it actually to To solve some problems I was having with a side project, I kind of— it’s not an official, like, sanctioned fork, I guess, but I use a Mastodon client called MetaText. And it had gotten quite long in the tooth, shall we say. It hadn’t been updated in years. And there were API changes in Mastodon and GoToSocial, which specifically my server is. I run a single instance Mastodon server. in my home lab. So, yes, I’m that kind of nerd. And so, I started to use, you know, kind of use Cloud Code to see if I could modernize the app, get it to compile in modern Xcode and all that stuff.
Jerimiah Ham [00:02:31]: And that worked pretty well. I was pretty happy with that. But we started— I started hitting, you know, this sort of wall where, wow, I realized I wasn’t just fixing it up for myself in the sense of like, I’ll just change a few words and then it’ll be good. The tester in me was like, okay, now am I just creating more off-the-job testing work for myself? And I realized I was, which, you know, that there’s trade-offs for that. But I’d also been using Claude for quite a few months at that point, and I’ve been using it on non-mobile projects, which work really, really well. There’s a lot of observability. If you’re writing, like, specifically a Java API server, you can run that locally, and all the logs are right there, and you can see what the network’s doing, and you can— ask Claude to start and stop the server and tell it what port to use. It’s just, you know, it becomes a very— like managing the app and how it’s running and getting data from it, it becomes the least of your worries, whereas that used to be, you know, a manual process.
Jerimiah Ham [00:03:48]: You’re copying, pasting, you’re in the terminal, you’re doing all this by hand. So, I found myself when I was working with the mobile device, falling into that same pattern. I had to do everything myself. It was all manual. Like, launch the simulator, make a build in Xcode, install it, press all the buttons to make, you know, go through that whatever I just fixed. If I needed logs, I needed to actually launch the app in Xcode and look at that little screen in the Xcode IDE because that’s the only place that logs are available. And again, the network calls weren’t visible unless you run a proxy like ProxyMan or Charles or something like that. Anyway, I just realized that there was a huge difference in the developer and test experience when using mobile devices versus developing on just, you know, on something that can just run on a laptop or a server.
Jerimiah Ham [00:04:51]: So, I kind of just started thinking about what it would take to do that. At one point, I looked around to see if there was anything that could do this that was already available. And this was in, I don’t know, March, maybe April, earlier this year. And there were a couple things, but they were all either closed source and/or commercial, right? So, maybe you get a free trial. like, here, 5 free hours of device use or something like that. Or at the extreme, there was a company that charged at the starting price $700 a month for, you know, AI agentic coding capabilities. And at that point, that just was like, all right.
Alan Richardson [00:05:40]: Yeah.
Jerimiah Ham [00:05:41]: Okay. It’s on. And And was basically recommitted to making something that was both highly functional for my use case and also something that I could share with the world.
Alan Richardson [00:05:56]: So, is your default normally to make things open source?
Jerimiah Ham [00:06:03]: It has been so far. I’m finding that my instincts for business and commercialization are extremely bad, or maybe just low. I don’t know how to make money off anything that I make. Maybe, you know, maybe I should partner with a business consultant someday. I mean, I do think about it. Like maybe, I guess the farthest I’ve developed it is just a vague plan in my mind to try and like, you know, maybe keep the Quorum Core kind of the open source thing and build, you know, because it really is built for agents first. It has no interface. No visual interface whatsoever.
Jerimiah Ham [00:06:41]: It’s just, you know, new stuff that Claude can do. So it’s built as a tool for the AI, not a tool for humans right now, right? So, you know, maybe in the future, that core stays open source, and I can build something that is more built for humans, that has dashboards and, you know, a visual interface, something that people You know, people care about that kind of stuff a lot. And so we’ll see. Right now, I’m just kind of focused on improving the core, you know, taking user feedback when I get it. And, you know, and just, you know, I do use it at work too. In fact, a lot of people on our team have started using it. You know, initially I told some people about it and they were like, oh yeah, that’s cool. And then, you know, a few months later, one person like actually started using it and they were like, oh, wow, you guys, look at this.
Jerimiah Ham [00:07:44]: And then, you know, so they asked me to do an internal company presentation on it for some devs. I thought maybe 3 people would show up and like there were all— every mobile dev in the company showed up, including remote people. So, there is real interest in it once— Once, um, once the devs sort of see the value, uh, in using it and what, what it can really do, it’s kind of like something that you have to be, um, maybe have to be shown rather than just explained it to.
Alan Richardson [00:08:17]: So I think, so one of the attractive things there is when you look at the feature list and you’re not just, it’s not like, okay, we’ve got Appium that will automate some stuff. you can control the app with it, but you’ve also got the, um, server starting things up. So I assume that when it’s automating, it’s hooking into your Xcode or Android rendered mobile device on the screen. So you can see it there. I assume that it can hook through into your actual device if it’s connected to your laptop as well.
Jerimiah Ham [00:08:53]: Yeah.
Alan Richardson [00:08:53]: Because you’re doing that through the normal, um, dev kit. So you’ve got different visualizations you can use at the same time. Some of the use cases you’ve got on the website are, uh, comparing it, the iOS with the Android. So how, how would it do that?
Jerimiah Ham [00:09:14]: Well, um, since Claude is running everything, um, I’m kind of guessing what your question is. I don’t actually remember that, um, that use case. So I’m guessing functionality or not visual-wise. But the idea is that since Claude is running the session, if you will, you know, it’s gonna— if you ask it to do something, say, okay, run through the login on iOS and then run through it on Android and let’s compare. I don’t know, what do we want to compare? We want to compare how long it takes? Do you want to compare visually how the screens look? Do you want to compare how complete the sort of UI accessibility coverage is for all the screens? Do you want to compare the chain of network calls that happen during startup? All those things are possible. It just, you know, depends on what you ask for. And then Claude decides how to use the tools that Quorum exposes to do that. And the— I mean, the really interesting thing— I mean, the whole— this whole thing is interesting, right? AI has changed so much just in the last year, and the models are so much better.
Jerimiah Ham [00:10:31]: You used to have to be very specific about, like, do it— do this, do it this way. I expect the answers in this format. And, you know, you— You had to be really specific to get what you wanted out of it. And these days, the models are— they actually work better almost the less you specify exactly how to do something and more— and give it more of a goal or what you want to see at the end. So I remember in my sort of early development and I was experimenting just to really to try and get a sense of what the thing was capable of. I just finished, I think, implementing the proxy. So I said, okay, do a login action, and I wanna try and generate as much sort of auditability and traceability for what is happening during the startup of the app. So I said, you know, probably wanna include network transactions, and screenshots, and, you know, whatever actions you’re doing, like what text fields you’re filling in, all that stuff.
Jerimiah Ham [00:11:47]: And just, you know, assemble it for me in a document. So it’s like it said, okay. And then it just went to town. And it went through the whole thing. I was watching it on a simulator. And then I hadn’t specified what the output should be. And it wrote— it made this beautiful webpage that was all formatted, timelines, and screenshots. It was incredible.
Jerimiah Ham [00:12:11]: It completely exceeded my expectations for what I’d asked it to do. So not only functionally, but in the output as well. So I don’t know, we’re just at— we’re just in a very interesting time where you can— it almost depends on your— what you get almost depends on how Yeah, how much you ask for, but also how much, how competent the model is, and they just get better and better.
Alan Richardson [00:12:41]: Yeah, I don’t know, it’s matching my experience as well. Because I’m not building a lot of skills. I don’t build a lot of skill documentation. I don’t maintain a lot of guidance documentation in the agents.md file. Because I’m finding I don’t need to. what I’m relying on, certainly when I’m coding, is the existing code base. That’s where it’s pooling most of its information. And then I’ll put as much in the spec that I give it in the first place.
Jerimiah Ham [00:13:10]: That’s right.
Alan Richardson [00:13:12]: And you have the advantage that you’re providing an MCP server, so you’re not giving it a set of skills or discussions. The MCP server has its tools list, which is exposed, that the AI system can pull out. And the MCP interface is supposed to be getting better. Have you tried to adopt the new kind of RESTful MCP, or are you still using the older MCP?
Jerimiah Ham [00:13:38]: I think it’s the new RESTful MCP. But honestly, I’m not sure. I know it is the newest version that does sort of like loading on demand. So ‘Cause Quern at this point has like 107 different tool calls. And if you loaded all that up, it would be ridiculous. So it goes, you know, on a— it loads tools as it needs them, basically. It does load, like there’s sort of a, you know, hello, this is Quern, this is what this tool is for, and this is the general class of tools available. So it’s like a, you know, like a Claude MD for the MCP, I guess.
Jerimiah Ham [00:14:17]: And there are— I think there’s exactly one skill that I’ve written, and that is really— I guess it’s specific for Qwern. It’s not technical. It’s just instructions to Claude to say, hey, we’re gonna build a knowledge base for this app that both Claude and Quern can use to have some additional functionality. That’s something, I don’t know, we can— it’s more, it’s sort of, uh, I would say that is the part that I’m actively, actively developing right now. And we can talk about it more later or right now if you want, whatever.
Alan Richardson [00:14:59]: So it is on my list. So we’ll see if we get— so we should probably go back to the basics a little bit and describe some of the Quern functionality because we’ve mentioned Okay. You’ve got a whole bunch of tools. What tools are the most important in there to help people with mobile testing?
Jerimiah Ham [00:15:17]: Sure. So the big one is UI control. So what we normally think of as UI automation, if you’re a traditional mobile SDAT, you write— you use Appium, you use native XCUITests iOS. You use Espresso or Compose for Android. And those are, you know, hard-coded test scripts that get compiled, they get installed onto the device, and they get run. Appium, you know, sort of does it differently. It’s got its own little runtime that can interpret live actions. So I would say what we’re doing is closer to Appium, but only for physical devices.
Jerimiah Ham [00:16:09]: Everything is a lot easier on the simulator or the emulator. Or I should say it’s different. It’s more direct for the iOS simulator. But in the end result, and this is something that I’ve tried to do for the whole project, is to try and make the API like one surface. So the API knows, you know, if you’re using an Android phone or an iOS phone. And it— so if you send it a command, you know, to press a button at coordinates or whatever, or with accessibility label or whatever, the API knows how to do that for Android or iOS simulators or emulators. know, basically 4 platforms really that it’s sort of supporting because there’s significant differences between all those 4 at the API level. And so it knows what tool to use, or the API knows what tool to use, and it’ll do the right thing.
Jerimiah Ham [00:17:09]: So you don’t have to keep this matrix of 4 different ways to do every single action in your head all the time. You let the API take care of that. And then Claude doesn’t have to think about that. Either, because it has a single API surface. It’s like, press the button with this, or, you know, whatever the command is. And that keeps its own context clean. It makes it less jumbled, you know, over time. It doesn’t have to think about too many things at once, which really helps it focus and be more effective when it’s trying to do something.
Jerimiah Ham [00:17:41]: You know, there are— there’s always little rabbit holes where something goes wrong, and I invariably ask Claude to You know, what’s— why isn’t Quorum doing the thing I want it to do? And then that’ll, you know, create a little rabbit hole in its context. And usually that’s harmless. So that’s fine. It’s all part of the process and helps me improve the tool. But that’s the big thing. So iOS control— or sorry, UI control is the number one thing, and paired with Yeah, paired with screenshots is probably the number one thing that gives Claude this way, just way more visibility into what your app is doing. And not visible, not just visibility, because it’s— it can read the UI tree and it can interpret screenshots. So it’s sort of 2 layers of different kinds of visibility.
Jerimiah Ham [00:18:32]: But then being able to make a decision like, oh, I see the login button and I want to press it. Before, you know, a stock Claude, maybe you could figure that out. Like, I need to install this tool, I need to do this, I need to read this, and, you know, it’ll go through a whole thing. And it’s so good that it’ll eventually figure it out, sure, maybe. But this way, it’s very clear, right? So it knows exactly what to do, it knows what tool to use, and it realizes that it can do that too, which sometimes it’s really— sometimes it’s charming. Like, you see Claude, And maybe it tries to do something. And, you know, these sessions are not deterministic all the time. So sometimes I catch it, I catch Claude not— its first instinct is not to use Quorum for whatever reason.
Jerimiah Ham [00:19:22]: So I say, hey, you know, go ahead and use Quorum for this stuff. And then it reads a thing and it’s like, oh my gosh, I didn’t know this. I didn’t realize this was available to me. And, you know, whether it’s sycophancy or just genuine delight in the model. It’s hard to tell. I’ll take the delight because I’ve had many more than one interaction like that where Claude is just like, oh my God, what is this? This is amazing. I’ve never seen anything like this before. So it’s kind of funny that that’s a dimension in it.
Jerimiah Ham [00:20:01]: Okay. So back to the list. So there’s UI, control, and visibility. The next one I think is definitely still a work in progress, but I have the foundation laid, and that is real-time logs interpretability and filtering. When you plug into— I don’t know if you’ve ever looked at a raw ADB, the full raw ADB log coming out of an Android device, but it is a lot. It is tons and tons and tons of stuff. There’s a reason that there’s a shortcut in Android IDE to say, just my application, please, because there’s so much else going on. So, filtering a real-time log is super important.
Jerimiah Ham [00:20:52]: And so, I’ve got some basic tools to do that. There’s like a ring buffer where the log goes through, and then you can also put a filter on that. Because you can easily exhaust that ring buffer and the log that you were looking for from 2 minutes ago is already gone. It’s already out of it. So I would say that’s probably the place that’s most ripe for improvement right now, but it is functional. But I’ve seen Corinne and Claude struggle with that part of it probably the most. It takes a little figuring out to tune those filters and to even see what the options are.
Alan Richardson [00:21:29]: Okay.
Jerimiah Ham [00:21:29]: But still very, very super useful. And then the third big thing is the proxy, the network proxy. That’s like, that’s just a whole can of worms in itself. And I don’t know if we’ve all struggled with setting up a mobile proxy on a real device. You have to install certificates and set the proxy manually in the in the Wi-Fi network settings. And there’s just— it’s a very error-prone process and generally a manual process. And generally, it’s not a free tool either. It’s like, hey, you— here, John, download ProxyMan.
Jerimiah Ham [00:22:11]: Okay, you can monitor 3 network endpoints. But if you want more, you know, you got to pay. So, I found an open-source software-only proxy tool called MITM Proxy, Man-in-the-Middle Proxy. It’s a Python— I think it’s a Python.
Alan Richardson [00:22:32]: I’m glad you told me what it stood for. I didn’t know what it stood for. I thought it was MITM Proxy.
Jerimiah Ham [00:22:38]: I didn’t know it was Man-in-the-Middle. Yeah, Man-in-the-Middle, which is, you know, the classic network attack, you know, sort of nomenclature. that you want to watch out for. But very, of course, very effective. That’s what a network proxy is, is a man-in-the-middle. So I found this project and it does have to install a— not a system extension, but some kind of a user space kernel extension. So you need permission to do that. But after you get through that hurdle, then it works great.
Jerimiah Ham [00:23:20]: And one thing, super, super awesome thing I discovered, like when you use ProxyMan or Charles Proxy, those other tools, and you turn— you want to— if you’re running an application on your machine and you want to proxy into it, you have to turn on the whole system. network proxy. And that can— that basically can, like, break any other application that’s running on your machine that isn’t aware of the proxy or doesn’t know what to do. And it can be this infuriating dance where you enable the proxy to look at your thing, and then you have to remember to disable the proxy so you can go back to your web tool and update your work or look something up or whatever. And it’s an awful— friction-filled workflow. And I didn’t realize that MITM Proxy could do this until even after I put it in Quern. But you can turn on the proxy per application instance. So the upshot of that is that you can enable the proxy just for one simulator that you’re running or all the simulators you’re running.
Jerimiah Ham [00:24:33]: But it leaves the rest of your system alone. Nothing is blocked. You can have You can use your computer just as you would expect even while the proxy is enabled on the simulator. That was— it seems like a small detail, but you don’t know how happy I was to discover this.
Alan Richardson [00:24:51]: That might be a Mac thing because it has more permissions.
Jerimiah Ham [00:24:55]: Could be.
Alan Richardson [00:24:56]: I think it’s a little bit easier on Windows to do that.
Jerimiah Ham [00:25:00]: Yeah, it could be.
Alan Richardson [00:25:01]: I was using MITM Proxy Um, a couple of days ago. And one of the interesting things about proxies is they are— they use a lot of libraries in order to connect to the internet. And I found a bug in MITM Proxy where it doesn’t handle some requests, which are perfectly valid thing not to handle. If you’ve got a 204 message and it actually has content, which it’s not supposed to have— a 204 is not supposed to have content—
Jerimiah Ham [00:25:29]: Mm-hmm.
Alan Richardson [00:25:30]: MITM Proxy will strip the content out before passing it on to you. But it’s not the only proxy that does that. I was excited, so I, I went down a little rabbit hole of testing a few proxies, and, uh, OWASP ZAP— it’s not OWASP ZAP anymore, it’s just ZAP Proxy— also does that. The only proxy I could find so far that doesn’t do that is Burp Suite. Oh yeah, Burp. Um, it’s really important, unfortunately, if you’re testing really extreme things, that you have to know the limits of the tools you’ve got. But it was good to know that multiple proxies have the same issue. So, um, it’s not just the one thing.
Alan Richardson [00:26:09]: But MIT Proxy looks good. For people who haven’t used it, it’s an interesting tool to use because it’s almost like a headless proxy because it’s sitting there in the CLI, but it’s got a web interface on top. So it’s a very different experience from using one of the desktop proxy tools.
Jerimiah Ham [00:26:25]: I haven’t even looked at the web interface. I’ve only used the command line and only as a way for Claude to be able to use it. I was so focused on the AI tooling that I barely explored it using it as a human.
Alan Richardson [00:26:40]: So we should jump back to the AI stuff. So you’ve got— so this may be why there’s so few videos of Quirred out there, because when you start up—
Jerimiah Ham [00:26:50]: I haven’t made any.
Alan Richardson [00:26:51]: Yeah.
Jerimiah Ham [00:26:52]: I really should.
Alan Richardson [00:26:54]: You’ve just got the Um, command line interface for the server running and everything else is headless behind the scenes. And then you’ll be using something like Claude or Codex, which is basically just a command line interface as well. So you’ve got 2 command line interfaces. Then if you’re lucky, you’ll have your simulator on screen showing you things are happening. That’ll be the most visual part of this.
Jerimiah Ham [00:27:17]: That’s right. Showing the simulator. And I also built a tool, like if you’ve got your phone plugged in or maybe several phones. If you’re probably— I don’t know, I hate to say serious dev, but if you’re the kind of dev that has 10 phones connected to your dev machine all the time, they’re probably not all laying around on your desk. Or maybe they are because you need to see them, right, and interact with them. But I built a tool that just mirrors the screen of your plugged-in real iOS phone onto your desktop. So you can put it in a drawer or put it behind your laptop or put it somewhere else. It doesn’t have to be right at hand for you to be able to see what’s going on.
Jerimiah Ham [00:28:03]: Because you also don’t have to interact with it maybe as much because Claude through Quern can launch the program, can interact with it, and can do all the things that you’d normally be doing when you’re— You inevitably have to interact with your phone or do some manual testing or change a setting or whatever. This, yeah, the Quorum’s ability to operate your phone at a very high level kind of changes how close at hand your actual phone needs to be. It does still need to be plugged in.
Alan Richardson [00:28:41]: Yep. And the fact that you’ve got an MCP server with a an AI coding agent communicator changes the way that you would test. Because when you think of an automated testing tool, you wouldn’t think of what you’ve built, because what you’ve built is something that you’re feeding commands into an AI, which is then doing stuff on your behalf. So it’s much more like an exploratory type of using the phone than automating it. Like, I know that you support scripts and things, and we’ll probably talk about that later, but— But it’s a different style of doing mobile automated execution and using AI.
Jerimiah Ham [00:29:18]: It is. And I think I came to it just through my learning how to use cloud code in general. Because I do, I don’t know, just a little bit about me, I have an overactive inventive mind. I have, you know, ideas for little personal apps all the time, it seems like, many of which I’ve made at this point. I made an MP3 CD burning app. That is like niche of niche. Like, who cares, right? But I have a CD player in my old car and realized my iPod died one day. I was running a vintage iPod and it died one day and It just occurred to me I don’t have any offline music in the car.
Jerimiah Ham [00:30:08]: So I’m like, okay, I’m gonna burn some CDs. Looked at what’s out there. Again, it’s all either paid or crap. So I was like, okay, I’m gonna take this opportunity to make it because I come from— it helps that I come from an audio engineering background when I was in school. So I understand codecs and transcoding and audio files and all this stuff pretty well. And, you know, spent about a month refining this thing. And it’s great. It’s like drag and drop, reorder, click the button, boom, you’ve got 7 albums on a single CD, pop in your car, you’re done.
Jerimiah Ham [00:30:43]: So, you know, that was for me. And but I still, even though it was for me, it was like this, you know, still self-training on how to use an AI agent better, not only for like This is the ticket that was assigned to you on the Jira board, and, you know, you don’t have much stake in it personally, maybe, besides just getting it to the next step. You know, you feel like a cog in the machine, you know, whatever. I’m just— I’m really into greenfield programming and not— and sort of letting a program develop organically as I code new parts of it. Figuring out the next step, you know, having it— sure, having an overall vision in mind of kind of where I want to go to, but not being like, okay, this is the map, this is the road, it’s been calculated, we’re going to go here, here, here, here. I like to, you know, if I get new information at that town and I was going to go right, but the new information suggests let’s go left, let’s explore that, I won’t— then I’ll take that road. And it just had really, really good results. with that methodology, especially using Claude, because I’ll flip back and forth from, you know, asking Claude to do a specific thing to saying, okay, let’s talk about this and brainstorm different things.
Jerimiah Ham [00:32:06]: I give it ideas and that sparks new ways, new ideas inside Claude. So it’s It’s almost like— I’ve been a musician for a while too, and it’s the difference between trying to write a whole project or a whole song yourself, where it’s only you, you know, just trying to come up with stuff, and being in more of a jam session kind of scenario where you play something and it inspires a new idea in somebody else in the room that you never would have thought of. But, you know, all of their previous knowledge and experience and whatever make them have that thought. And then they start playing that. And then that spurs yet another new thought in you that would not have existed because it needed the other person to contribute their thing. So there’s a lot of— there’s this lot of like this brainstorm jam session that I do with Claude. Even during the Quorn development, I would code a feature. We’d try it out for a while and then at the end, I would say, I have a session, I would say, okay, let’s take a step back from using this tool.
Jerimiah Ham [00:33:19]: Tell me how— tell me, Claude, how easy was it to use this for you? What were the friction points that were, you know, you reached for the tool and it was the wrong thing or you thought you were using it the right way but it was the wrong way? Give me feedback about your experience using this. So I was treating Claude as the customer for this whole thing. I think that’s really important.
Alan Richardson [00:33:47]: Yeah, because you’re not trying to have multiple agents doing coding. Because people are— one of the things that’s interesting at the moment is we’re all learning how to use AI to develop things, and we’re all kind of doing it our own way. And we see lots of stuff online about you must do it this way, you must build systems that will build systems, you must create lots of skill files, you must do this.
Jerimiah Ham [00:34:07]: Yeah.
Alan Richardson [00:34:07]: So I would be interested in drilling a little bit more into how you’re using AI because you’ve described it a little bit. Because, um, we’re in the position where we’re kind of product owners, where we have a vision for something we want to do, so we guide it through that. We’re also in the position where we are developers, so we understand code and we could write the code ourselves and we can review the code. So do you actually review the code that is being written? Not at all.
Jerimiah Ham [00:34:33]: I haven’t opened an IDE except for to be able to look at a formatted MD file nicely in what feels like 6 months or maybe a year. I don’t look at the code at all. It’s all indirectly verified through testing.
Alan Richardson [00:34:52]: So is it indirectly verified through testing or through usage?
Jerimiah Ham [00:34:58]: Oh, well, both, yes. Indirectly verified through usage, directly verified through testing. Maybe that’s a better way to say it. But I don’t read code anymore. Even on, you know, slow grind personal projects where, like, I don’t know, there are still some developers who are really into the code itself, right? They’re trying to get They’re trying to— they’re actually using Claude to optimize the, you know, small pieces of code that they write, and especially for performance. And there’s certainly a lot of merit in that when you’re going for performance. But for the little, you know, kind of podunk things that I’m— feels like I’m writing, performance is not the main thing unless there’s some kind of huge blocker for it. Most of it’s functionality.
Jerimiah Ham [00:35:48]: And And, like, to reference what you were saying earlier about so many ways to develop software, and now people are getting into orchestrators with multiple agents and this kind of stuff. And that’s nice, but it really is creating a factory. And I, for my personal projects, I wanna be in the woodshop. I wanna operate the table saw and cut that piece of wood and make— my one table. I don’t need to build a factory that can create a million tables. So, you know, there’s just so many different ways and motivations behind, you know, the way different people are using these systems, and especially, you know, different companies.
Alan Richardson [00:36:38]: And I think it’s important to try and show as many different views of that as possible, because otherwise people just see the online, very vocal influencers saying you must do it this one way. There are other ways. And it’s basically, if you are building something and you are getting results, then it’s working.
Jerimiah Ham [00:36:55]: Yeah, exactly right.
Alan Richardson [00:36:57]: I mean, I kind of jump between, sometimes I will read the code. For some of the things that I’m building, I read the code most of the time. Some of the things I’m building, like I’m also working on a kind of fractal rendering and movie type app because I like fractals. I haven’t looked at the code of that at all because that’s just a private personal project. But if it’s something that I’m putting out, then I’m much more likely to read the code because I still like elegance of code. So even— so it depends what I’m doing, but the most important thing are the results and using it. So also you mentioned So, are you getting the AI to write a lot of unit-level tests? Yes.
Jerimiah Ham [00:37:44]: Lots of unit-level tests that I also don’t read.
Alan Richardson [00:37:48]: But are you— so, even though you don’t read it, are you helping it architect them? Like, with a testing background, do you say, I want some at this level, I want some at this level?
Jerimiah Ham [00:37:56]: Occasionally, but honestly, not very often. No. I pretty much am kind of reactive. And this is definitely not a best practice, I would say. But for my own projects, I kind of only get in there when there’s problems. And then if there is a problem, I say, Claude, this seems to be— we hit a bug here. And I’ll say, what kind of tests could If we had written that would have caught this condition. And then it’ll be like, oh, yeah, this, this, this, this.
Jerimiah Ham [00:38:34]: That’s a whole class I forgot. I’m like, good. Okay, please do that now. I’m not micromanaging any of it.
Alan Richardson [00:38:43]: Yeah. And it makes it— I do that as well. And what I do find, which you probably won’t notice because you’re not looking at the code, is that once you have all those examples, the AI starts to add and expand those. So I do, I do often look at the test code more than I look at the functional code, but also very often I’m working in very small chunks, which allows me to review it. I’m not working on huge features, which would then be impossible to review the code that’s coming in. But it’s just interesting to see the different styles people have because I think they get hung up. And what’s also interesting is you’re using Claude.
Jerimiah Ham [00:39:18]: Yep.
Alan Richardson [00:39:19]: Claude sounds like it has a more specific personality than Codex. I tend to use Codex.
Jerimiah Ham [00:39:25]: Codex is very businesslike. It is very businesslike. Uh, I’ve, I’ve only used it a few times and it’s honestly, it seemed a little curt to me. Um, I do like my agent to have a little bit of a personality. Um, and like, because I, uh, you know, I am a pretty lighthearted guy. I like to make jokes with it even in the middle of a session. Um, I will like, I don’t know, one time A couple months ago, Claude proposed something. It asked me, should I do this? And I’m like, you know, make it so, you know, a Star Trek reference.
Jerimiah Ham [00:39:56]: And it responded, aye, aye, Captain, you know, and I was just tickled by that. So I don’t know if Codex does that because I haven’t used it nearly as much.
Alan Richardson [00:40:06]: So it doesn’t do it to me. So maybe it just adapts to the personality of the person that’s using it.
Jerimiah Ham [00:40:10]: I have heard that that is very likely.
Alan Richardson [00:40:13]: Yeah, it’s good that the different agents have different personalities. But what is then also interesting is, have you tried to use the automating through Quern with smaller models or with open code? Or is it— has your experience mainly been Claude?
Jerimiah Ham [00:40:32]: That is on my list. It’s something I’m very curious about. And I just haven’t had the time to really do it. But I would love to. I keep trying and not getting very far. I’m having a hard time finding the right combination of, like, open code or some harness and a model that responds well to the tool calls. Every time I try and set that up, there’s, like, some catastrophic failure where it’s just It’s not working right, and it’s obviously not working right. So I’m not going to— so I just keep— every couple of months, I’ll try it again to see, okay, does this combination work? Is this thing— I see a video on YouTube, and I’m like, oh, I’ll try that.
Jerimiah Ham [00:41:19]: And maybe I can get it working finally, maybe not. At this point, I just haven’t had much luck.
Alan Richardson [00:41:26]: I mean, that’s also one of the issues with AI at the moment. There’s too much to experiment with. So it’s much easier to just focus on the results. So when you use an open code, have you tried to use the open code Zen or you try to run with local models?
Jerimiah Ham [00:41:41]: I’ve never tried. It’s only been local models. I’ve never used any cloud models except for Claude, Codex, and, well, Gemini, although I barely count that.
Alan Richardson [00:41:55]: So next time you experiment with open code, sign up with token code Zen, don’t put your credit card in, then you get access to their free models in the cloud. And then you can experiment with different types of models and you can get access to a lot of the free models that you’d be running locally.
Jerimiah Ham [00:42:09]: I see.
Alan Richardson [00:42:10]: That sounds great. The other thing I use is OpenRouter, which Stripe have just bought, which makes me nervous. But OpenRouter, you can then choose the very small models that I would run locally because I used to run things like Llama 3.8 locally, but then I would have to change the context window and configure that. And I just find it a lot easier to run through OpenRouter. And Llama 3.8 is so cheap through OpenRouter. It’s like 1 cent, sometimes 0.0001 cent per call. So it’s an easy way of experimenting. I’ll give that a try.
Alan Richardson [00:42:49]: If you ever get the time.
Jerimiah Ham [00:42:50]: Yes.
Alan Richardson [00:42:50]: Away from not doing your music and not doing your private projects and not—
Jerimiah Ham [00:42:54]: I don’t do music that much anymore, unfortunately. But, uh, yeah, it’s hard. But I still remember. Yeah, it is, it is. And it— I mean, uh, in this, this last year, it’s been, uh, crazy. I know that, you know, the timeline’s sort of a little bit different for everybody for when, um, I don’t know, AI-pilled is such a loaded term, but when you had your sort of light bulb moment And you’re like, oh, wow, this is really cool, really different, really powerful. I can’t stop playing with this. I can’t stop looking at this.
Jerimiah Ham [00:43:32]: I can’t stop thinking about it. What am I going to do? And you start to dive in and you start to work on projects. And then you realize that, yes, that awesome feeling of, wow, I could do I can do anything I can think of, turns into, I don’t have enough time to do everything I can think of. And then you have to actually prioritize your own fantasy projects, which is like this whole new skill that we haven’t had to use before because they were never even— even one of them was probably not even possible before in a decent timeframe.
Alan Richardson [00:44:10]: Yeah.
Jerimiah Ham [00:44:10]: It’s just a whole new level of things. I don’t know about you, but I know when I’m really in the zone and focused on something, I’ll go, you know, do a week’s worth of really intense AI coding, AI development, and come out of it and it’ll feel like 2 months has gone by. It’s like this weird time compression. It’s very hard to describe.
Alan Richardson [00:44:36]: It’s also because you’ve managed to achieve so much over that week. It’s unbelievable how much— I mean, I’m working on projects I’ve had started for 5, 6, 7 years and making so much more progress now than I have done over that time. It’s unbelievable.
Jerimiah Ham [00:44:51]: Yeah.
Alan Richardson [00:44:52]: But I think it’s at this point in time, if you’ve got a vision for something and you’ve got the budget to pay for the AI, then you can achieve it. I think one of the benefits of having a reduced budget when using AI is that it forces you to take a break because you run out of tokens and credits.
Jerimiah Ham [00:45:10]: That’s probably true. Yeah.
Alan Richardson [00:45:13]: Once you start seeing the value, you then just keep upping your spend. And then you get into the point where I can’t run out of tokens. So, yeah, I’m never going to stop.
Jerimiah Ham [00:45:22]: I have decided that my limit, well, I will spend, personally, I have the $100 a month Plaid subscription. And that is enough for me to, if I want to, for better or for worse, be actively using Claude all day long without hitting a session limit. And that on one or two projects. I can’t drive like 7 OpenWhales and 7 projects from that. That’s too much. But I don’t want to do that either. When I’m into a project, I really like to keep my own personal context, my brain, in the project. So, I’m holding a lot of it in my mind.
Jerimiah Ham [00:46:08]: And I just find I get a lot more results and satisfaction out of working like that rather than trying to juggle 7 things at once. That’s just too chaotic for me.
Alan Richardson [00:46:21]: Yeah. So, I don’t like juggling it at the same time, but I move from one project to into the next. But sometimes I will be working on several in different threads. Just start off in the morning, then ignore it for a while, then come back.
Jerimiah Ham [00:46:35]: Yeah, for sure.
Alan Richardson [00:46:37]: But on that note, earlier on you mentioned the app knowledge base. So in order to avoid holding everything in your head, you’re working on the app knowledge base for the Quern. So how’s that working?
Jerimiah Ham [00:46:49]: Well, it’s, I think it’s It’s working pretty well. And it started off with, you know, I don’t know, it seems like every feature is just like a solution to some problem I noticed. And the problem I noticed after I had built the basic functions of Quern, the UI control, the log analysis, and the proxy, is that you start a new session and you say, hey, here’s my, you know, we’re gonna work on this part of of the app today. Please put a new build, boot a simulator, and then let’s get to work. And it would spend, you know, 5 or 10 minutes trying to figure out how to get to the screen that I had mentioned in my opening salvo. So I realized that it basically, it didn’t, you know, Claude itself—
Alan Richardson [00:47:42]: Yeah.
Jerimiah Ham [00:47:45]: We all know that the context resets and it doesn’t know anything about anything except what it knows at the beginning. It doesn’t know anything about your stuff, right, every time it starts. And I don’t know why I didn’t— it didn’t occur to me that it would be different or wouldn’t be different for mobile apps. But it was a surprise. Like, oh, yeah, of course.
Alan Richardson [00:48:09]: Claude doesn’t—
Jerimiah Ham [00:48:11]: It doesn’t know my app from Adam. So I realized I needed to start to create some kind of documentation that Claude could use to understand my app more fully instead of me having to reintroduce it every time. It’s the same problem as having to reintroduce the parameters and the goals and the of whatever project you’re working on to the fresh agent every time. That’s, you know, early struggles. So anyway, so I did— I’ve been absorbing content from various places, YouTube, Medium, and who knows where else. And this idea of sort of a knowledge graph or knowledge map, knowledge base, kind of kept popping up. You know, there was a lot of people trying to build second brains. in March and April.
Jerimiah Ham [00:49:07]: And some people were doing it with vector databases. Some people were doing it with just straight-up MD files. So I decided I’d try the MD file method. And this is not, you know, it’s not something— it’s not gonna— you’re not gonna end up with 1,500 MD files or something, you know. Even a really complicated mobile app, you’re probably gonna have You know, 60 or 100 different views, and each view is its own file, and it’s got some, you know, it’s got some information in it. But importantly, it is a graph. So each screen file contains, you know, what its parents or parents might be and where that screen goes next. So it gives It gives Claude a logical and constrained idea of what are the possibilities here on the screen, what can I do, without having to rederive them every time.
Jerimiah Ham [00:50:09]: It contains a unique fingerprint of some UI accessibility labels so that one single query to the UI tree will let Claude be able to determine exactly where in the app it is. rather than guessing, which is, you know, if you’re not telling it, it’s going to guess. And guessing costs time and tokens and can be frustrating. So that’s kind of the main thing. And as I was alluding to, or as I mentioned at the beginning, that’s the only— I believe it’s the only skill I have in Quorn itself. And it’s basically like a little recipe that says, okay, Claude, We’re gonna use Quern and we’re gonna, you know, walk through every screen in the app and we’re gonna create an inventory. Each file gets— each screen gets its own file. It’s formatted like this.
Jerimiah Ham [00:51:06]: Keep going. This— and by the way, this is— it’s designed not to be a fully automated process. It’s designed to be A process that you— an activity that you do along with Claude while it’s going through this. So it’ll show, you know, it’ll bring up a screen and that will trigger you as the app expert. That’ll trigger like maybe some extra little side information that Claude wouldn’t discover by itself or isn’t obvious. Like, oh, this screen is only shown, you know, the first 3 times you start the app up. You know, some little detail like that. that would take forever for Claude to discover.
Jerimiah Ham [00:51:48]: And it’s so, like, buried in, you know, the deep context of your mind about your own app or the app that you are, you know, are working with and that you’re the test specialist for or whatever. Those— you’ve got 100,000 little details like that built up over time the more you test an app, right? And it becomes this sort of implicit tribal knowledge that’s only in your brain. You can socialize it if there’s lots of different testers. You can tell people about it. I work at a very small company. There’s only one tester on each dev team. So there’s no way to socialize that. It generally doesn’t get written down.
Jerimiah Ham [00:52:28]: It’s something that you get with experience. But it is something that you— these are little details that can come to mind when you get a little visual cue. It’s as if you’re onboarding, you know, a new coworker and you’re showing them the app for the first time and you’re in that mindset of like, you know, try and transmit as much information that you know to the new person. So seeing the new screen, when you’re in that mindset, you think of those things like, oh, this is like this thing. And this only happens if you’re a premium tier member. This only happens if you’re this. You know, so it helps. It really, really helps Claude establish a much more comprehensive picture of what your app is, how it works, what it’s for, what are the different constraints.
Jerimiah Ham [00:53:19]: You know, it might not even occur to you at the beginning to say this app, you know, has 3 tiers of use. It has a free and generic, you know, generic subscription and a premium subscription. Why would you tell it that at the beginning? You know, you should, but there’s no prompt and there’s no— we don’t have any habits, you know, formed around these interactions to even give you a clue what to dump. So that skill guides you through.
Alan Richardson [00:53:50]: So for that skill, is that a skill— so you’ve got Quern sitting there as a server. You’ve got your project. I assume you create a separate project folder for each app that you’ll be testing. Do you put the skill in that folder, or do you rely on this, on the Quern server picking up the skill?
Jerimiah Ham [00:54:10]: The skill lives in the, in the Quern MCP, so it’s generic, and it really is just a skill file. But the output of what happens when you use that skill is that, because the idea is that You launch Claude inside the folder of your iOS app or whatever, the code base you’re working on. So it’s in there. It knows everything about your code base. It launches the app. So when you use that skill, it builds— it actually creates a .quern folder in your code repo, and it will put the knowledge base inside there.
Alan Richardson [00:54:49]: Okay.
Jerimiah Ham [00:54:49]: So it becomes part of your Git history. It becomes versionable, updatable, and I would recommend that you add it to version control because the whole idea is that when the app changes, you can update that and it’s always present or always current.
Alan Richardson [00:55:08]: The knowledge base stuff is really interesting because it is— when you’re coding, it’s fine because you’ve got your knowledge base, which is the entire code base. It doesn’t have all the decisions you made and the reasons, but Sometimes what I’m starting to get AI now to do is add more comments that describe why certain things exist. Comments are usually bad for humans, but for AI, they’re great because it keeps them up to date all the time.
Jerimiah Ham [00:55:32]: Yeah.
Alan Richardson [00:55:32]: But when it comes to testing, all testing is driven by a model. And what we do is we build that model up over the time. Very often we don’t make it actually written down, but you’re taking a process to build up a model of the application And I’m really interested in that lightweight approach because you’ve just got the Markdown files, you’ve got references to other Markdown files, you’re creating a graph. Is it all in a single folder or are you trying to structure it?
Jerimiah Ham [00:56:05]: I think it’s all in a single folder. Yeah. Yeah. It’s, it’s, yeah, it’s proven pretty effective so far. As far as it’s actually, it’s great. I mean, I can— sorry, too many thoughts at once. It’s great because I really can get started on a fresh session right away. There’s no question from Claude about what it needs to do to get to the part of the app that I wanna look at or work on or test or whatever.
Jerimiah Ham [00:56:38]: That’s great. And it even gives— kind of special superpowers. This is just something we discovered along the way of making this knowledge base. Apps often have deep links in them that will take you to a specific part of the app. And Claude sort of— well, Claude and I both realized that we could, instead of navigating through 6 screens to get somewhere, you could use the deep link as almost a teleport to go right directly to a place in the app, which again saved time. And it was just a very neat realization. I’ve never considered using a deep link as a primary navigation tool inside either a test script or for anything else because there was no— nothing else, I guess.
Alan Richardson [00:57:35]: Yep.
Jerimiah Ham [00:57:36]: So, and usually opening a deep link is a total pain in the butt. You know, you’ve got to make a web page that contains a deep link or send a text message to yourself with a deep link and/or do something from the CLI or whatever. And asking Claude to open a deep link, especially using QWERN, is trivial. It’s just super, super easy. And at this point, it’s built into the API. So, yeah, it’s just— there’s whole new ways of interacting with the app and getting the results that you want that are sort of revealed by this toolset.
Alan Richardson [00:58:21]: Well, I mean, this knowledge-based approach that you’re using is interesting because it almost requires you to have a frontier model and toolset. Because if you reduce the model and you reduce the tooling, you need a more formal model of your application because it doesn’t have its larger context. It can’t understand it as easily. It doesn’t know how to navigate it. So a lot of the commercial tools, when they’re trying to build this, they have to formalize that knowledge base. more carefully and more directly. So it becomes much more like a graph database rather than just a set of Markdown files. I mean, a set of Markdown files are essentially a graph, but theirs is a much more formal version.
Jerimiah Ham [00:59:05]: Right.
Alan Richardson [00:59:05]: And they have to go things like, well, let’s keep up to date, let’s do self-healing on the locators and the deep links and everything else before letting the AI loose on it, because otherwise it will confuse the AI.
Jerimiah Ham [00:59:17]: Mm-hmm.
Alan Richardson [00:59:17]: But you’ve got— it’s one of the really interesting things when you just take a coding tool with a frontier model, you can do so much that the commercial tools give you with so few resources. Like, you’ve just got to set a markdown file, which is— and you could basically say to Claude, have a look through this and see if there’s anything new added into the application. And it would go away, do that, investigate. You can’t do that with a smaller model subset, and you need to code it in.
Jerimiah Ham [00:59:49]: Yeah, I think that’s right. And unfortunately, I— this tool was not primarily built for non-frontier model use. But I am— but I definitely am trying to refine the way that Quern uses the KB to be more programmatic and less model interpretation dependent. A feature that I think it’s pretty much complete, but I didn’t want to ship it before I went on vacation for, you know, right before I went on vacation for a month. That seemed risky, irresponsible, because I actually have, I mean, I have a very, very modest number of regular users of this tool. It hasn’t exactly gone viral or anything like that. But many of them are coworkers and they actually depend on it. So I didn’t want to break it.
Jerimiah Ham [01:00:44]: Anyway, what I’ve been doing is using the graph and the fingerprint to make it so that when you call an action, like hitting a button that you know or you expect to go to the to open up the next screen in the app, right? So the knowledge base knows about that. Claude can know about it if it reads the MD files. But what I want to— what I’ve been trying to do is that you can make the API itself programmatically know what screen is supposed to come next. And using that same fingerprint, it can tell you whether that action resulted in the expected outcome or not. So instead of success being 200, button pressed, success becomes expected screen landed and loaded.
Alan Richardson [01:01:47]: Yep.
Jerimiah Ham [01:01:48]: And that gives— so that can— I think that could hopefully start to work around the limited model sophistication problem because you get a lot more information that isn’t dependent on a model having to interpret it. You get real information about if you landed on the right screen, if everything’s proceeding according to plan, and it still knows where it is in the graph, all that stuff. So that’s kind of the thing I’m really most excited about. And there’s a principle in there But I guess I sort of derived from thinking abstractly about why is it so easy for me to tell instantly when something goes wrong and it’s so hard for Claude to even realize that something went wrong, right? And of course, it’s because I’m looking in real time at the app screen. And if it does something unexpected, it’s instant. I instantly know. It’s like this massive interrupt that triggers when something unexpected happens. And Claude just doesn’t have that kind of sensory perception, especially not real-time.
Jerimiah Ham [01:03:09]: So, and the other thing I realized about my vision is that I’m not like telling my eyes, okay, look at the screen. and pay really close attention to see if it’s gonna do the thing you think or if you’re gonna be surprised by something else. It happens automatically. The visual comes into my brain without me thinking about it. It’s context, free context that I’m constantly getting fed. And so I wanted to create something That was more akin to that for Claude. So that it gets this— it gets information or context about an action it took automatically and for free without having to ask for it specifically. Because the pattern I saw it getting into was like, okay, it’ll press the button.
Jerimiah Ham [01:04:06]: Good, the button press worked. Okay, now what screen am I on? Did it work? It has to go through and answer that question. And if it forgets, if it gets overconfident and says, oh, I know how to get to this part of the app. I press this and then I press this and then I scroll here and then I press that. If something goes wrong at the first screen, it’s like timed out waiting for the success of the next 2 things because those screens are not the screens that are up. So that’s kind of the operating principle I’m trying to come at it from. is to try and give, try and almost force new information into Claude’s input stream as effortlessly as possible.
Alan Richardson [01:04:52]: So one of the interesting things there is earlier on we discussed that we’ve not got multiple agents and systems of things, but what you’re describing now is perhaps it would be useful to have multiple agents. So you’ve got your main Claude thread that you’re interacting with, But another one sitting there that knows what screen you’re on and also has access to knowledge base, but it’s just constantly fed screenshots and told, tell me if anything goes wrong, interrupt us and let us know. And that could come through the MCP server, be information.
Jerimiah Ham [01:05:20]: That, that, yeah, I mean, the ideal would be, you know, a model that is super good at analyzing a real-time video stream. That, you know, there are models for doing things like that, like YOLO, that can do object identification and things like that. That would be actually— that would be really, really interesting if you could train a YOLO model on your app so that it can know what screen it’s on by visual matching. Then you get real-time analysis, this in the same way that our eyes work in real time.
Alan Richardson [01:05:51]: Yeah.
Jerimiah Ham [01:05:51]: And YOLO stuff, you can run, you know, if it’s a trained model, it doesn’t have to be a supercomputer. It can run on a You know, Pi 4 or something in real time. Very, very interesting idea.
Alan Richardson [01:06:05]: So that’s the advanced stuff. So what people, when they hear this stuff, the AI, you’ve got your coding agent, what capabilities do you have for turning some of the exploration into pre-canned scripts that can be run independently of the AI?
Jerimiah Ham [01:06:24]: Sure. So the bonus and maybe the challenge of the way I made Qorn is that every MCP tool call is actually just a wrapper that calls the API. You know, there’s a little mini server, the Qorn dev server, Qorn debug server I think it’s called, that is running on your machine. And when, you know, when you don’t make any API calls, it’s not doing anything. But it’s always there. And so every time you’re doing an interaction or you’re setting up a proxy or looking at a filter or tuning, you know, any, any Quarkus call really is an API call. And that means that you can write a script for all of these actions. And I think it’s, you know, in the In the simple paradigm of that, you would start off with just writing scripts that record, you know, it’s almost like an automatic script automation writer, you know, it’s like record my actions kind of thing.
Jerimiah Ham [01:07:37]: Only instead of recording your button presses, it’s going to record Claude’s button presses, right? Which is what it’s really doing. Because you don’t specify, you know, when you’re working with an agent, you don’t specify the thing. You say, do this, you know, operate this device, and then it presses the buttons. So once you get a script like that, and since it’s a generic REST, you know, HTTP API, you can write it in any language you want— JavaScript, Python, Rust, if you’re so inclined. You know, I don’t know why you would, but interpreted languages are are great because it doesn’t require any compile time, super easy to rerun them, you don’t have to install them anywhere, whatever. So you get basically a list of actions: press the button, enter the password, verify you’re on the right screen, and then if that all works, then your test passed, you know, or your script runs Successfully, I should say. So that’s kind of the like beginner mode for how to use the API. And I still, I don’t have a huge script library for any apps I’m working on.
Jerimiah Ham [01:08:51]: I honestly haven’t used it that much. I’ve used it more in— Claude has used it more than me, I should say, because sometimes it wants to do it wants to do some more deterministic repetitive testing. And instead of it, you know, driving Quern, it’ll actually write a Quern script and be able to do those test runs over and over again. Sometimes it’s like, I want this to succeed, so I’ll write a script that should work, and then I’ll keep making changes in the code until it gets there. It’s almost doing its own test-driven development, I guess.
Alan Richardson [01:09:32]: Yep.
Jerimiah Ham [01:09:34]: And so, that’s mostly what I’ve used it for. I am— I guess I’m still trying to figure out the best way to integrate some of those concepts into Qorn itself. Right now, it’s very much like DIY. If you have a way to use the API, then by all means do it. But it’s not really a hugely supported family of use cases that I’ve spent a lot of time on. But it is interesting.
Alan Richardson [01:10:09]: That’s the benefit of working with a CLI coding to— with a frontier model. You don’t need to. If you wanted to write the scripts, you just have Claude say, go and use the API, pull out the things, and write the script.
Jerimiah Ham [01:10:22]: That’s it. That’s it. And I mean, it really— and I’m still trying to get my head around it. It turns everything that I’ve done for the last 20 years as an application and mobile tester on its head, right? We don’t need to worry about the mechanics of writing UI test scripts if we don’t want to. You can still work the way you are accustomed to, of course. But there’s just this incredibly new and upside-down way of working now. And it takes some getting used to. And I’m not even sure when it’s right to use it and when it’s right— or when it would be wrong to use it that way.
Jerimiah Ham [01:11:05]: I’m still developing a sense of it. I think many of us are. But it’s interesting.
Alan Richardson [01:11:13]: I mean, What you’re building is something that you pair with. I mean, you could hook it up with agents that have a system and they could do things on their own, but you’re using it as something that you pair with when you’re testing and using on real devices and simulators.
Jerimiah Ham [01:11:27]: That’s right.
Alan Richardson [01:11:28]: You’re using your brain and eyes as you’re doing it. Claude has the ability to look at HTTP requests and the accessibility tree and everything else underneath that you don’t get a chance to see. It can point things out there. So you’re using it to augment your ability to interact more easily, observe at levels that you can’t do, flag things up that you would otherwise not notice because you’re looking at that part of screen. But you can tell Claude, when we do this, I want you to look over here. I want you to look at how the memory’s fluctuating. I mean, that’s how I use it for coding. You’re using it for testing in that way.
Jerimiah Ham [01:12:02]: Yeah.
Alan Richardson [01:12:03]: And it’s just once you’ve got the MCP, there to help you control it. It is a completely different way of working. And it is interesting how things change. Because I don’t feel when you’re doing that, that you’re a passenger, you’re still directing.
Jerimiah Ham [01:12:21]: Yeah, yeah. Have you heard of Corey Doctorow’s human centaur versus reverse centaur? His terminology? His— so the idea is that if you’re using AI in a way that you are the one that’s still driving it and you’re— it’s augmenting your own needs and desires, then you are the centaur. You’re the human and the AI is the horse body that gives you extra speed and strength and abilities or whatever. If it’s the other way around, if the AI is your boss and it’s telling you what to do and you have very little agency, and if there’s any nuance that can’t— that makes it hard to decide if you should do thing A or thing B because it can be interpreted differently, there’s no recourse to go up and ask your app as a boss what you should do. It’s either— you succeed or you don’t and you get fired, whatever. That’s the reverse centaur. So, some other head is the brain of the centaur and you’re being asked to be the horse body. You’re the Amazon delivery person, you’re the whatever is the gig economy flavor of the month job, flavor of the decade maybe.
Alan Richardson [01:13:49]: So, Yeah, it’s interesting. It’s like, how long is AI, how fast is going to AI advance? Whether it’s decade, year, month, who knows? All right, so we’ve been at this for a while, so we should probably stop because we covered an awful lot of stuff. And the main driver here is we should be directing people off to quern.dev. And just in case people don’t get it through my accent, how would you pronounce it?
Jerimiah Ham [01:14:14]: My American accent says quern. Like Q-W-E-R-N, even though it’s Q-U.
Alan Richardson [01:14:21]: But it isn’t W, so no one go to Q-W-E-R-N.dev. No, yeah, it’s Q-U-E-R-N.dev. Yes. Right. And you’re also active on Reddit, so people will find you there if they search for Quern and LinkedIn.
Jerimiah Ham [01:14:36]: A little bit, a little bit. Yeah, X is probably where I’m posting most of these most these days. I’ve actually honestly had a hard time cracking Reddit. In places where I think the tool would be most interesting to people, they don’t want to hear my pitch, I guess, you know? And I don’t know how to say, hey, use this thing and by the way, I’m the creator, so therefore It’s a pitch. Me even mentioning it is a pitch.
Alan Richardson [01:15:11]: Have you been banned yet? I’ve been banned from some of the Reddits for linking to my stuff.
Jerimiah Ham [01:15:16]: No, I haven’t been banned. But so, yeah. So, I guess I’ve had a little bit more success on Twitter. But— and I’m on LinkedIn and I talk about it too. And I do have a Medium thing where I post about other stuff too, occasionally. So I don’t think I posted about Quorn yet, but—
Alan Richardson [01:15:38]: So I haven’t found your Medium. So I’ll put some links in, and if you send me some links, we can add some more. So is there anything else that we should mention and cover before we finish?
Jerimiah Ham [01:15:47]: I don’t know. Just have fun with it. It is, like I said, it’s free, it’s open source, it’s local. You can, of course, use cloud models or whatever, but it’s not right now. You need to have a real device or a simulator running on your own machine. You know, I’ve got future plans for CI servers and maybe network accessibility. I guess it’s easier with Android devices because you can ADB into a remote device. Use your best judgment.
Jerimiah Ham [01:16:23]: Have fun. You’re not gonna Don’t worry about whether you’re doing it right or not. Just try and play with it. That’s my biggest advice. And please reach out if you’ve got questions, concerns. The website is pretty bare bones, totally AI slop-generated front page. And actually, the whole thing is. It just wasn’t my priority.
Jerimiah Ham [01:16:52]: Check that out. But if it might, it’s easy to miss that there’s a lot of documentation on that quern.dev website too. So don’t be afraid to dive in there.
Alan Richardson [01:17:00]: And also the benefit is you as a human don’t have to read it because you’re going to be sticking up the Quern server. You’re going to have your coding tool. Your coding tool is going to go and read the documentation and do all that for you. So it will magically know what to do.
Jerimiah Ham [01:17:16]: Yeah, right, right, right. No, that’s not true at all. It’s a good idea. There’s still a lot of nuance in using this tool, and it’s a good idea to read through the documentation because it’s almost its own— it is its own knowledge base in a way that it’s giving you the list of gotchas and things you should watch out for, setup hints, best practices. Like if you’re going to use an Android emulator, always use one that’s of this certain class. So that you can have that proxy install automatically. You know, things like this, little tips and things about how to get the most out of your setup so that Quorum can use it most effectively.
Alan Richardson [01:17:58]: Yeah, so read the workflow guides.
Jerimiah Ham [01:18:00]: Yeah.
Alan Richardson [01:18:00]: That will give you an idea of what it’s doing, then all the rest is features. Cool. All right, thank you so much for turning in. I like it when people are building building stuff. I like it when people are building stuff that they’re releasing as open source. So, and what you’re doing is really interesting. If I was working with mobile apps, I would be trying to use this. So that’s as good a recommendation as I can give.
Jerimiah Ham [01:18:26]: Yeah. Well, thanks so much for having me. I was just delighted to be able to talk about this project. It was a real passion project. And it’s really neat to see To see it being noticed by other people and that other people see, um, the real value in it. That’s, that’s all I can ask for.
Alan Richardson [01:18:46]: Cool. All right, quern.dev, Q-U-E-R-N dot D-E-V. And, uh, thanks for turning in. And now I’ll click stop. Thank you. Bye.