A Trail of Code - Boomstash

TLDR; When we use AI we don’t need to be as precious about architecture decisions as we used to. The principles we implement are important, the technology, less so, provided we have thought about exit strategies.

I wrote BoomStash.com as a spin off project from an internal tool I created to help me keep up to date with podcasts and videos. This post describes the thought processes involved in building and architecting it with AI.

What is BoomStash?

First a quick description of the end result, then I’ll describe the thought processes, approach and decisions.

BoomStash.com is an AI driven Podcast Summary site, but rather than ‘just’ summarize. I wanted to make the summary informative, actionable and provocative.

Informative - short summary of the content
Actionable - a few key takeaways
Provocative - what if you tried X?

I like the What Ifs? the most. I read them and think - “Nah”, but then I also have to think “why not?” - sometimes that’s easy because they might be ridiculous, and some might be completely unrelated to what I’m working on - but my head then translates them into my context.

And some I reject just because it doesn’t sit well with my approach, but that rejection helps me keep in touch with my values and approaches.

And sometimes… I add a new item to my todo list.

BoomStash has longer summaries that are behind a paywall - although for ‘Recommended Episodes’ I do show the Final Notes summary which is a longer list of summary points - they may not be as actionable as the take away summary points.

The paywall has the Final Notes and a Briefing Report summary.

The paywall is implemented using Patreon - rather than a shopping cart.

patreon.com/boomstash

And the different levels correspond to different summary formats available.

a Monthly magazine format PDF with summaries of the episodes split by category ‘dev’/‘business’
a 4 times a month - concise PDF which contains the short summaries and Final Notes
a 4 times a month - briefing PDF which contains the short, Final notes, and Brieifing summaries
access to the summaries online at BoomStash.com as well as on PDF

BoomStash Grew Organically

I didn’t set out to create a public Podcast Summary site.

BoomStash grew organically from my own needs.

My First AI Experiments

I think the first experiment that many people code with AI is a summarization tool.

If you have a podcast of your own then it’s pretty easy:

scan your folder of podcast recordings
run them through a Whisper based tool to get a transcription
run the transcription through an LLM with a “Summarize this”

I wanted to keep up to date with more Podcasts and Youtube Channels so I wrote a local tool that subscribes to podcasts and then transcribes and summarises them for me.

Initially this was just Software Testing podcasts because I never had time to listen to any.

My List of Software Testing Podcasts

And then I gradually expanded the list.

I also wanted to experiment with auto reporting so I hooked Pandoc into the mix to create PDF reports of the podcasts and released this to my EvilTester Patreon supporters.

I originally used Ollama and local models for summarisation but found OpenRouter to be more reliable and provided a larger context for summarization.

All of this was done using a local Python, hand-coded, Flask application. And I created various UI screens to let me experiment with different prompts and models.

And it stayed this way for a while.

Then I setup the Software Testing Roundup newsletter and created some additional hooks to output the Summaries in list format to include at the bottom of the newsletter.

For the Newsletter I monitor multiple platforms to find recommended content, but I wasn’t really keeping up with all the Youtube channels so I expanded my internal tool to handle YouTube and videos as well - I needed to do this because I kept hitting daily limits with the Notebook LM Summaries. So now I switch between Notebook LM and my internal tool for video summarization.

I find it much faster to: generate a summary - check if I’m interested - read the transcription - and then go to the video if it really seems worth it. Rather than scroll through Youtube and skip through videos.

AI Coding

When I started working with AI Coding I was able to expand my internal tool using AI, while also working on other projects, so I started adding additional capabilities that I wouldn’t have time for if I was just hand coding.

And then I thought… what if I made this more public. I could probably do that with AI coding and build on what I had.

But… I’d need a completely new platform. I couldn’t release my internal tool because the setup actions are too high, and really what I want to release are the artifacts of having gone through my summarisation tool.

What to release?

I’d already wrestled with my conscience around detailed summaries when releasing Testing Podcasts to Patreon.

My internal tool can generate very detailed minute by minute summaries, but that feels closer to a transcript and takes away much of the value from the podcast. My aim around the summaries is to pull out some overview, and hopefully add new value, to direct attention to the full source of value in the content.

I decided that I’d never release a transcript, and that’s why I focused on creating summary styles that would add value, rather than detract too much from the episode.

So I thought that if I ever went public then I’d have a ’less is more’ approach, and try to generate ’new’ content from the Summary - that’s where the Action Points and the What Ifs? came from. Most podcasts don’t end with a summary and contemplative message.

The public site is designed to add value to the reader of the summary, and point them towards the official content.

Architecture Decisions

I decided to have:

my internal tool do all the summarization work
the internal tool ‘send’ summaries to a public server system via an API
the public server system would then render the summaries
the public server would have a front end with some sort of repository for the summary data

Future requirements would revolve around notifications, authentication to gate summary levels, possibly admin scope… but I would architect that in later. The most important thing initially was viability.

Viability considerations:

could AI build this?
could this be hosted for free?
could I use this to learn something fast?

I almost went with a static site. I use static site builders for most of my web sites. But I wanted to experiment with a more dynamic architecture and explore the newer (non-aws) startup type cloud providers.

e.g. Railway, Vercel, Loveable, etc.

I was already using Railway for TestPages and API Challenges. And I could probably add that to my existing plan limits without breaking the bank, but that wouldn’t really let me experiment with AI Coding tooling and other hosting options.

Architecture wise I decided to have:

front end
REST API for admin publishing content to system
Database

I was tempted to have an S3 style file storage for the summaries but I felt that was too close to a static site and I’d done that before with API Challenges, so I wanted a more traditional architecture.

I then looked around at providers.

Vercel seemed to have a free plan that would work for what I needed to get started, and since Next.JS seems to be the default tech for Vercel, I thought I’d try it.

I’d also heard a lot about SupaBase - particularly its support for authentication - and it had a free plan that I could start with.

I thought with both technologies I could see how quickly I could get started with AI tooling. I didn’t really try to optimise at an architectural level because I couldn’t imagine having much traffic at launch and thought I’d re-architect if necessary if it scaled.

I did decide on a Front End/API split from Database to allow me to explore different platforms. If necessary I could switch to a different database provider, move to an S3 style storage for content if the DB grew too big and just create indexes in the DB, and possibly move everything into the server platform as flat files if necessary.

And worst case scenario - I use a static site.

My main consideration was ‘get something live’, ‘have a traditional, rather than optimised’ architecture, ‘keep it flexible’. And I didn’t worry too much about hosting decisions because I can change them easily.

I was able to create a functional integration between Vercel and Supabase, with an API to populate data, and rendering through Next.js in a few days using AI coding tools.

This speed was an important validation. We don’t have to overthink and over-design the initial version.

If we do decide to migrate to something else then we use the existing system as the spec for the next one that we have AI code for us.

Defer Decisions

Working with Agile we are used to deferring fixed and hard to change decisions until we have learned as much as possible - valuing flexibility to change and learn.

Now, when using AI, even the decisions that were hard to change, are becoming quite easy to change.

If we want to change from React to Vue - the AI can probably do that in the background in a few hours.

As a result, I view the choice for Next.js as a ‘first step’.

If feels a little sluggish at times, but I haven’t done anything to optimise the data loading beyond CDN caching. At some point I will try to optimise further, and that might require moving more content on to Vercel storage rather than loading everything from Database.

Thing is… I don’t feel that I ‘have’ to stick to Next.js, or Vercel, or SupaBase. They are now just implementation details, and when I want to experiment I can conduct multiple experiments at the same time.

The implementation doesn’t feel like a fixed decision any more, it feels like a tactical way to deliver the business value.

Optimisation

I did have to optimise in order to keep within the free plan on Vercel. So this was a business priority.

Vercel and Supabase have built in analytics and observation tools and I could see very quickly that I needed to do some sort of optimisation if I was going to keep within the Vercel free plan.

This came down to using the built in Vercel CDN caching more. That brought the concerning limits down.

It will primarily be the metrics and costs that drive initial optimisation.

Authentication and Authorisation

I chose SupaBase because it has authentication and authorisation built in, and can probably do a better job of this than I can - and can certainly do it faster. So I didn’t have to put much thought into implementing security. Just some thought into what the requirements were.

Security is much more of an architectural decision when you are building the implementation from scratch e.g. if you’re building a Kubernetes cluster on AWS you’re going to have to put much more architectural thought into the security.

I just dumped all the data into SupaBase, and used SupaBase configuration for authentication and then Row Level Security rules for authorization of different summary levels.

SupaBase has a lot of OAuth configuration options so I deferred thoughts about ‘how will I let people login’ until later.

I started looking at various payment providers: Stripe, Paddle, etc.

Then decided to use Patreon instead.

I’d used Patreon’s OAuth features before when building TesterHQ to create a Patreon login section for my basic hand crafted online course system and knew that I could handle different support tiers to provide different levels of access.

Content Types and Platform Syndication

Patreon also offered some other benefits for the type of site I was building.

Patreon:

sends out announcement emails to supporters - so no need to integrate an email sending system.
hosts PDFs as attachments on posts - so no need to manage S3 storage or pay for ingress/egress/storage of the PDF reports

I actually built my ‘what do you get if you sign up’ levels the way I did as a result.

The most expensive part of the site is the online hosting and people viewing summaries online.

And I’m not even sure if that is how people would prefer to access them.

Releasing the content as PDFs is essentially ‘free’ to me because Patreon hosts it. So the online summary access moved into the more expensive tier.

I wanted to have a low tier that still provided a PDF benefit so I thought about creating a magazine format PDF - I read most of my magazines as online representations now, so I thought that would work. And if I could automatically generate a 60 - 100 page magazine format for the summaries every month then that wouldn’t overwhelm the reader with content.

And people who want to consume summaries more frequently in more detail can use the 4-times-a-month more detailed content.

And… everyone needs a free tier.

So the individual episodes, weekly summaries online, RSS feeds, and summary lists emailed through Patreon would cover that.

I was tempted to use substack since that was the système du jour for a while, but it doesn’t offer an upload API, nor does it offer OAuth, and if you charge then you are the merchant of record and have to handle EU VAT - so that all seemed like a bad idea.

But… I do syndicate the 4-times-a-week summary list reports to substack.

Reports

PDFs are all generated locally and distributed to the various sites.

Examples downloadable for free from Patreon:

All of this output is generated from Pandoc with some custom templates.

Not Very Enterprise

I do find that teams go too far, too quickly.

Creating Authentication layers for internal systems that don’t have much functionality because we spent all the time on the authentication layers.
Creating Admin features for managing users, when there are 4 users and this could be handled with SQL inserts.
Database when a file system could work.

AI can help with this now because we can go quickly and not sacrifice the business value by prioritising tech features above business features. But… every layer of tech and functionality increases the amount of testing required and increases the perceived permanence of the implementation.

Identifying the fastest, easiest and most cost effective way to delivery the business value without unduly increasing risk, should always have been one of the core decision making processes around Enterprise development.

But… enterprise has typically been: long drawn out ‘design’ processes, highly technical infrastructure, and slow business value delivery. Hence the need to ‘migrate’ a million spreadsheets when the system is live - because that’s what the business had to use to survive until the system was ‘ready’.

AI makes it much easier to get something working, without feeling that you have to stick to something permanent. We do have to think through the business risks and choose technology that minimizes that risk - some enterprises ‘standardize’ in order to avoid business risk and by doing so reduce the flexibility for experimentation. Unfortunately, many business processes are experimental, not fixed, adaptive, and as soon as we build a fixed and permanent system on the back of that we limit the business responsiveness.

An Ongoing Experiment

BoomStash.com is a spin-off experiment. Creating a ‘bigger’ architecture than strictly necessary for a summary hosting site. But offloading much of the implementation to AI and existing technologies.

The ‘core’ of BoomStash - my internal AI subscription and summarisation engine- is what I really experiment with:

Better and faster transcriptions - I have 5 different transcription approaches to see which works best for different podcasts.
6 different low cost LLM models that I feed summary prompts through
3 different ’large’ summary approaches that I’m comparing
different download approaches to handle CDNs, bot protection, etc.
and more

At the moment I’m still central to the process and read through every summary before it goes live - primarily because the output is a result of me having learned from the podcast episode (not every podcast episode goes live, its an opinionated filtered set of summaries).

This doesn’t feel too onerous because part of my learning process is reading through all the summaries - I’d rather do this than read a lot of blogs and newspapers - it’s one way I keep up to date.

I’m almost at the point where summaries will go live without intervention, and then the human review process would be to retract those that I think add less value, and re-generate those that have issues.

TesterHq is a secondary spin off from this - a much simpler static site implementation which only handles the Software Testing Podcasts - I don’t filter these - all Testing podcasts that I know of are summarised to that site.