From iterative prompting to software

A pattern keeps showing up in my agentic engineering flow:

1. Iteratively prompt the model in an attempt to reach a solution process or to expose blockers
2. Stop paying the model tax for repeated tasks
3. Encode the context into repeatable software

I recently ran into this multiple times in one of my side projects, which analyzes and visualizes gpx files for cycling activities.

To make the project useful, I needed to find a repeatable way to sync new activities to it.

Start with the exploration

My first idea was to export the activities, as gpx files, from my Strava profile. But was this feasible or did I need to explore other options? What are the trade-offs of this approach? Where are the challenges?

I prompted the agent to export the latest activity from my profile. To answer these questions, I guided the agent via an exploratory discussion through challenges and decisions, until it finally managed to download the activity files.

As a result we resolved:

• How can we get an authenticated session to access the activities?
• Once we're authenticated what's the quickest way to find the activities?
• Once we found the activities, how can we programmatically download the gpx files?
• What kind of activities do we want to export? How can we filter and clean the data?

Once the agent conversation reached a point where we were able to extract the files, the exploration phases ended.

We knew:

1. It is feasible to programmatically extract the files we need
2. What we needed to know about the platform to achieve our goal
3. Trade-offs in the practical process
4. Refined requirements

Then stop prompting

Clearly prompting the agent each time to import new activities was not a long-term solution. This process would be too slow, too costly, and even worse, the indeterminism of LLMs would always result in a slightly different process and results, but...

We were now in posession of a context with all necessary decisions and steps. Since the agent can only interact with its environment via programmatic tools, it was clearly possible to turn the steps into a reusable script. At this point it was time to enter plan mode to specify the the program we needed. As expected, the plan with this context needed only minor adjustment and the resulting system worked as expected.

The program had some clear advantages over letting the agent perform these steps. Not only is it much faster and cheaper, but it's also highly repeatable, auditable and could easily and cheaply be evolved and refined

The Point

This side-project already required such a flow for multiple features and it turns out to be a useful pattern.

Turning the process into code turns the slow and indeterministic process of the LLM into something that can be cheaply and quickly repeated. It also provides the advantage that the steps can easily be verified, audited and evolved.

Of course, not all process can or should be approached in such a fashion, but it is useful to consider if we are in a situation where this pattern could help.