Trillium Update: Lessons from Steering Claude and Building on Its API

Since sharing what I've learned in my initial post about Trillium a few months back, I've gotten much further with it.

There was about a month of legislative activity (~February/March) that gave me a lot of opportunities to test the progression of bills and see what summaries I would get as they moved along in the legislature.

I've also continued to develop this primarily through steering/prompting AI so that I can continue to see where it excels and where it falls short in a low-stakes environment. And maybe because this is a project with an AI workflow as part of its core functionality, I got to witness some behaviour from the AI that I normally don't see or experience in my day job.

Where I've gotten the project to today:

• it runs daily cron jobs every day to check for new or updated bills
• runs a weekly task to generate a newsletter that stays in draft until I have a moment to review it before sending it
• creates a summary that gives a decent amount of detail without being overwhelming (even when handling omnibus bills)
• Running costs are negligible (no cloud resources, runs entirely on a laptop)

So now that this project has reached another milestone, I though I'd share all the lessons I've learned along the way, along with some annoying "gotcha"-s I ran into too.

There's too much to share for a single post so for this one, I'm going to focus on the things I learned involving Claude and "AI-driven" development.

The Price of Getting Started - Claude API Rate Limits

Since this project requires interacting with an AI provider via an API, I had to shell out for some usage credits beyond my Claude Pro plan.

I initially put in $5 USD to get started, but then quickly ran into rate limiting issues related to the maximum input tokens per minute (ITPM) when I tried to experiment with results using Sonnet.

Although the tiers/limits have since changed (at the time of writing, around the first week of July) this is what I was working with:

My initial experiments with individual bills were pushing me just beyond the 30K ITPM for Sonnet, and the creation of the initial aggregated summary was more in the 80K range (and sometimes as high as 120K), so even using a smaller model like Haiku wasn't enough for my purposes.

Which meant that I had to shell out another $35 USD to reach Tier 2 and get much more allowances with respect to the ITPM:

Having to pay almost $60 CAD for a relatively small, proof-of-concept side project was extremely annoying to say the least because I didn't want to spend that much money in AI credits at that moment.

I think it's also worth noting that, for those just getting started in their careers, this sort of thing is a financial barrier that can prevent people from experimenting with (and therefore learning) how to use AI effectively.

However, as I mentioned a moment ago, Claude has since changed their rate limits, so the problems I ran into a few months ago should hopefully not happen to others going forward, at least for smaller projects that are similar in nature to Trillium:

This experience also highlighted for me just how meaningful of a difference the cost can be across models.

Given what I was using the AI for (summarizing text), the much higher cost of Sonnet compared to Haiku and the difference in terms of the outputs in the early experiments (minimal), I decided to use Haiku going forward in Trillium.

Random, unexpected delights

There were a couple of instances where Claude genuinely surprised me with its behaviour - once where I had accidentally made a mistake in the prompt, and another where it suggested a meaningfully good improvement on how I was sending API requests to it to improve the quality of the aggregated newsletter while minimizing token usage.

In the first case - I was messing around with the prompt that resulted in the text of the bill not being available for summarizing and aggregation.

Rather than completely fabricate the contents of the bill, the AI responded with:

I appreciate you providing this context, but I need to let you know directly: I cannot write this newsletter section because no actual bill content has been provided.
All four bills (99, 84, 100, 98, and 97) are missing their legislative text or substantive descriptions. Without this source material, I cannot ethically write summaries for a public newsletter—doing so would require me to speculate or invent details, which would be misleading to readers.

Good job Claude (or more specifically - good job Anthropic folks). I'm curious to know in what other contexts the AI will come to similar conclusions and also push back like this without any sort of guardrails within its prompt(s).

In the second case - I noticed two issues:

• longer aggregated newsletters were getting truncated; and
• random failures were being raised by Pydantic because of a missing required property on a data model (summary).

I prompted Claude to investigate both issues and recommend some fixes. I had a theory about the second problem but wasn't initially sure about the first.

For the truncation issue - it turned out that I didn't have a high enough value for the maximum amount of tokens (max_tokens parameter on the Messages.create endpoint) before the response would get cut off.

I was overly conservative here when I was attempting to use Sonnet earlier on instead of Haiku in order to make sure I didn't accidentally spend too much on tokens with my test runs.

For the random failures - I suspect that the results were so good here because it essentially amounts to Claude looking at its own documentation, but I'm nonetheless happy with the outcome.

I had failed to tell the model to set the value of summary to the results of a specific prompt, and the model had lumped summary with another variable on the data model (implications), leaving summary as None .

As a result, the following error would occasionally be thrown by the Pydantic validation:

{"detail":"Summarizer pipeline failed: 1 validation error for SummarizeResponse\nimplications\n Field required [type=missing, input_value={'summary': '**What This ...ty behind closed doors'}, input_type=dict]

So where the prompt previously read like the following:

Provide:                                                                                                                                               
- 1. A clear, plain-language summary of what this bill does (no more than 2-3 short paragraphs). Avoid legal jargon — explain it as you would to a neighbour.

it turned into:

Return your response by populating BOTH fields of the structured output:

- Set `summary` to a clear, plain-language summary of what this bill does (no more than 2-3 short paragraphs). Avoid legal jargon — explain it as you would to a neighbour. Do NOT include implications here.

To lesson learned on my end: enforcing structure and specifying what outputs live in which variables within a prompt will help avoid this particular type of unexpected surprise going forward.

To wrap up

I hope to eventually experiment with other models (Sonnet, Opus, etc.) and across different providers (GPT, Mistral, and an open model) and see the differences in the results.

Because of the early rate limits and not wanting to go bankrupt providing more money up front to Anthropic when the old rate limiting rules were around, I decided to not do that at this time, and instead focus on other parts of the project.

There's more learnings that I plan to share but I'm going to write about them in future posts since this one is getting a bit lengthy already.

Thanks for reading this far, happy coding! 👩‍💻