Trillium Updates: Adjusting models, prompts, and other configuration

For those who are new to my blog, I've been working on a side project that creates a weekly AI-generated newsletter that summarizes the bills that are debated and (potentially) become law in Ontario's provincial parliament.

With API credits being added to Claude plans, I decided to do some experiments with different model types that are used in the workflow. In particular, I looked at adding a more powerful LLM judge for the bill summary accuracy checker/revisor step of the newsletter generation.

Switching from Haiku 4.5 to Haiku 5.5

I’ve been waiting a long time for this model upgrade to be released by Anthropic so this was the first change that I made.

Below is a screenshot of what Haiku 4.5 generated for a bill that was added relatively recently:

One of my issues with this result is that this is a lot of words without much meaning. The ‘what’s happening’ is not useful beyond the first sentence.

Mentioning that there’s a new approval process is good, but who’s an “adoption practioner”? Who’s a “director”, and a director of what?

The most you’d be able to get from this is that “changes were made to the adoption process involving regulatory oversight”, which can be said in 1-3 sentences, not 4 paragraphs.

This is the Haiku 5.5 version:

It gets more detailed, and also uncovered that this bill goes beyond changes to the adoption process.

That ‘what’s happening section’ is still not the most pleasant to read as it's covering a lot of different topics in a single paragraph , but I was at least happy to see that it was a bit less vague than Haiku 4.5 in certain areas.

Adding Sonnet 5.5 as a judge and revisor

At the final evaluation step of the bill summaries in the workflow, I would notice in my logs that they would fail.

In the current workflow, a failure could be due to the generated copy using terminology that was deemed unclear, being longer than the limits I set for it, or not being accurate to the source material (e.g. missing content in the generated summary that should be there because it exists in the original bill).

For cost reasons, I had been using Haiku as both the summarizer and the judge, but this really isn’t the best choice practically speaking. For something that's judging content that’s less “black-and-white”, you want a model that’s more powerful than the one that’s used to generate the original output.

With cost being less of a factor for me now that I have access to API credits as part of my Claude plan, I updated the judge to be Sonnet 5.5.

Additional miscellaneous changes

Alongside the above I:

1. Increased the max token limit for the output from 4096 tokens to 16,000

4096 was originally chosen for cost reasons, but would result in truncation that was awkward to read and reduced the quality of the generated summary.

2. Removed the limits on what’s submitted as the “full” bill to the LLM judge

This was capped at 2,000 characters for, again, cost reasons.

Now the full bill text is submitted to both Haiku and Sonnet for summarization and the evaluation of the accuracy of the generated summary.

3. Increased maximum number of revisions from 1 to 3

In my initial implementation, I had only allowed 1 revision iteration in order to (again) manage costs on my end.

Increasing this number appeared to have the effect of making the copy of the text clearer, but there’s some non-determinism on the selection of topics to include (like the tow truck thing) that appeared in some versions and not others.

This was the final version after all of these changes:

You’ll notice reading through this that, in the “what this means for you” section around adoption, that a ‘Director’ is still not defined, but it’s a lot more concise than before (as are the other paragraphs in this section).

The “what’s happening?” section reads a bit more cleanly in my opinion, but this is an unpleasant run-on paragraph that the changes in prompts so far haven’t been able to shake. I’ll be looking into improving in the future.

Next steps

Aside from dealing with the persistent run-on paragraph I mentioned above, the Ontario Legislative Assembly will begin sitting again starting October 27th, so I’ll have lots of opportunities to test this out and iterate on changes.

If you’re interested in seeing the results, subscribe to the newsletter here. They'll be sent out on Fridays/Saturdays depending on when I get a chance to review the generated summary.

And, if you’re feeling extra generous, please send me your thoughts/feedback via email! 🙏

Until next time!