LLMs as a primary user of APIs and SDKs

I have an upcoming talk at the “J on the Beach” conference about experiments in tooling within Sentry's Python SDK to better support AI-assisted developers, and to help maintainers, such as myself, better manage the shifts in what type of work needs to be done in response to the influx of issues and pull requests that are AI-generated or AI-assisted.

The question at the heart of this talk that I’m exploring is: what happens when your primary audience is no longer just humans? What changes when large language models (LLMs) and AI agents become a user “persona” one has to consider?

What has changed?

For one: new tooling had to be introduced. One of the more impactful changes we made was the creation of AI Skills that create personalized experiences for each individual project in setting up and configuring the SDK.

These AI Skills are easier to create, maintain, and use than the programmatic command-line (CLI) setup “wizards” of the past which, by their nature, can't reason about the project being targeted for setup and adapt to its unique needs.

Skills are also more context-efficient. They're conditionally loaded based on the criteria outlined in the front matter of the skill file.

And because Skills can offer workflows with context that the training data or doc strings for individual functions/methods can’t always offer, they can enable LLMs and agents to achieve their goals more effectively and deterministically than if one were to rely entirely on user prompts and the model's training data.

On the administrative side as the maintainer of an open source project: it’s resulted in a shift of where I spend my time.

For instance, it’s become easier to create reproductions of issues that are opened by external users.

This has saved significant amounts of time that previously would have been spent either, in the case where the initial description is too vague, trying to get additional details from the user, or trying to get a sample project configured just right in order to reproduce the behaviour described by the user.

However, the reduced barrier to creating issues and pull requests means that there’s more for me to review than there was previously, leading me to spend more time triaging and code reviewing.

What hasn't changed?

The need for public APIs with stable contracts that have predictable behaviours.

And what I mean by "stable contracts and predictable behaviours" here is that changes to the API follow backwards-compatibility based on semantic versioning. No breaking changes in a minor or a patch version shenanigans (I'm looking at you OpenAI and Pydantic Python packages 👀).

This has renewed importance with this new “user” because, when LLMs and AI agents go to interact with SDKs/APIs, they rely on:

• their training data;
• documentation offered by the SDK on the public API methods within it; and
• any additional context provided (e.g: user prompts, Skill files, retrieval-augmented generation (RAG)).

Which means that the more significant changes that happen in minor/patch versions, the more maintenance is required to ensure that the LLMs have the additional context they need to use the methods on the SDKs and APIs correctly.

And the more context you need to give, the higher the cost of using an LLM with the SDK/API because of the increased token consumption related to that additional context.

Final thoughts

There’s more to consider when thinking about AIs as an end-user that are user/project-dependent that I didn't touch on here.

AI accessibility based on model and context window sizes may come into play as not everyone can afford or has access to the frontier models. There may even be the question of whether LLMs and AI agents should even be treated as a first-class "user" in the first place.

But this has been an interesting and fun thread to pull on for me, and I can’t wait to talk about it more with folks in Spain next month.