Testing Skills is Vital: Markus Oberlehner Talks About Developing Storyblok’s Agent Skills

Markus Oberlehner is a Senior Developer Experience Engineer at Storyblok. Focused on the ongoing upgrades and improvements to Storyblok’s OSS ecosystem, Markus and his team leave no function unturned in their quest to make Storyblok's SDKs and packages joyful for developers.

In this interview, Markus talks about the newest addition: agent skills, a set of open source AI tools designed to optimize how agents interact with Storyblok using the MCP server and CLI:

• storyblok-use-storyblok: router skill and foundation for the entire Skills package
• storyblok-use-cli: guidance for when and how to use the Storyblok CLI
• storyblok-use-mcp: instructions for using the Storyblok MCP server
• storyblok-model-content: best practices for content modeling
• storyblok-build-frontend: framework-dependent information about best practices for Storyblok frontends
• storyblok-implement-figma: from a Figma design to Storyblok schema

Read on to learn about the development process and the focus on quality assurance, and find out how to test non-deterministic systems and feed the test results back into further development.

Skills are, in essence, Markdown files that provide additional information to models. How does one make sure that this additional information actually improves anything? It can be quite easy to fall victim to wishful thinking when building skills.

Subjectivity really is the bane of developing AI tools, and automated testing and evaluation for something like agent skills is absolutely indispensable. That's why we didn’t just write a few lines of Markdown for the Storyblok skills package but also built an entire framework that tests skills-enhanced agent performance across many scenarios and models. We build test cases that we prompt agents to handle, measure their output quality and problem-solving capabilities, and iterate on the results. Think of it as integration tests for AI tooling.

This sounds like something that every serious skills writer probably does to some extent. Is there a thriving ecosystem of eval frameworks, and do you use any established tooling?

The ecosystem is in its early evolution, and there is currently no established off-the-shelf framework that fits our needs. So we just built our own.

Can you tell us about the general architecture of the eval setup?

The broad strokes are pretty similar to standard integration tests. For example, there’s an isolated environment the evals run in, ensuring results aren’t affected by any external system state. Then there are multiple scenarios, which you can think of as analogous to a single test case. A scenario is built from an initial setup and a prompt. Taken together, they yield a result that we can evaluate. The entire configuration is straight code—no domain-specific languages or other unnecessary bells and whistles.

Developing tests for AI tooling can be challenging. First, there's the general non-determinism of it all, and then you have to contend with a multitude of models. How do you deal with that?

It's true that in theory there's an infinite list of models to test against, and it's obviously not feasible to run the evals with each and every one. But this isn’t at all a unique problem. Like with unit tests or integration tests for other software, there comes a point when you, as the developer, have to put your foot down. Testing ever more edge cases is possible, but buys you nothing. Instead, you just get buried in code, and the time to run the entire suite—and in the case of LLMs, also the costs—eat you alive, all for diminishing returns. That’s why we test with a selection of models that we're confident would give us the insights we need. No more and no less.

An AI coding agent could use (or decide not to) any number of tools and generate code that is more or less adequate. What are the actual test criteria, and how do you evaluate them?

The test criteria are part deterministic and part non-deterministic. An important aspect of the skills is to get agents to use the Storyblok MCP server when the task calls for it. Verifying that an agent uses the MCP server in a given scenario is easy to detect and entirely deterministic. On the quality side, spinning up an LLM judge to evaluate something like code quality is the obvious play. When the deterministic tests are green, we count that as a pass and record the quality assessment for later review.

But a single pass with several moving pieces in a non-deterministic setup probably isn’t enough?

Right. A single pass really doesn’t tell us much. We always perform multiple runs to make sure the skills work, and misbehaving agents are strict outliers. Running the evals means running every scenario with every model multiple times. This also allows us to track the stability of agent behavior, and we can then go the extra mile and visualize the current state plus the eval history in a nice little dashboard.

How does this setup help further development of the skills themselves?

There’s a kind of eval-driven-development that we use. Every run of the eval suite gives us valuable information on how well the agent performs. Besides the LLM judge's assessments, we also use the agent’s transcripts to find areas to improve. This is pretty important to make sure that the skills actually do work. You’ve probably experienced how even without skills, today’s agents have a pretty good shot at figuring out most software development challenges—but this relies on trial and error, which in turn invites inefficiencies at best and subpar results at worst. Skills should put agents on the right track from the start, and if that fails, it becomes obvious in the transcript. Any time an agent goes on a tangent instead of reaching for the right tool, is an opportunity for improvement: either by tuning the skill’s Markdown instructions or by giving the agent a different kind of guidance.

What do you mean by “a different kind of guidance”?

Like a ready-to-use script. As an example, uploading assets to Storyblok involves a non-obvious procedure. Agents can usually figure it out but could easily do something subtly wrong. When we built the storyblok-implement-figma skill, agents struggled with assets. Even when they managed to upload files, they continued writing new one-off scripts for this purpose. We found that the best way to reduce that kind of behavior was to include properly tested scripts in the skills. This helps the agents be more reliable and efficient and introduces a healthy dose of determinism for exactly the processes that need it the most.

What’s next for evals?

The evals will obviously continue to grow and evolve alongside the skills themselves. I’d also like to add more models and model providers to the scenario matrix, and keep an eye on the eval framework ecosystem. Our current setup works really well, but this is a rapidly evolving field that I want to stay on top of.

Get the skills now!

From frontend code to content modeling, Storyblok’s agent skills turn your coding agents into seasoned Storyblok developers. Install them now and help your super-intelligent AI assistants wield Storyblok’s SDKs, CLI, and MCP server.