Now loading...
Anthropic has rolled out significant upgrades to its Skill-creator tool, empowering users to develop evaluations, execute benchmarks, and ensure their custom AI skills remain effective amid rapid advancements in language models. These features are immediately accessible through Claude.ai and the Cowork platform, as well as via a dedicated plugin for Claude Code and directly from the company’s GitHub repository.
Since introducing Agent Skills back in October, Anthropic has observed that the majority of creators come from domain expertise backgrounds rather than technical engineering roles. These individuals often possess deep knowledge of their specific processes but lack simpler methods to verify if a skill performs reliably with updated models, activates appropriately, or enhances output after modifications.
The latest enhancements to Skill-creator aim to instill greater assurance in skill development by incorporating elements of rigorous software engineering practices, such as testing and iterative refinement, all without demanding any programming expertise.
Custom skills typically divide into two main types. The first, known as capability uplift skills, assist Claude in performing tasks that the underlying model might handle inconsistently or not at all. For instance, the company’s document creation skills exemplify this by embedding specialized techniques and patterns that yield superior results compared to basic prompting. The second category, encoded preference skills, outline sequences of actions that align with a team’s established workflows, even if the model can manage each step independently. Examples include guiding through NDA evaluations based on predefined standards or compiling weekly reports by pulling data from multiple sources.
This categorization is important because it influences testing needs. Capability uplift skills could diminish in relevance as models grow more capable, with evaluations helping detect when that occurs. Encoded preference skills tend to endure longer but rely on accurately mirroring real-world processes, which evaluations can confirm. In both cases, systematic testing transforms intuitive skills into verified ones.
With the new tools, Skill-creator simplifies crafting evaluations, which function as targeted tests to confirm Claude’s responses align with expectations for specific inputs. Users define test scenarios, including any necessary files, outline desired outcomes, and the tool assesses compliance. This mirrors software testing but remains accessible to non-coders.
Take the PDF handling skill, for example: it once faltered with non-editable forms, requiring precise text placement without field guides. Evaluations pinpointed the issue, leading to a solution that ties positioning to detected text locations for better accuracy.
Evaluations serve dual purposes in maintaining skill quality. They first identify regressions, where evolving models or systems cause previously functional skills to underperform, providing alerts before disruptions affect operations. Second, they reveal when base model improvements render certain capability uplift skills redundant, as the model passes tests unaided, indicating built-in assimilation of the skill’s methods.
A new benchmark mode standardizes these assessments, measuring success rates, processing times, and resource consumption after model changes or skill tweaks. Results and evaluations remain under user control, storable locally, integrable into monitoring systems, or incorporated into continuous integration pipelines.
To accelerate testing, Skill-creator now employs multi-agent support, launching separate instances for parallel evaluations in isolated environments. This prevents context spillover between tests and delivers quicker, cleaner metrics on tokens and duration.
Additionally, comparator agents enable unbiased A/B testing, pitting skill variants or enabled-versus-disabled configurations against each other. These agents evaluate outputs anonymously to determine tangible improvements.
Beyond output validation, reliable activation is key, especially as skill libraries expand. Vague descriptions risk unnecessary activations or misses, so Skill-creator now refines them by analyzing against example prompts and proposing adjustments to minimize errors. Testing on Anthropic’s document skills showed enhancements in activation accuracy for five of six publicly available ones.
Looking forward, as AI models advance, the boundary between detailed skill instructions and high-level specifications may fade. Currently, skill files offer step-by-step guidance on execution; in the future, simple natural-language descriptions of desired results could suffice, with the model handling implementation. This evaluation system represents progress toward that goal, where defining success criteria might eventually define the skill outright.
Users can dive in right away on Claude.ai or Cowork by prompting Claude for Skill-creator assistance. For Claude Code, the plugin is available on GitHub, alongside the full repository.
