Generate

The generate command creates structured evaluation criteria from your tests.

Generate Rubrics

Auto-generate rubrics from expected outcomes:

agentv generate rubrics evals/my-eval.yaml

This analyzes each test’s criteria field and creates structured rubric criteria with appropriate weights.

How It Works

1. Reads each test’s criteria
2. Uses an LLM to decompose the criteria into individual checkable rubric items
3. Assigns weights based on importance
4. Writes rubric criteria back under assertions: - type: rubrics

Example

Before:

tests:
  - id: quicksort
    criteria: Explains quicksort with time complexity and examples
    input: Explain quicksort

After running agentv generate rubrics:

tests:
  - id: quicksort
    criteria: Explains quicksort with time complexity and examples
    input: Explain quicksort
    assertions:
      - type: rubrics
        criteria:
          - Explains divide-and-conquer approach
          - Describes partition step
          - States O(n log n) average time complexity
          - Provides a concrete example