Monocle Test Gen

Generate a Monocle fluent test

You are translating a natural-language description into a runnable pytest file that uses monocle_trace_asserter (the fluent API from monocle_test_tools) to (1) run an agent under Monocle instrumentation and (2) assert against the traces it emits.

Workflow

1. Load the test plan. Before anything else, look for a {project}_test_plan.md file in the working directory — where {project} is the project/agent name (e.g. lg_travel_agent_test_plan.md), falling back to a plain test_plan.md if no project-prefixed file exists. Use Bash ls/Glob to find it (e.g. ls *_test_plan.md test_plan.md 2>/dev/null). If found, READ it and treat it as the primary source for the three things below — the agent topology, exact agent/tool names, inputs, and validation scenarios are usually already enumerated there (e.g. in scenario tables grouped by ID). The user's request then selects WHICH scenario(s) from the plan to generate as tests. If no test plan file exists, fall back to extracting everything from the user's description.

1. Extract three things from the test plan (preferred) or the user's description. If any is missing or ambiguous, ASK before generating code — guessing here produces tests that compile but assert the wrong thing.

• Agent under test — the Python import path of the agent/crew/workflow object, AND which framework it belongs to (maps to an agent_type string — see table below).
• Input — the prompt/message/dict passed to the agent. Preserve it verbatim from the test plan / description.
• What to validate — one or more of: a tool was called (with optional input/output substring), an agent was invoked, the final output contains/equals a string, token/duration limit, an Okahu/BERTScore eval result.

1. Pick the framework runner string from the agent's framework:

| agent_type string | Framework | |---|---| | google_adk | Google ADK | | langgraph | LangGraph | | crewai | CrewAI | | llamaindex | LlamaIndex | | strands | Strands Agents | | openai | OpenAI Agents | | msagent | Microsoft Agent Framework / Semantic Kernel |

1. Decide sync vs async. Default to async + run_agent_async + @pytest.mark.asyncio unless the user explicitly says the agent is synchronous. Almost every framework runner in this repo is async.

1. Locate the file. Place new tests under test_tools/tests/integration/ with filename test__fluent.py. If a file matching the framework + agent already exists there, ADD a new async def test_... function to it rather than creating a duplicate file. Use Bash ls / Grep to check first.

1. Generate the test following the template below. Then run it (or at least pytest --collect-only the file) to confirm it parses, unless the user said not to.

Test template

import pytest
from monocle_test_tools import TraceAssertion
from  import 

@pytest.mark.asyncio
async def test_(monocle_trace_asserter: TraceAssertion):
    # 1. Run the agent under instrumentation
    await monocle_trace_asserter.run_agent_async(
        , "", ""
    )

    # 2. Assertions — one chained call per logical check.
    # Chaining narrows context: called_agent(A).called_tool(T) means "tool T inside agent A".
    monocle_trace_asserter \
        .called_agent("") \
        .called_tool("") \
        .contains_input("") \
        .contains_output("")

    # 3. Optional: end-to-end output / performance
    monocle_trace_asserter.contains_output("")
    monocle_trace_asserter.under_token_limit(1_000_000)

For a sync agent, drop @pytest.mark.asyncio, drop await, and call run_agent(...).

Fluent API cheat sheet

Assertion methods return self. Span selectors (called_tool, called_agent, and their does_not_* counterparts) narrow the span context for every subsequent assertion in the chain — that's how you say "this tool, called inside this agent, received this input."

Every assertion accepts an optional message="..." kwarg for a clearer failure string.

Run / load

| Call | Use | |---|---| | await asserter.run_agent_async(agent, agent_type, *args, session_id=None) | Drive an async agent | | asserter.run_agent(agent, agent_type, *args) | Drive a sync agent | | asserter.load_spans(JSONSpanLoader.from_json("traces/x.json")) | Offline: load pre-recorded spans instead of running |

Span selectors (narrow context)

| Call | Use | |---|---| | .called_tool(tool_name, agent_name=None) | Assert tool was called; optionally only inside that agent | | .does_not_call_tool(tool_name, agent_name=None) | Assert tool was NOT called | | .called_agent(agent_name) | Assert agent was invoked | | .does_not_call_agent(agent_name) | Assert agent was NOT invoked |

Input / output content

| Exact | Substring | |---|---| | .has_input(s) / .has_output(s) | .contains_input(s) / .contains_output(s) | | .has_any_input(*s) / .has_any_output(*s) | .contains_any_input(*s) / .contains_any_output(*s) | | .does_not_have_input(s) / .does_not_have_output(s) | .does_not_contain_input(s) / .does_not_contain_output(s) | | .does_not_have_any_input(*s) / .does_not_have_any_output(*s) | .does_not_contain_any_input(*s) / .does_not_contain_any_output(*s) |

with_comparer("similarity" \| "default" \| "bert_score" \| "metric" \| "token_match") overrides equality for has_* (not contains_*).

Performance

| Call | Use | |---|---| | .under_token_limit(N) | Total tokens across selected spans `) to find the registered names. ADK/CrewAI/LlamaIndex agents and tools have explicit string names that must match exactly.

• Don't omit @pytest.mark.asyncio on async tests. Without it, the coroutine is never awaited and the test silently passes.
• Don't chain unrelated checks on the narrowed context. After called_tool(T), an under_token_limit applies only to that tool's spans. If you want a workflow-wide token limit, call it on a fresh monocle_trace_asserter line (not chained).
• contains_* ignores the configured comparer. It always does substring matching. Use has_* + with_comparer("similarity") for semantic matching.
• Don't write declarative TestCase lists in a fluent test. Those are the @monocle_testcase style — a different skill. The fluent file should contain only async def test_... functions using monocle_trace_asserter.
• The pytest plugin is auto-loaded. If running outside the repo where monocle_test_tools is installed, the user's conftest.py must register pytest_plugins = ["monocle_test_tools.pytest_plugin"] — the repo root and test_tools/tests/integration/conftest.py already do this, so new tests in that directory need no extra setup.

After generating

1. Print the file path you wrote to.
2. Show the user the suggested command to run it, e.g. pytest test_tools/tests/integration/test__fluent.py -v.
3. Do NOT run the test automatically unless the user asked — most agent runs hit live LLM APIs and cost money.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: monocle2ai
• Source: monocle2ai/skills
• License: Apache-2.0

Install and usage instructions live in the source repository linked above.