# Multimodal Ui Flow Analyzer

> Analyze annotated UI screenshots and markdown documentation to generate agent-consumable UI flow specifications. Use when processing web app UI flows described via markdown + screenshots into structured, automation-ready knowledge.

- **Type:** Skill
- **Install:** `agentstack add skill-boweneos-ui-flow-agent-skills-ui-flow-agent-skills`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [boweneos](https://agentstack.voostack.com/s/boweneos)
- **Installs:** 0
- **Category:** [Web & Browser](https://agentstack.voostack.com/c/web-and-browser)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [boweneos](https://github.com/boweneos)
- **Source:** https://github.com/boweneos/ui-flow-agent-skills

## Install

```sh
agentstack add skill-boweneos-ui-flow-agent-skills-ui-flow-agent-skills
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Multimodal UI Flow Analyzer

This skill enables you to analyze static web app UI flows described via **markdown + annotated screenshots** and output **agent-consumable knowledge** for downstream AI code agents.

---

## When to Use This Skill

Activate this skill when:
- Processing UI documentation that includes annotated screenshots
- Converting visual UI flows into structured automation specs
- Generating test automation guidance from UI walkthroughs
- Creating machine-readable UI interaction sequences

---

## Core Workflow (7 Steps)

### Step 1: Normalize Input Markdown

Before analysis, ensure each UI step follows this structure:

```md
## Step N: 

**Intent:**
What the user is trying to accomplish.

**User Action (Text):**
Plain-language description of the interaction.

**Visual Reference:**

**Visual Annotations:**
- Box / arrow / highlight descriptions
```

If the input doesn't follow this format, restructure it first.

---

### Step 2: Apply Vision-Aware Analysis Rules

When analyzing screenshots:

1. **Identify interactive UI elements** (buttons, inputs, menus, links)
2. **Map visual annotations** (boxes, arrows, highlights) to UI elements
3. **Infer user intent** from both text and visual cues
4. **Ignore decorative elements** that are non-interactive
5. **Assume static UI** (no animations or runtime state changes)

---

### Step 3: Process Each Step Atomically

Analyze **one step at a time**, never the entire document at once.

For each step, extract:
- The UI element being interacted with
- Its visual characteristics and location
- Its technical role in the web application
- Preconditions and resulting state

---

### Step 4: Treat Annotations as Ground Truth

**Annotation Priority Rules:**
- Highlighted areas are authoritative targets
- Prefer annotated elements over textual ambiguity
- If text and image conflict, **image evidence wins**

Map annotations explicitly:
```json
{
  "annotation_mapping": {
    "red_box": "Primary action button",
    "arrow": "Cursor movement direction",
    "highlight": "Target input field"
  }
}
```

---

### Step 5: Generate Structured Output

Produce output in the canonical format (see templates in `assets/templates/`).

**Per-Step JSON Format:**

```json
{
  "step_id": "step-N",
  "intent": "Description of user goal",
  "action": "click|type|select|scroll|hover",
  "ui_element": {
    "type": "button|input|link|menu|dropdown",
    "label": "Visible text or aria-label",
    "visual_location": "Position description",
    "identification_strategy": [
      "visible text equals 'X'",
      "role=button",
      "data-testid='element-id'"
    ]
  },
  "precondition": "Required state before action",
  "resulting_state": "Expected state after action"
}
```

**Flow Markdown Format:**

```md
# UI_FLOW: 

## Metadata
- App: 
- Flow Type: Static UI Interaction
- Source: Annotated screenshots + human-authored text

---

## Step 1
**Intent:** 

**Action:**
- type: 
- target:
  - role: 
  - text: ""
  - location: 

**Preconditions:**
- 

**Postconditions:**
- 

**Automation Notes:**
- 
```

---

### Step 6: Add Automation Hints

For each step, include:

1. **Stable DOM selectors** (prefer semantic)
   - `role=button` + visible text
   - `data-testid` attributes
   - `aria-label` values

2. **Brittle selectors to avoid**
   - Pixel-based positions
   - Absolute CSS selectors
   - Dynamic class names

3. **Wait conditions**
   - Elements to wait for before action
   - Loading states to handle

---

### Step 7: Validate the Output

Before finalizing, verify:

- [ ] All steps have clear preconditions
- [ ] Step ordering is logical and complete
- [ ] No ambiguous UI references remain
- [ ] Each action has a defined resulting state
- [ ] Selectors are stable and semantic where possible

---

## Output Templates

Use templates from `assets/templates/`:

| Template | Purpose |
|----------|---------|
| `step-output.json` | Single step structured output |
| `flow-output.md` | Complete flow specification |
| `automation-hints.md` | Test automation guidance |

---

## Example Interaction

**Input:** User provides markdown with annotated screenshot showing a "Create Project" button highlighted with a red box.

**Analysis Process:**
1. Parse step structure from markdown
2. Identify red box annotation → maps to button element
3. Extract button text: "Create Project"
4. Determine location: "top-right of main content area"
5. Infer action type: click
6. Define precondition: "User is on Projects dashboard"
7. Define postcondition: "Project creation modal opens"

**Output:**
```json
{
  "step_id": "step-2",
  "intent": "Create a new project",
  "action": "click",
  "ui_element": {
    "type": "button",
    "label": "Create Project",
    "visual_location": "top-right of main content area",
    "identification_strategy": [
      "visible text equals 'Create Project'",
      "role=button"
    ]
  },
  "precondition": "User is on Projects dashboard",
  "resulting_state": "Project creation modal opens"
}
```

---

## Edge Cases

### Ambiguous Annotations
If multiple elements are highlighted, process them in visual reading order (top-to-bottom, left-to-right).

### Missing Screenshots
If a step lacks a visual reference, flag it and proceed with text-only analysis. Note reduced confidence in output.

### Complex Multi-Element Interactions
For drag-and-drop or multi-select, describe both source and target elements with separate identification strategies.

### Dynamic Content
If the UI shows dynamic content (lists, tables), describe the interaction pattern rather than specific instances.

---

## Constraints

- **DO NOT** assume backend logic or API behavior
- **DO NOT** infer state beyond what's visible
- **DO NOT** generate pixel coordinates as primary selectors
- **ALWAYS** prefer semantic selectors over structural ones
- **ALWAYS** document uncertainty when present

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [boweneos](https://github.com/boweneos)
- **Source:** [boweneos/ui-flow-agent-skills](https://github.com/boweneos/ui-flow-agent-skills)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-boweneos-ui-flow-agent-skills-ui-flow-agent-skills
- Seller: https://agentstack.voostack.com/s/boweneos
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.