Dataflows Consumption Cli

> Update Check — ONCE PER SESSION (mandatory) > The first time this skill is used in a session, run the check-updates skill before proceeding. > - GitHub Copilot CLI / VS Code: invoke the check-updates skill. > - Claude Code / Cowork / Cursor / Windsurf / Codex: compare local vs remote package.json version. > - Skip if the check was already performed earlier in this session.

> CRITICAL NOTES > 1. To find the workspace details (including its ID) from workspace name: list all workspaces and, then, use JMESPath filtering > 2. To find a dataflow by name: list all dataflows in the workspace and filter by displayName client-side — there is no server-side name filter > 3. getDefinition is a POST, not GET — even though it reads data

dataflows-consumption-cli — Dataflows Gen2 Consumption via CLI

Table of Contents

| Task | Reference | Notes | |---|---|---| | Finding Workspaces and Items in Fabric | [COMMON-CLI.md § Finding Workspaces and Items in Fabric](../../common/COMMON-CLI.md#finding-workspaces-and-items-in-fabric) | Mandatory — READ link first | | Fabric Topology & Key Concepts | [COMMON-CORE.md § Fabric Topology & Key Concepts](../../common/COMMON-CORE.md#fabric-topology--key-concepts) || | Environment URLs | [COMMON-CORE.md § Environment URLs](../../common/COMMON-CORE.md#environment-urls) || | Authentication & Token Acquisition | [COMMON-CORE.md § Authentication & Token Acquisition](../../common/COMMON-CORE.md#authentication--token-acquisition) | Wrong audience = 401; read before any auth issue | | Core Control-Plane REST APIs | [COMMON-CORE.md § Core Control-Plane REST APIs](../../common/COMMON-CORE.md#core-control-plane-rest-apis) | Includes pagination, LRO polling, and rate-limiting patterns | | Job Execution | [COMMON-CORE.md § Job Execution](../../common/COMMON-CORE.md#job-execution) || | Gotchas, Best Practices & Troubleshooting | [COMMON-CORE.md § Gotchas, Best Practices & Troubleshooting](../../common/COMMON-CORE.md#gotchas-best-practices--troubleshooting) || | Tool Selection Rationale | [COMMON-CLI.md § Tool Selection Rationale](../../common/COMMON-CLI.md#tool-selection-rationale) || | Authentication Recipes | [COMMON-CLI.md § Authentication Recipes](../../common/COMMON-CLI.md#authentication-recipes) | az login flows and token acquisition | | Fabric Control-Plane API via az rest | [COMMON-CLI.md § Fabric Control-Plane API via az rest](../../common/COMMON-CLI.md#fabric-control-plane-api-via-az-rest) | Always pass --resource; includes pagination and LRO helpers | | Job Execution (CLI) | [COMMON-CLI.md § Job Execution](../../common/COMMON-CLI.md#job-execution) || | Gotchas & Troubleshooting (CLI-Specific) | [COMMON-CLI.md § Gotchas & Troubleshooting (CLI-Specific)](../../common/COMMON-CLI.md#gotchas--troubleshooting-cli-specific) | az rest audience, shell escaping, token expiry | | Quick Reference | [COMMON-CLI.md § Quick Reference](../../common/COMMON-CLI.md#quick-reference) | az rest template + token audience/tool matrix | | Consumption Capability Matrix | [DATAFLOWS-CONSUMPTION-CORE.md § Consumption Capability Matrix](../../common/DATAFLOWS-CONSUMPTION-CORE.md#consumption-capability-matrix) | Read first — shows what ops are available | | REST API Surface (Consumption) | [DATAFLOWS-CONSUMPTION-CORE.md § REST API Surface](../../common/DATAFLOWS-CONSUMPTION-CORE.md#rest-api-surface-consumption) | List, Get, Parameters, getDefinition, Jobs | | Dataflow Definition Exploration | [DATAFLOWS-CONSUMPTION-CORE.md § Dataflow Definition Exploration](../../common/DATAFLOWS-CONSUMPTION-CORE.md#dataflow-definition-exploration) | Decode mashup.pq, queryMetadata.json, .platform | | Parameter Discovery and Analysis | [DATAFLOWS-CONSUMPTION-CORE.md § Parameter Discovery and Analysis](../../common/DATAFLOWS-CONSUMPTION-CORE.md#parameter-discovery-and-analysis) | Types, formats, M code patterns | | Refresh and Job Monitoring | [DATAFLOWS-CONSUMPTION-CORE.md § Refresh and Job Monitoring](../../common/DATAFLOWS-CONSUMPTION-CORE.md#refresh-and-job-monitoring) | LRO pattern, job instances, polling best practices | | Agentic Exploration Pattern | [DATAFLOWS-CONSUMPTION-CORE.md § Agentic Exploration Pattern](../../common/DATAFLOWS-CONSUMPTION-CORE.md#agentic-exploration-pattern-chat-with-my-dataflows) | 6-step discovery sequence | | Security and Permissions Model | [DATAFLOWS-CONSUMPTION-CORE.md § Security and Permissions Model](../../common/DATAFLOWS-CONSUMPTION-CORE.md#security-and-permissions-model) | Permission matrix by operation | | Common Errors | [DATAFLOWS-CONSUMPTION-CORE.md § Common Errors](../../common/DATAFLOWS-CONSUMPTION-CORE.md#common-errors) | Error codes and resolutions | | Gotchas and Troubleshooting Reference | [DATAFLOWS-CONSUMPTION-CORE.md § Gotchas and Troubleshooting](../../common/DATAFLOWS-CONSUMPTION-CORE.md#gotchas-and-troubleshooting-reference) | 12 numbered issues with cause + resolution | | Quick Reference One-Liners | [consumption-cli-quickref.md](references/consumption-cli-quickref.md) | az rest one-liners for all consumption ops | | Discovery Patterns | [discovery-queries.md](references/discovery-queries.md) | Definition decoding, parameter extraction, connection analysis | | Script Templates | [script-templates.md](references/script-templates.md) | Copy-paste bash and PowerShell templates | | Tool Stack | [SKILL.md § Tool Stack](#tool-stack) || | Connection | [SKILL.md § Connection](#connection) || | Agentic Exploration ("Chat With My Dataflows") | [SKILL.md § Agentic Exploration](#agentic-exploration-chat-with-my-dataflows) | Start here for dataflow exploration | | Query Execution | [SKILL.md § Query Evaluation](#query-evaluation) | Execute individual queries; responses are Apache Arrow binary |

---

Tool Stack

| Tool | Role | Install | |---|---|---| | az CLI | Primary: Auth (az login), Fabric REST API via az rest | Pre-installed in most dev environments | | curl | Alternative HTTP client for REST calls | Pre-installed | | jq | Parse JSON responses, extract fields, format output | Pre-installed or trivial | | base64 | Decode definition parts from base64 | Built into bash; PowerShell uses [Convert]::FromBase64String | | bash/pwsh | Script execution | Pre-installed |

> Agent check — verify before first operation: > ``bash > az account show >/dev/null 2>&1 || echo "RUN: az login" > command -v jq >/dev/null 2>&1 || echo "INSTALL: apt-get install jq OR brew install jq" > ``

---

Connection

Resolve Workspace ID and Dataflow ID

Per [COMMON-CLI.md](../../common/COMMON-CLI.md) Finding Workspaces and Items in Fabric:

# Find workspace ID by name
WS_ID=$(az rest --method get \
  --resource "https://api.fabric.microsoft.com" \
  --url "https://api.fabric.microsoft.com/v1/workspaces" \
  --query "value[?displayName=='My Workspace'].id" --output tsv)

# Find dataflow ID by name within workspace
DF_ID=$(az rest --method get \
  --resource "https://api.fabric.microsoft.com" \
  --url "https://api.fabric.microsoft.com/v1/workspaces/$WS_ID/dataflows" \
  --query "value[?displayName=='Sales Data Pipeline'].id" --output tsv)

Reusable Connection Variables

# Set once at script top
WS_ID=""
DF_ID=""
API="https://api.fabric.microsoft.com/v1"
AZ="az rest --resource https://api.fabric.microsoft.com"

---

Agentic Exploration ("Chat With My Dataflows")

Discovery Sequence

Run these in order to fully explore a workspace's dataflows. See [references/discovery-queries.md](references/discovery-queries.md) for extended patterns.

# 1. List workspaces → find target
az rest --method get --resource "https://api.fabric.microsoft.com" \
  --url "$API/workspaces" --query "value[].{name:displayName, id:id}" -o table

# 2. List dataflows → enumerate all
az rest --method get --resource "https://api.fabric.microsoft.com" \
  --url "$API/workspaces/$WS_ID/dataflows" \
  --query "value[].{name:displayName, id:id, desc:description}" -o table

# 3. Get dataflow properties
az rest --method get --resource "https://api.fabric.microsoft.com" \
  --url "$API/workspaces/$WS_ID/dataflows/$DF_ID"

# 4. Discover parameters
az rest --method get --resource "https://api.fabric.microsoft.com" \
  --url "$API/workspaces/$WS_ID/dataflows/$DF_ID/parameters" \
  --query "value[].{name:name, type:type, required:isRequired, default:defaultValue}" -o table

# 5. Get definition → decode mashup.pq
RESPONSE=$(az rest --method post --resource "https://api.fabric.microsoft.com" \
  --url "$API/workspaces/$WS_ID/dataflows/$DF_ID/getDefinition")
echo "$RESPONSE" | jq -r '.definition.parts[] | select(.path=="mashup.pq") | .payload' | base64 --decode

# 6. Check job history
az rest --method get --resource "https://api.fabric.microsoft.com" \
  --url "$API/workspaces/$WS_ID/dataflows/$DF_ID/jobs/instances" \
  --query "value[].{status:status, type:invokeType, start:startTimeUtc, end:endTimeUtc, error:failureReason}" -o table

Agentic Workflow

1. Discover → Run Steps 1–3 to list and identify dataflows.
2. Parameters → Step 4 to understand inputs and defaults.
3. Definition → Step 5 to inspect M queries, connections, staging config.
4. Monitor → Step 6 for refresh history and error patterns.
5. Iterate → Drill into specific queries or connection details.
6. Present → Summarize findings or generate a reusable script (see [script-templates.md](references/script-templates.md)).

---

Gotchas, Rules, Troubleshooting

For full platform gotchas: [DATAFLOWS-CONSUMPTION-CORE.md](../../common/DATAFLOWS-CONSUMPTION-CORE.md) Gotchas and Troubleshooting Reference and [COMMON-CLI.md](../../common/COMMON-CLI.md) Gotchas & Troubleshooting (CLI-Specific).

MUST DO

• Always az login first — az rest uses the active session. No session → cryptic failure.
• Always --resource "https://api.fabric.microsoft.com" — wrong audience = 401.
• Handle pagination — repeat requests with continuationToken until absent/null.
• Handle LRO for getDefinition — may return 202 Accepted with Location header; poll until complete.
• Decode base64 before inspecting — definition parts are base64-encoded.
• Use POST for getDefinition — it is NOT a GET endpoint.

AVOID

• Hardcoded GUIDs — always discover via list-then-filter pattern.
• Assuming getDefinition is GET — it is POST (common mistake).
• Ignoring pagination — list endpoints may return partial results.
• Polling too aggressively — respect Retry-After headers on 429s.
• Expecting getDefinition with Viewer role — requires Read+Write (Contributor+).

PREFER

• az rest over raw curl — handles auth automatically.
• List-then-filter pattern — no server-side name filter for dataflows.
• Exponential backoff for job polling — 5s → 10s → 20s → 30s cap.
• jq for response parsing — cleaner than shell string manipulation.
• JMESPath --query for simple field extraction directly in az rest.
• Env vars (WS_ID, DF_ID, API) for script reuse.

TROUBLESHOOTING

| Symptom | Cause | Fix | |---|---|---| | 401 Unauthorized | Token expired or wrong audience | az login; ensure --resource "https://api.fabric.microsoft.com" | | 403 Forbidden on getDefinition | Viewer role (Read-only) | Requires Contributor role or higher (Read+Write) | | 404 Not Found | Wrong workspace or dataflow ID | Re-discover via List Dataflows API | | getDefinition returns 202 | Large definition or server load | Poll the Location header URL until operation completes | | Empty parameters array | Dataflow has no parameters | Expected behavior — check mashup.pq for IsParameterQuery | | Base64 decode shows garbled text | BOM in encoded content | Strip UTF-8 BOM (\xEF\xBB\xBF) when decoding | | 429 TooManyRequests | Rate limited | Respect Retry-After header; implement exponential backoff | | Duplicate results in list | Re-using stale continuationToken | Always use the token from the most recent response | | OperationNotSupportedForItem | Wrong item type | Verify item is type Dataflow via Get Item |

---

Examples

Example 1: List All Dataflows in a Workspace

Prompt: "List all dataflows in this workspace."

Agent response:

az rest --method get \
  --url "https://api.fabric.microsoft.com/v1/workspaces/${WS_ID}/dataflows" \
  --resource "https://api.fabric.microsoft.com" \
  --query "value[].{Name:displayName, Id:id, Type:type}" -o table

Example 2: Decode a Dataflow Definition

Prompt: "Show me the Power Query M code inside this dataflow."

Agent response:

# Step 1: Request definition (POST returns 202 with Location header).
# az rest cannot expose response headers, so capture Location via curl + az-acquired token.
TOKEN=$(az account get-access-token --resource "https://api.fabric.microsoft.com" --query accessToken -o tsv)
LOCATION=$(curl -sS -X POST \
  -H "Authorization: Bearer $TOKEN" -H "Content-Length: 0" \
  "https://api.fabric.microsoft.com/v1/workspaces/${WS_ID}/dataflows/${DF_ID}/getDefinition" \
  -o /dev/null -D - | tr -d '\r' | grep -i "^location:" | awk '{print $2}')

# Step 2: Poll until definition is ready
DEF=$(az rest --method get --url "${LOCATION}" \
  --resource "https://api.fabric.microsoft.com")

# Step 3: Decode mashup.pq to see the Power Query M code
echo "$DEF" | python3 -c "
import json, base64, sys
parts = json.load(sys.stdin)['definition']['parts']
for p in parts:
    if p['path'] == 'mashup.pq':
        print(base64.b64decode(p['payload']).decode('utf-8'))
"

Example 3: Check Refresh Job History

Prompt: "Show me the recent refresh history for this dataflow."

Agent response:

# Get recent job instances for a dataflow
az rest --method get \
  --url "https://api.fabric.microsoft.com/v1/workspaces/${WS_ID}/dataflows/${DF_ID}/jobs/instances?limit=5" \
  --resource "https://api.fabric.microsoft.com" \
  --query "value[].{Status:status, Start:startTimeUtc, End:endTimeUtc, Id:id}" -o table

Example 4: Discover Parameters from Definition

Prompt: "What parameters does this dataflow accept?"

Agent response:

# After decoding the definition (see Example 2), extract parameters:
echo "$DEF" | python3 -c "
import json, base64, sys
parts = json.load(sys.stdin)['definition']['parts']
for p in parts:
    if p['path'] == 'queryMetadata.json':
        meta = json.loads(base64.b64decode(p['payload']).decode('utf-8'))
        for qname, qmeta in meta.get('queriesMetadata', {}).items():
            if qmeta.get('queryGroupId') == 'parameters' or 'IsParameterQuery' in str(qmeta):
                print(f'Parameter: {qname}')
"

---

Query Evaluation

Execute an individual query from a dataflow and inspect results. Responses are a raw Apache Arrow IPC stream with Content-Type: application/vnd.apache.arrow.stream — not a JSON envelope. The first four bytes of a valid stream are the IPC continuation marker ff ff ff ff. Parse with pyarrow.ipc.open_stream().

> Wire format: executeQuery returns the raw Apache Arrow IPC byte stream (Content-Type: application/vnd.apache.arrow.stream) — not JSON. Don't try to parse it with jq — there is no JSON envelope to extract. Use --output-file to save the bytes and parse as Arrow (see Examples 5–7).

> Failures return HTTP 200: executeQuery returns 200 OK with application/vnd.apache.arrow.stream even when the underlying source query fails (Kusto SEM0100, T-SQL syntax error, missing column, etc.). The error is embedded inside the stream's PQ Arrow Metadata section as {"Error":"..."} — see [dataflows-authoring-cli § mashup-preview.md → Detecting failures inside the Arrow body](../dataflows-authoring-cli/references/mashup-preview.md#detecting-failures-inside-the-arrow-body) for detector snippets. Naive HTTP-status checks will treat failures as success.

> **Intent split (canon

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: microsoft
• Source: microsoft/skills-for-fabric
• License: MIT

Install and usage instructions live in the source repository linked above.