# Pipeline Migration

> >

- **Type:** Skill
- **Install:** `agentstack add skill-microsoft-skills-for-fabric-pipeline-migration`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [microsoft](https://agentstack.voostack.com/s/microsoft)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [microsoft](https://github.com/microsoft)
- **Source:** https://github.com/microsoft/skills-for-fabric/tree/main/plugins/fabric-skills/skills/pipeline-migration

## Install

```sh
agentstack add skill-microsoft-skills-for-fabric-pipeline-migration
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

> **Update Check — ONCE PER SESSION (mandatory for live migration runs)**
> The first time this skill is used in a session, run the **check-updates** skill before proceeding.
> - **GitHub Copilot CLI / VS Code**: invoke the `check-updates` skill.
> - **Claude Code / Cowork / Cursor / Windsurf / Codex**: compare local vs remote package.json version.
> - Skip if the check was already performed earlier in this session.
> - **Skip in eval / guidance-only mode**: if the user's prompt explicitly forbids tool/API calls (e.g. *"Do not call any APIs or create any resources"*) or the skill is being invoked for offline JSON translation / eval, do **not** run the update check — answer the prompt directly. The check is only required for live-migration runs that will call Fabric or Synapse APIs.

> **CRITICAL NOTES**
> 1. To find workspace details (including its ID) from a workspace name: list all workspaces, then use JMESPath filtering
> 2. To find item details (including its ID) from workspace ID, item type, and item name: list all items of that type in that workspace, then use JMESPath filtering
> 3. Fabric does NOT have a "Dataset" item type — all dataset properties are **inlined** into activity `typeProperties`
> 4. Linked Services map to Fabric **Connections** — in pipeline activity JSON, `referenceName` on the activity's `linkedService` block uses the Fabric connection **display name** (not the connection GUID); the connection GUID is used only on Fabric REST API calls
> 5. Notebook activities change from `SynapseNotebook` to `TridentNotebook` and reference notebooks by **GUID**, not by name
> 6. Synapse global parameters become a **Variable Library** item in Fabric, referenced as `@pipeline().libraryVariables.`. Variable Library `Number` types are not consumable in pipelines — Synapse `Float`/`Double` are mapped to `String` for runtime compatibility
> 7. The `Validation` activity type does not exist in Fabric — it must be rewritten as `GetMetadata` + `IfCondition`
> 8. **Triggers are intentionally excluded** from this skill — recreate schedules manually in Fabric after migration
> 9. SSIS package execution, SHIR-exclusive connectors, and Databricks activities are **parked** — see [pipeline-gotchas.md](resources/pipeline-gotchas.md)

# Synapse Pipelines → Microsoft Fabric Data Factory Migration

## Prerequisite Knowledge

These companion documents provide general Fabric REST patterns. **Do NOT read them upfront** — reference only when a specific phase requires a pattern not already covered in this skill's resource files:

- [COMMON-CORE.md](../../common/COMMON-CORE.md) — General Fabric REST API patterns, authentication & token audiences, item discovery via JMESPath
- [COMMON-CLI.md](../../common/COMMON-CLI.md) — `az rest` / `az login` CLI patterns, authentication recipes, pipeline run/schedule operations
- [ITEM-DEFINITIONS-CORE.md](../../common/ITEM-DEFINITIONS-CORE.md) — `DataPipeline` and `VariableLibrary` item definition structures (pipeline-content.json, variables.json)
- [SPARK-AUTHORING-CORE.md](../../common/SPARK-AUTHORING-CORE.md) — Fabric notebook item creation (needed when notebook items don't exist yet in Fabric)

> For the notebook side of the migration (the notebooks that pipeline activities call), use the companion **synapse-migration** skill to migrate the notebook content itself.

---

## Table of Contents

| Topic | Reference |
|---|---|
| **Pre-Migration Assessment** (run first) | [pipeline-assessment.md](resources/pipeline-assessment.md) |
| **Migration Orchestrator** | [pipeline-orchestrator.md](resources/pipeline-orchestrator.md) |
| API-Driven Migration Workflow | [§ API-Driven Migration Workflow](#api-driven-migration-workflow) |
| Activity Type Mapping | [activity-mapping.md](resources/activity-mapping.md) |
| **Notebook Activity Migration** (primary focus) | [notebook-activity-migration.md](resources/notebook-activity-migration.md) |
| Linked Services → Connections | [linked-service-to-connection.md](resources/linked-service-to-connection.md) |
| Dataset Inlining | [dataset-inlining.md](resources/dataset-inlining.md) |
| Global Parameters → Variable Library | [global-parameters-to-variable-library.md](resources/global-parameters-to-variable-library.md) |
| Pipeline Gotchas & Parked Activities | [pipeline-gotchas.md](resources/pipeline-gotchas.md) |
| Validation & Testing | [validation-testing.md](resources/validation-testing.md) |
| Migration Report | [migration-report.md](resources/migration-report.md) |

### Context Loading Guide

> **IMPORTANT — Load only what you need.** Do NOT read all resource files upfront.

| When | Read This File |
|---|---|
| User asks for an assessment, scope, or plan **before** migrating | [pipeline-assessment.md](resources/pipeline-assessment.md) |
| User asks to migrate a full pipeline workspace | [pipeline-orchestrator.md](resources/pipeline-orchestrator.md) |
| User asks about activity type mapping or unsupported activities | [activity-mapping.md](resources/activity-mapping.md) |
| User has SynapseNotebook activities (most common) | [notebook-activity-migration.md](resources/notebook-activity-migration.md) |
| User asks about linked services or connections | [linked-service-to-connection.md](resources/linked-service-to-connection.md) |
| User asks about Copy, Lookup, or GetMetadata with datasets | [dataset-inlining.md](resources/dataset-inlining.md) |
| User has global parameters to convert | [global-parameters-to-variable-library.md](resources/global-parameters-to-variable-library.md) |
| User hits SSIS, SHIR, Databricks, or other blockers | [pipeline-gotchas.md](resources/pipeline-gotchas.md) |
| Post-migration verification | [validation-testing.md](resources/validation-testing.md) |
| Generating a migration summary | [migration-report.md](resources/migration-report.md) |

---

## API-Driven Migration Workflow

### Authentication

| Target | Token Audience |
|---|---|
| Synapse Data Plane (pipelines, datasets, linked services) | `https://dev.azuresynapse.net` |
| Synapse ARM (global parameters, workspace properties) | `https://management.azure.com` |
| Fabric REST API (create pipelines, connections, Variable Libraries) | `https://api.fabric.microsoft.com` |

> Use `az account get-access-token --resource  --query accessToken -o tsv` to acquire tokens.

### Synapse Data-Plane API Reference

| Operation | Endpoint |
|---|---|
| List all pipelines | `GET https://{ws}.dev.azuresynapse.net/pipelines?api-version=2020-12-01` |
| Get pipeline definition | `GET https://{ws}.dev.azuresynapse.net/pipelines/{name}?api-version=2020-12-01` |
| List all datasets | `GET https://{ws}.dev.azuresynapse.net/datasets?api-version=2020-12-01` |
| Get dataset definition | `GET https://{ws}.dev.azuresynapse.net/datasets/{name}?api-version=2020-12-01` |
| List all linked services | `GET https://{ws}.dev.azuresynapse.net/linkedservices?api-version=2020-12-01` |
| Get linked service | `GET https://{ws}.dev.azuresynapse.net/linkedservices/{name}?api-version=2020-12-01` |
| Get workspace (global parameters) | `GET https://management.azure.com/subscriptions/{subId}/resourceGroups/{rg}/providers/Microsoft.Synapse/workspaces/{ws}?api-version=2021-06-01` |

### Fabric API Reference

| Operation | Endpoint |
|---|---|
| List connections | `GET https://api.fabric.microsoft.com/v1/connections` |
| Create connection | `POST https://api.fabric.microsoft.com/v1/connections` |
| Create pipeline item | `POST https://api.fabric.microsoft.com/v1/workspaces/{wsId}/items` |
| Update pipeline definition | `POST https://api.fabric.microsoft.com/v1/workspaces/{wsId}/items/{id}/updateDefinition` |
| Get pipeline definition | `POST https://api.fabric.microsoft.com/v1/workspaces/{wsId}/items/{id}/getDefinition` |
| Create Variable Library | `POST https://api.fabric.microsoft.com/v1/workspaces/{wsId}/items` (type: `VariableLibrary`) |
| List notebooks in workspace | `GET https://api.fabric.microsoft.com/v1/workspaces/{wsId}/notebooks` |

---

## Assessment Mode (Optional but Recommended)

Before creating any items in Fabric, run the **pipeline assessment** to understand scope, complexity, and blockers. The assessment is read-only — it queries Synapse APIs only and produces a markdown report.

**Copilot workflow — no Python script file needed:**
1. Ask the user: *"What is the name of your Synapse workspace?"*
2. Auto-discover subscription ID and resource group via `az account show` and `az synapse workspace show`
3. Run the assessment code inline from the terminal
4. Print the full report output directly in the chat

| When to Use | Action |
|---|---|
| User wants to understand migration scope before committing | Ask for workspace name → load [pipeline-assessment.md](resources/pipeline-assessment.md) → run inline |
| User asks "what will and won't migrate?" | Run assessment, present the Executive Summary section |
| User asks for a migration plan or scoping document | Run assessment, print report in chat |
| User has already decided to migrate | Skip assessment — go straight to Migration Phases below |

> To also save the report to disk, pass `output_path=f"pipeline-assessment-{SYNAPSE_WS}.md"` to `generate_assessment_report()`. The `PipelineAssessment` objects it produces feed directly into the migration scripts in [pipeline-orchestrator.md](resources/pipeline-orchestrator.md).

---

## Migration Mode (Inline — No Script File Required)

Copilot performs the migration directly from the terminal. No Python files to save or run manually.

**Ask the user for:**
1. Synapse workspace name (reuse from assessment if already run)
2. Fabric workspace name
3. Which pipelines to migrate — specific names, or `*` for all
4. Optional name suffix to append to each pipeline in Fabric (e.g. `_migrated`) — leave blank to keep the original name

**Everything else is auto-discovered:**
- Subscription ID and resource group via `az account show` + `az synapse workspace show`
- Fabric workspace ID via `GET /v1/workspaces` filtered by display name
- Notebook GUIDs in Fabric via `GET /v1/workspaces/{wsId}/notebooks` (required for `SynapseNotebook → TridentNotebook`)
- Connection names in Fabric via `GET /v1/connections` (when datasets reference linked services)

**What is fully automated inline:**
- ✅ `SynapseNotebook` → `TridentNotebook` — type rename, GUID lookup, remove `sparkPool`/`sessionConfiguration`, fix timeout to 12h max
- ✅ All compatible activity types — pass through with minor property adjustments
- ✅ Dataset inlining into activity `typeProperties`
- ✅ Global parameter expressions rewritten to `@pipeline().libraryVariables.`
- ✅ Pipeline JSON assembly and deployment to Fabric via REST

**Copilot checks before starting — will pause and report if:**
- A notebook referenced by a `SynapseNotebook` activity does not exist in the Fabric workspace yet
- A Fabric Connection is missing for a linked service referenced by dataset activities

> Load [pipeline-orchestrator.md](resources/pipeline-orchestrator.md) for the complete inline runner.

---

## Migration Phases (Execute in Order)

| Phase | Source | Target | Resource |
|---|---|---|---|
| Phase 0 | Synapse notebooks (referenced by pipeline activities) | Fabric Notebooks | **synapse-migration** skill |
| Phase 1 | Synapse global parameters | Fabric Variable Library | [global-parameters-to-variable-library.md](resources/global-parameters-to-variable-library.md) |
| Phase 2 | Synapse linked services | Fabric Connections | [linked-service-to-connection.md](resources/linked-service-to-connection.md) |
| Phase 3 | Synapse datasets | Inlined into activities | [dataset-inlining.md](resources/dataset-inlining.md) |
| Phase 4 | Synapse pipeline activities | Fabric pipeline activities | [activity-mapping.md](resources/activity-mapping.md) + [notebook-activity-migration.md](resources/notebook-activity-migration.md) |
| Phase 5 | Assembled pipeline JSON | Fabric DataPipeline item | [pipeline-orchestrator.md](resources/pipeline-orchestrator.md) |
| Final | — | Validation | [validation-testing.md](resources/validation-testing.md) |

> **Phase 0 must precede Phase 4**: Fabric Notebook GUIDs are needed before TridentNotebook activities can be written.
> **Phase 2 must precede Phase 3**: Connection names must exist before they can be referenced in inlined datasets.

---

## Activity Type Quick Reference

Full mapping table, before/after examples, and parking decisions are in [activity-mapping.md](resources/activity-mapping.md).

| Synapse Activity | Fabric Equivalent | Status |
|---|---|---|
| `SynapseNotebook` | `TridentNotebook` | ✅ Migrated — see [notebook-activity-migration.md](resources/notebook-activity-migration.md) |
| `Copy` | `Copy` | ✅ Migrated — datasets inlined |
| `Lookup` | `Lookup` | ✅ Migrated — datasets inlined |
| `GetMetadata` | `GetMetadata` | ✅ Migrated — datasets inlined |
| `Validation` | `GetMetadata` + `IfCondition` | ✅ Migrated — split into 2 activities |
| `ForEach` | `ForEach` | ✅ Compatible |
| `IfCondition` | `IfCondition` | ✅ Compatible |
| `Switch` | `Switch` | ✅ Compatible |
| `Until` | `Until` | ✅ Compatible |
| `Wait` | `Wait` | ✅ Compatible |
| `Fail` | `Fail` | ✅ Compatible |
| `SetVariable` | `SetVariable` | ✅ Compatible |
| `AppendVariable` | `AppendVariable` | ✅ Compatible |
| `ExecutePipeline` | `ExecutePipeline` | ✅ Compatible — add `workspaceId` |
| `WebActivity` | `WebActivity` | ✅ Compatible |
| `Script` | `Script` | ✅ Compatible — update connection refs |
| `Delete` | `Delete` | ✅ Migrated — datasets inlined |
| `Filter` | `Filter` | ✅ Compatible |
| `SparkJobDefinition` (Synapse SJD) | `SparkJobDefinition` | ✅ Update GUID refs |
| `HDInsightSpark` | `TridentNotebook` or `SparkJobDefinition` | ⚠️ Rewrite required |
| `AzureMLBatchExecution` | `WebActivity` | ⚠️ Rewrite as REST call |
| `AzureFunctionActivity` | `WebActivity` | ⚠️ Rewrite — use function URL + key |
| `DatabricksNotebook` | ⛔ Parked | See [pipeline-gotchas.md](resources/pipeline-gotchas.md) |
| `DatabricksSparkJar` | ⛔ Parked | See [pipeline-gotchas.md](resources/pipeline-gotchas.md) |
| `DatabricksSparkPython` | ⛔ Parked | See [pipeline-gotchas.md](resources/pipeline-gotchas.md) |
| `ExecuteSSISPackage` | ⛔ Parked | See [pipeline-gotchas.md](resources/pipeline-gotchas.md) |
| `AzureBatch` | ⛔ Parked | No Fabric equivalent |
| `Custom` | ⛔ Parked | No Fabric equivalent |

---

## Must / Prefer / Avoid

### MUST DO
- **Migrate notebooks before pipelines** — Fabric TridentNotebook activities require notebook GUIDs, not names. Use the **synapse-migration** skill first
- **Create Fabric Connections before building pipeline JSON** — linked service names in Synapse become connection references in Fabric; you need the connection names before inlining datasets
- **Inline all dataset definitions** — Fabric Data Factory has no Dataset item type; all `inputs`/`outputs` dataset properties must be embedded in each activity
- **Replace `@pipeline().globalParameters.`** with `@pipeline().libraryVariables.` after creating the Variable Library
- **Replace Validation activities** with a `GetMetadata` + `IfCondition` pair — `Validation` does not exist as an activity type in Fabric
- **Remove `sparkPool` and `sessionConfiguration`** from migrated notebook activities — pool selection and session config belong in the Fabric Environment attached to the notebook

### PREFER
- **Variable Library with Value Sets** for dev/test/prod environments — use `@pipeline().libraryVariables.` with environment-specific Value Sets instead of pipeline-level parameters for environment promotion
- **Parameterized `notebookParameters`** over hardcoded values in TridentNotebook activities — mirrors Synapse parameterized notebook pattern
- **Test notebook activities individually** before running the full migrated pipeline — notebook GUIDs are the most common source of failure
- **OneLake Lakehouse sources/sinks** for Copy activities where applicable — eliminates the need for external connections for data alrea

…

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [microsoft](https://github.com/microsoft)
- **Source:** [microsoft/skills-for-fabric](https://github.com/microsoft/skills-for-fabric)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-microsoft-skills-for-fabric-pipeline-migration
- Seller: https://agentstack.voostack.com/s/microsoft
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.