Workflow Debugging

AEM Workflow Debugging — Cloud Service

Production-grade debugging for the AEM Granite Workflow engine, launcher, Inbox, Sling Jobs, thread pools, and purge on AEM as a Cloud Service (AEMaaCS).

Audience

AEMaaCS developers and operators (and the IDE LLM acting on their behalf) diagnosing stuck or failed workflows on a local AEMaaCS SDK or a cloud environment — Developer Console, Sling Job Console, and Cloud Manager Logs available; no production Felix Console JMX or filesystem access.

Variant Scope

• AEM as a Cloud Service only.
• Not for AEM 6.5 LTS / AMS. If the target is 6.5 LTS, stop and load the 6.5-lts variant of this skill — JMX-based remediation, Felix Console runtime config, AMS log filesystem access, and jstack thread dumps documented there do not apply on AEMaaCS.
• No JMX access on AEMaaCS production. Diagnosis is read-only via Developer Console, Sling Job Console, and Cloud Manager Logs. Never recommend JMX-based remediation (restartStaleWorkflows, purgeCompleted, terminate, retryFailedWorkItems) — those are 6.5-LTS-only mechanisms. Use Inbox Retry, Purge Scheduler (OSGi config in Git), custom servlets like StaleWorkflowServlet, and Cloud Manager pipeline-driven config changes instead.
• All remediation lands via Git + Cloud Manager pipeline: OSGi configs in ui.config, custom servlets in core, ACLs in ui.apps/.../repoinit. There is no Felix Console write access on cloud environments.
• See [reference.md](reference.md) for runbook locations and diagnostic tool pointers.

Dependencies

This skill is largely self-contained but routes back into the dev skills when the root cause is a code or model defect:

• workflow-development — when the diagnosis is "process step throws / not registered / leaks resources"
• workflow-model-design — when the diagnosis is "model has wrong split rule / missing transition / wrong step type"
• workflow-launchers — when the diagnosis is "launcher not firing / re-trigger loop"
• workflow-triaging — load instead of this skill if the user is mining Cloud Manager Logs across multiple environments rather than diagnosing one

---

When to use this skill

• Workflow stuck, not progressing, failed, not starting, task not in Inbox, purge/repository bloat, permissions, queue backlog, thread pool exhaustion, auto-advancement not working.
• User provides thread dumps, Sling Job console output, or error.log excerpts from Cloud Manager.
• Environment: AEM as a Cloud Service (no JMX; use Developer Console, Cloud Manager logs).

---

Step 1: Map symptom to first action

| Symptom | symptomid | First action | |---------|------------|--------------| | Workflow stuck (not advancing) | workflowstucknotprogressing | Open instance; note current step type. No work item → stale. | | Task not in Inbox | tasknotininbox | Confirm Participant step; assignee = logged-in user; Inbox filters. | | Workflow not starting (launcher) | workflownotstartinglauncher | Launcher enabled; path/event match payload. | | Workflow fails or shows error | workflowfailsorshowserror | Instance history; error.log for instance ID; payload and process. | | Step failed, retries exhausted | stepfailedretriesexhausted | Logs → process.label → Inbox Retry, or bulk via custom servlet (see Step 6). | | Stale (no current work item) | staleworkflownoworkitem | Deploy a custom StaleWorkflowServlet to your core bundle; invoke with ?dryRun=true. | | Repository bloat / too many instances | repositorybloattoomanyinstances | Purge Scheduler OSGi config in Git (PID: com.adobe.granite.workflow.purge.Scheduler). | | User cannot see or complete item | usercannotseeorcompleteitem | Assignee / initiator / superuser group; enforce*Permissions flags. | | Cannot delete model | cannotdeletemodel | Count RUNNING instances via Workflow Console → terminate → delete model. | | Slow throughput / queue backlog | slowthroughputqueuebacklog | Sling Job statistics; Granite Workflow Queue queue.maxparallel; Sling thread pool. | | Auto-advancement not working | workflowautoadvancefailure | Check default thread pool saturation; Sling Scheduler; timeout jobs. | | New workflow not working | workflowsetupvalidation | Model sync, launcher, process registration, permissions. |

---

Step 2: Decision tree (workflow stuck)

1. No current work item? → Stale. Deploy a custom StaleWorkflowServlet to your core bundle; call GET /bin/support/workflow/stale?dryRun=true to enumerate, then POST ...?dryRun=false to restart.
2. Participant step → Assignee exists? Inbox visible? Payload accessible? Dynamic participant resolver returning correct user?
3. Process step → Search error.log for instance ID. Check: process.label registered, payload path exists, bundle active, no exception in execute().
4. OR/AND Split → Condition evaluates correctly? Routes exist? No dead-end branches? Model synced?

---

Step 3: Thread dump & thread pool analysis

Thread dumps and status-producer output on AEMaaCS are obtained via Developer Console → Status → Thread Dump / OSGi / Sling Jobs / Sling Scheduler. For anything not exposed in Developer Console, open an Adobe Support ticket — never attempt to SSH into an AEMaaCS pod.

3a. Sling default thread pool (critical path)

The Sling Scheduler ApacheSlingdefault uses ThreadPool: default. This pool runs:

• Oak observation events
• All Quartz-scheduled jobs — including the workflow timeout-detection scheduler that emits com/adobe/granite/workflow/timeout/job events to the Sling Job system (the job itself then runs on the Granite Workflow Queue, see Step 3c)

Check in thread pool output (Developer Console → Status → Sling Thread Pools, /system/console/status-slingthreadpools):

| Field | Healthy | Problem | |-------|---------|---------| | active count | 0 (jobs waiting) | | Failed Jobs | 0 | > 0 (step failures) | | Active Jobs | 0-N | 0 when Queued > 0 (jobs not picked up) |

Check topic statistics for workflow model:

• Topic: com/adobe/granite/workflow/job/var/workflow/models/
• High Failed Jobs / low Finished Jobs ratio → process step throwing exceptions

Check Granite Workflow Queue configuration:

• Type: Topic Round Robin
• Max Parallel: 0.5 OOTB on the AEMaaCS SDK (50% of available CPU cores). Adobe's Workflows Best Practices recommends between half and three-quarters of processor cores. Verify the running value at Developer Console → /system/console/slingevent before assuming. Override via org.apache.sling.event.jobs.QueueConfiguration-.cfg.json in Git if you need to raise it.
• Max Retries: 10

3d. Sling Scheduler

Check the Sling Scheduler page (Developer Console → Status → Sling Scheduler, /system/console/status-slingscheduler):

• This page lists Quartz-style schedulers, not Sling Job topics. The workflow-related entry visible here is the periodic WorkflowStatsMBean collector (used by the Statistics MBean). nextFireTime: null means the trigger was deregistered.
• The com/adobe/granite/workflow/timeout/job topic itself is a Sling Job, not a Quartz job — check it on the Sling Jobs page (/system/console/slingevent), not here.
• Confirm ApacheSlingdefault uses ThreadPool: default — that's how the periodic timeout-detection scheduler reaches the workflow engine.

---

Step 4: Error log patterns

Download error.log from Cloud Manager → Environments → Logs, or use log streaming.

| Pattern | Cause | Action | |---------|-------|--------| | Error executing workflow step | Process step exception | Check stack; fix process code or payload | | getProcess for '' failed | No WorkflowProcess registered | Deploy bundle; match process.label | | Cannot archive workitem | Archive failure → stale risk | Detect and restart stale workflows | | refreshing the session since we had to wait for a lock | Lock contention on /var/workflow | Reduce (not raise) parallelism — lower queue.maxparallel on the Granite Workflow Queue, or stagger launchers. Raising parallelism makes this worse. | | Terminate failed / Resume failed / Suspend failed | Permissions (not initiator/superuser) | Check enforceWorkflowInitiatorPermissions; add to superusers | | PathNotFoundException (workflow/payload) | Payload/launcher path missing | Verify payload exists; check launcher config path | | Error adding launcher config | Launcher config path not created | Create /conf/global/settings/workflow/launcher/config | | retrys exceeded - remove isTransient | Transient workflow failed after retries | Fix process code; instance persisted for admin handling | | RejectedExecutionException | Thread pool full with ABORT policy | Increase pool size or change policy to RUN via config; fix stuck threads | | Workflow is already finished | Terminate on completed/aborted instance | Check logic calling terminate | | Workflow purge '' : repository exception | Purge JCR error | Check permissions; repo health |

---

Step 5: Configuration checklist (Cloud Service — all via Git + pipeline)

Every config below is an OSGi JSON file under ui.config/src/main/content/jcr_root/apps//osgiconfig/config.author/ (or config.author.prod/ / config.author.stage/ for run-mode scoping).

| Config file (PID) | Property | Guidance | |-------------------|----------|----------| | com.adobe.granite.workflow.core.WorkflowSessionFactory.cfg.json | cq.workflow.job.retry | Default 3; raise for flaky external calls. | | org.apache.sling.event.jobs.QueueConfiguration-.cfg.json | queue.maxparallel | Real parallelism knob for workflow jobs (factory PID targeting topics com/adobe/granite/workflow/job/*). OOTB on the AEMaaCS SDK is 0.5 (50% of CPU cores); Adobe's Workflows Best Practices recommends between half and three-quarters of processor cores. cq.workflow.job.max.procs displayed in Felix Config Manager is an orphaned metatype label with no Java code path that reads it (verified against AEM source on master) — do not rely on it. | | com.adobe.granite.workflow.core.WorkflowSessionFactory.cfg.json | granite.workflow.enforceWorkitemAssigneePermissions | true = only the assignee can see / complete a work item. | | com.adobe.granite.workflow.core.WorkflowSessionFactory.cfg.json | granite.workflow.enforceWorkflowInitiatorPermissions | true = only the initiator (or superuser) can terminate / suspend / resume. | | com.adobe.granite.workflow.core.WorkflowSessionFactory.cfg.json | cq.workflow.superuser | AEMaaCS specific: point this at a group provisioned via repoinit (e.g. workflow-administrators), not hard-coded user IDs. Users are federated from IMS and rotate; groups are stable. Service-user mappings go in org.apache.sling.serviceusermapping.impl.ServiceUserMapperImpl.amended-*.cfg.json. | | com.adobe.granite.workflow.core.WorkflowSessionFactory.cfg.json | granite.workflow.inboxQuerySize | Max work items returned per Inbox query. OOTB 2000; raise if heavy users hit the cap. | | com.adobe.granite.workflow.core.WorkflowSessionFactory.cfg.json | granite.workflow.maxPurgeSaveThreshold | OOTB 20 — commit after this many purged instances. Raise carefully to reduce JCR overhead during large purges. | | com.adobe.granite.workflow.core.WorkflowSessionFactory.cfg.json | granite.workflow.maxPurgeQueryCount | OOTB 1000 — JCR query batch size during purge. Tune with maxPurgeSaveThreshold above. | | org.apache.sling.commons.threads.impl.DefaultThreadPool-default.cfg.json | blockPolicy | ABORT silently drops workflow timeout jobs — prefer RUN. See AEMaaCS caveat below this table. | | org.apache.sling.commons.threads.impl.DefaultThreadPool-default.cfg.json | maxPoolSize | OOTB on the AEMaaCS SDK is 10 (factory entry for the default pool); production environments may differ. Raise to 50 if many custom schedulers compete with workflow timeout detection. See AEMaaCS caveat below this table. | | com.adobe.granite.workflow.purge.Scheduler-.cfg.json | scheduledpurge.workflowStatus | Array-typed. Must be ["COMPLETED"], not "COMPLETED". Also: this PID has no scheduledpurge.cron — scheduling is driven by the Granite Maintenance Task window; any cron property is silently ignored. | | com.adobe.granite.workflow.purge.Scheduler-.cfg.json | scheduledpurge.daysold | 30 default; tune per environment. Factory PID — deploy one file per purge schedule. |

AEMaaCS caveat — Sling DefaultThreadPool config may be platform-reserved

Some Sling core configs are filtered or overridden by the AEMaaCS platform layer. The DefaultThreadPool config may land via pipeline and be silently ignored. Always verify after deploy:

• Open /system/console/status-Threads (Developer Console → Status → Threads).
• Find the default pool row.
• Confirm maxPoolSize and block policy reflect your config values.

If the numbers don't change, the PID is Adobe-managed on your environment — do not try to work around it. Open an Adobe Support ticket, attach a thread dump and the thread-pool status, and request Engineering lift the pool size or change block policy for that environment.

Permission and identity gotchas specific to AEMaaCS

• Never list IMS user IDs in cq.workflow.superuser. IMS principals change on re-invite. Reference a JCR group that repoinit creates. Correct repoinit syntax:

`` create group workflow-administrators add admin to group workflow-administrators ` The group keyword after to` is required — without it the repoinit parser fails and the entire script aborts on startup.

• Custom process steps that need elevated access must use a service user + ServiceUserMapperImpl.amended-*.cfg.json, not resolver.adaptTo(Session.class) + admin.
• enforceWorkflowInitiatorPermissions=true + an initiator who is a rotated IMS user leaves workflows unterminable except by superuser. On Cloud Service, prefer superuser-group membership for anyone expected to recover workflows.

---

Step 6: Remediation quick reference (Cloud Service)

| Action | Cloud Service approach | |--------|------------------------| | Retry failed work item (single) | /aem/inbox → select failure → Retry. History and audit trail preserved. | | Retry failed work items (bulk) | Preferred: iterate /aem/inbox UI — single-item Retry preserves the original instance, its history, and its audit trail. Not recommended: a "bulk" servlet using terminateWorkflow(wf) + startWorkflow(model, data) — this creates a new instance and loses the original history, step durations, and comments. Only use the replay approach with explicit customer approval and never for audit-regulated workflows (pharma, finance, legal). | | Restart stale workflows | Deploy a custom StaleWorkflowServlet to your core bundle. Always invoke GET /bin/support/workflow/stale?dryRun=true first; confirm scope; then POST ...?dryRun=false. Scope with &model= if you only want one model. | | Purge completed | Deploy com.adobe.granite.workflow.purge.Scheduler-.cfg.json with scheduledpurge.workflowStatus=["COMPLETED"] (array-typed) and scheduledpurge.daysold=. Triggered by the Granite Maintenance Task window — this PID has no scheduledpurge.cron; any cron property is silently ignored. Do not reference /libs/granite/operations/config/maintenance — on AEMaaCS /libs is the read-only code layer. One purge config file per schedule. Deploy via pipeline. | | Increase parallelism | queue.maxparallel on org.apache.sling.event.jobs.QueueConfiguration-.cfg.json (topics: com/adobe/granite/workflow/job/*). Adobe's Workflows Best Practices recommends staying between half and three-quarters of available CPU cores. The commonly cited cq.workflow.job.max.procs is an orphaned metatype label with no Java code path that reads it (veri

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: adobe
• Source: adobe/skills
• License: Apache-2.0
• Homepage: https://www.adobe.com/ai/overview.html

Install and usage instructions live in the source repository linked above.