A generic workflow engine, configured via a graphical diagram editor on top of an XML syntax – that’s what I tried and failed to develop more than 15 years ago. I did help build three generations of a simple “workflow” component integrated in our DAM product to drive asset ingestion and export, kept reading (see The State of Workflow and Decoupling Application Logic) and writing (Workflow awareness of DAM systems) about workflows – and hoping that one day, a powerful and beautiful workflow management system would make my work easier.

Camunda

That’s why I was thrilled to discover Camunda, a workflow engine with an open source, free community edition. It is standards-based, written in Java, and comes with a Web UI, REST API and graphical process diagram modeler. Here’s a screenshot of Camunda Modeler:

Camunda typically “drives” a workflow process by actively executing your Java code when there’s a task to perform. But you can also use external tasks (see “Implementation: External” and “Topic” in the screenshot above) to turn a task into a queue to be handled by external workers via the Camunda REST API. That pattern is useful for PHP developers like me, but also especially suited to service orchestration. The workflow engine sits there passively, taking only care of workflow definitions, instances, their flow and state. (More on that in the External Tasks for Service Orchestration webinar video.)

While Camunda is trivial to install and extensively documented, it took me a while to grasp the basic concepts – process definitions, process instances, tasks, gateways, variables – and I’m only scratching the surface so far.

Using Camunda from PHP

My example workflow is a minimal “asset processing pipeline”, extracting metadata from an asset file (using ExifTool) and rendering a thumbnail preview image (using ImageMagick) for images. This isn’t “service orchestration”, but a good enough test case to help me get to know Camunda.

First, I drew the above diagram in Camunda Modeler and deployed it to the server. Now I could start a new process instance using curl:

$ curl -X POST \
  http://localhost:8080/engine-rest/process-definition/key/asset-ingestion/start \
  -H 'Content-Type: application/json' \
  -d '{"variables":{"path":{"value":"/tmp/img.jpg"}}}'

I used the camunda-rest-client PHP library (not an official library) to implement a simple worker process that fetches and locks an open task, calls ExifTool or ImageMagick and marks the task as completed. Asset and thumbnail filenames are stored in process instance variables.

You can find all of my PHP code on Github, along with the process definition XML. Here’s some interesting code snippets:

Fetching the next open task, and locking it for exclusive processing by the current worker process:

$externalTaskService = new ExternalTaskService($camundaUrl);

$externalTaskQueryRequest = (new ExternalTaskRequest())
    ->set('topics', [['topicName' => $topicName, 'lockDuration' => $lockDuration]])
    ->set('workerId', $workerId)
    ->set('maxTasks', 1);

// Returns an array of task objects
$externalTasks = $externalTaskService->fetchAndLock($externalTaskQueryRequest);

Reading process instance variables:

$assetFilePath = $externalTask->variables->path->value;

Telling Camunda that the task is completed when the worker is done (so Camunda can automatically move on to the next workflow task), adding task results to the process instance using variables:

$externalTaskRequest = (new ExternalTaskRequest())
    ->set('variables', $updateVariables)
    ->set('workerId', $workerId);

$externalTaskService->complete($externalTask->id, $externalTaskRequest);

Notifying Camunda that the task failed (which makes Camunda report an “incident”):

$externalTaskRequest = (new ExternalTaskRequest())
    ->set('errorMessage', $errorMessage)
    ->set('retries', $retries)
    ->set('retryTimeout', $retryTimeout)
    ->set('workerId', $workerId);

$this->externalTaskService->handleFailure($externalTask->id, $externalTaskRequest);

In my example, I need to instantiate at least one worker process per task “topic”. Multiple instances would automatically process open tasks in parallel. Running a worker looks like this:

$ php worker.php \
  --camunda-url="http://localhost:8080/engine-rest" \
  --task-topic="asset-extract-metadata"
Fetched and locked <asset-extract-metadata> task <00a74f0a-abcd-11e8-afcc-02422b22e161> 
  of <asset-ingestion> process instance <00a664a4-abcd-11e8-afcc-02422b22e161>.
Completed task <00a74f0a-abcd-11e8-afcc-02422b22e161>

Batteries included

I love it when software developers care for transparency and visualization. Camunda Cockpit lets you peek into currently-running processes by showing how many of them are in which state, and letting you inspect and even change their variables:

It’s pretty amazing that a simple diagram plus a single 300-line PHP file get me a separation of process flow and task implementation, queues, parallel worker processes, error handling, and insights into executions.

I will definitely keep playing with Camunda. If you’re already using it in production, I’m happy to hear from you!