Building a Workflow Engine for your users

Users today are demanding customization and integrations from every product. Your users may want your product to support custom workflows to automate key user actions. Custom, user-defined workflows can be a powerful feature to add to your product, but it is not without challenges. To build a workflow engine you need to:

  • Define a series of pre-built tasks that user can select from
  • Create a way for users to use these tasks to build our their workflow
  • Build a system that can execute tasks from a given workflow whenever it is triggered

Building a system like this is difficult. To do this, you would traditionally need:

  • Queues and/or event streams to trigger workflows
  • Scheduling logic to determine what task to execute and when to execute it
  • Queues to handle processing of the tasks
  • A state store that handles passing context and interim state between tasks
  • Automatic retries of tasks that throw errors

Inngest can handle all of this for you enabling you to build a workflow engine in record time. Let's dig in to how you can do this:

What is a "Workflow?"

Workflow is an overloaded term, but in the scope of this guide, we define a workflow as:

A set of tasks that need to be executed in a specific order that may depend on the result from earlier tasks.

A simple example

We'll show a simple example, then walk through how to build something like this yourself. This example shows a custom workflow that might get triggered when a CRM product's user changes the status of one of their leads.

import { sendEmail, sendSlackNotification } from "../lib/emails"; // your custom code
import { inngest } from "./client";

export const workflowExecutor = inngest.createFunction(
  { id: "crm-workflow-executor" },
  { event: "crm/lead_status.changed" },
  async ({ event, step }) => {
    const { accountId } = event.data;

    // Load the workflow that the user defined
    const workflow = await step.run("Load workflow from database", async () => {
      return database.workflows.find({
        accountId,
        trigger: "lead_status_changed",
      });
    });
    // The workflow here is represented as a simple series of actions
    /*
      "workflow": {
        "name": "Approval",
        "actions": [
          { "type": "filter", "data": { "field": "status", "value": "proposal_created" } }
          { "type": "email", "data": { "template": "proposal", "to": "leadEmail" } },
          { "type": "slack", "data": { "channel": "C12345" } },
          { "type": "wait", "data": { "duration": "2 days" } },
          { "type": "email", "data": { "template": "proposal_follow_up", "to": "salespersonEmail" } },
        ]
      }
    */

    // We allow filter to terminate the workflow early
    let shouldContinue = true;

    // We loop over all actions as long as there are any remaining and no filter has exited
    while (shouldContinue && workflow.actions.length) {
      const action = workflow.actions.unshift();

      // We now run one of our pre-build actions:
      switch (action.type) {
        // The filter action sets shouldContinue and can end the workflow at any time
        case "filter":
          shouldContinue = await step.run("filter", async () => {
            // Filter uses the action's configuration to match against the values
            // in the triggering event:
            return event.data[action.data.field] === action.data.value;
          });
          break;

        // The email action just calls some external code that you imported
        case "email":
          await step.run("send-email", async () => {
            // Determine who the email should be sent to
            const sendToAddress = event.data[action.data.to];
            return await sendEmail({
              to: sendToAddress,
              template: action.data.template,
            });
          });
          break;

        // The slack action sends a pre-defined Slack notification to the user's channel
        case "slack":
          await step.run("slack-notification", async () => {
            // Grab the slack token from the database and post a message to the user's Slack
            const slackToken = await database.slack_tokens.find({ accountId });
            return await sendSlackNotification({
              channelId: action.data.channel,
              notificationType: event.data.status, // i.e. "proposal_created"
              slackToken,
            });
          });
          break;

        // The sleep action allows the workflow to pause for any amount of time
        case "sleep":
          await step.sleep("pause-workflow", action.data.duration);
          break;
      }
    }

    // All actions have been executed, or the filter has determined to terminate the workflow early
    // We're all done and we can return some data for our logs:
    return {
      status: "completed",
      workflowName: workflow.name,
      accountId,
    };
  }
);

How do I build this?

There are several steps that you can take to build something like this, but they're all dependent on what you want to deliver for your users. This will walk through the high level flow of what you need to do and how to build with Inngest:

  1. Define the structure to configure and store workflows. The simplest option might be an array of sequential events, but you also could define a DAG to allow for parallel workflow steps or complex ordering. In the above example, we use a simple array of actions.
  2. Determine the pre-built actions and their inputs/outputs. What do you want your users to be able to do? Some ideas might include: sending an email notification, updating another system via API, add a delay for n days, or triggering another part of your system.
  3. Write logic to traverse workflow structure. Keep it simple and flexible. You basically want to write logic that loops over the workflow and determines what to do next - that's it.
  4. Build your actions as steps. Now that you have your workflow traversal code written, you can conditionally run different Inngest steps (step.run(), step.sleep(), etc.).
  5. Send events to trigger the user workflows. Lastly, you just need to start triggering Inngest events from your system with inngest.send(). Events trigger Inngest functions to run, so you can add triggers in any part of your system, like when a user edits data, a data pipeline completes ingestion, or an external trigger like a webhook event.

With all of the above squared away, the rest is a UI task to allow users to define and customize workflows in your application. Inngest takes care of the hard parts so you can ship customizable workflow features to all of your users in a short amount of time.