Priostack · Engineering Blog · 5 April 2026 · 18 min read

BPM + Integration: The Two-Layer Architecture Behind Priostack

Architecture Engineering Deep dive

A common question when evaluating Priostack: "What happens when I need more than a job worker fetching tasks over REST? What if I need deduplication, content-based routing, or message correlation across multiple process instances?"

The answer is a two-layer architecture. Layer 1 is the BPM runtime - BPMN, DMN, CMMN execution with a clean REST job API. It is the complete runtime for the majority of use cases. Layer 2 is an optional Enterprise Integration Patterns (EIP) layer built into the same deployment: no broker, no extra service, just configuration. The two layers are entirely independent by design - pkg/integration never imports pkg/bpm.

This article walks through both layers in depth: what each one gives you, where the boundary sits, and - critically - how the integration layer makes idempotency, deduplication, and message orchestration a configuration problem rather than an infrastructure one.

1. The stack at a glance

2. Layer 1 - The BPM runtime

Layer 1 is the engine. It covers three process notations:

The REST job API

The fundamental interaction model is pull-based. Workers are external services - a Python microservice, a Node script, a Go binary, anything that speaks HTTP. They operate a three-step loop:

# 1. Activate  -  fetch a locked job from the engine
GET /api/v1/jobs/activate?type=credit-check&maxJobs=1
→ { jobKey, variables, processInstanceKey, retries }

# 2. Execute  -  your logic runs locally, fully isolated
# The engine waits. The job is LOCKED with a timeout.

# 3. Complete or fail
POST /api/v1/jobs/{jobKey}/complete   { "variables": { "approved": true } }
POST /api/v1/jobs/{jobKey}/fail       { "errorMessage": "timeout", "retries": 2 }

No broker. No persistent connection. No SDK required. If the worker dies between steps 2 and 3, the job lock timeout expires and the job re-enters the queue automatically.

Instance state and observability

Every process instance has a state maintained by the engine: ACTIVE, INCIDENT, COMPLETED, or TERMINATED. Incidents are raised automatically when a job exhausts its retries or when a worker calls the fail endpoint with retries remaining at zero. From the dashboard you can see exactly which element in the BPMN graph triggered the incident, inspect the variables at that point, retry from there, or cancel the instance.

3. Layer 2 - The integration layer

pkg/integration implements nine Enterprise Integration Pattern primitives, each modelling a concept from the Hohpe & Woolf EIP catalogue. All of them are plain Go structs - no framework, no DSL, no XML. You register them in a Registry and compose them into pipelines.

MessageChannel

A named conduit with two delivery modes: ChannelPointToPoint - each message delivered to exactly one consumer - or ChannelPubSub - each message delivered to all subscribers. Channels are typed to an ItemDefinition, so mismatched message schemas are caught at registration time rather than at runtime.

MessageEndpoint

Connects an external service (via an ArchiMate BusinessService.ID reference) to a channel. Inbound endpoints are entry points; messages queue there until the engine consumes them.

MessageRouter

Content-based routing using FEEL conditions evaluated against message payload variables. Routes are prioritised; the first matching route wins. An empty condition is the catch-all default. Exactly one branch fires per message.

router := &integration.MessageRouter{
    ID: "claim-router",
    Routes: []integration.Route{
        { ChannelID: "high-value",  Condition: `amount > 10000`,  Priority: 10 },
        { ChannelID: "auto-approve", Condition: `amount <= 500`,   Priority: 5  },
        { ChannelID: "standard",    Condition: "",                              }, // default
    },
}

MessageFilter

Drops messages that do not satisfy a FEEL predicate. Non-matching messages are silently discarded - they never reach the engine. Useful for idempotency tokens (filter already-seen correlation keys) and schema validation (filter malformed payloads before they cause incidents).

Aggregator

The most powerful primitive for idempotency and fan-in. The Aggregator accumulates related messages grouped by a FEEL CorrelationExpr, holds them until a CompletionCond is satisfied (or a Timeout fires), and then releases the batch as a single message.

agg := &integration.Aggregator{
    ID:              "order-results",
    CorrelationExpr: `orderId`,          // FEEL: group by this field
    CompletionCond:  `count(items) >= 3`, // FEEL: release when 3 results arrive
    Timeout:         30 * time.Second,   // force-release after 30s regardless
    OutputChannelID: "order-complete",
}

In a worker-failure scenario: Worker A and Worker B both complete the same job before the lock timeout expires (a race). Both completions arrive at the Aggregator under the same correlation key. The CompletionCond is written to expect exactly one result. The Aggregator holds the second completion and never forwards it. The engine sees one completion per job, server-side, with no worker-side logic required.

CorrelationContext

Maintains a map of correlation key → waiting process instance. The canonical use case is a BPMN Receive Task: the task suspends the instance and registers a CorrelationEntry. When a message arrives with a matching key, Match() returns the entry and removes it atomically. The second call with the same key returns nil. This is first-match-wins deduplication enforced at the engine boundary, implemented in 15 lines of Go with a single mutex.

ctx := integration.NewCorrelationContext(`paymentId`) // FEEL key expression

// When a ReceiveTask activates:
ctx.Register(paymentId, instanceKey, resumePlace)

// When a payment confirmation arrives:
entry := ctx.Match(paymentId)
if entry == nil {
    return // duplicate  -  ignore
}
// resume entry.InstanceKey at entry.ResumePlace

Splitter

Produces N output messages from one input. The SplitExpr is a FEEL expression that evaluates to a list; one message is emitted per element. Use this for fan-out - a single order confirmation that needs to trigger a fulfilment task, a notification task, and an audit task in parallel, each as an independent process instance.

MessageTranslator

Declares a schema transformation between two ItemDefinition types using a FEEL mapping expression. Separates integration logic (field renaming, unit conversion, structural transformation) from process logic. The BPMN process receives a correctly shaped payload regardless of what the upstream service emitted.

Pipeline

An ordered chain of EIP components - the Pipes-and-Filters pattern. Messages enter at step 0 and pass through each component in order. A typical pipeline might be: Filter → Translator → Router → Aggregator. Each component runs in order; the output of one feeds the input of the next.

pipeline := &integration.Pipeline{
    ID: "payment-ingest",
    Steps: []integration.PipelineStep{
        { ComponentID: "duplicate-filter",   ComponentKind: integration.ComponentFilter     },
        { ComponentID: "schema-translator",  ComponentKind: integration.ComponentTranslator },
        { ComponentID: "value-router",       ComponentKind: integration.ComponentRouter     },
        { ComponentID: "result-aggregator",  ComponentKind: integration.ComponentAggregator },
    },
}

4. Idempotency without a broker

The typical argument against REST polling for job workers is that it requires idempotency to be implemented by each worker individually. If a worker dies mid-execution, the job re-enters the queue and a second worker picks it up. Without a deduplication mechanism, the same logical action executes twice.

Layer 2 eliminates this requirement on the worker side. Here is the full server-side idempotency stack:

The critical insight: none of these mechanisms require the worker to maintain state. Workers remain stateless. The engine is the deduplication fence.

5. When to use which layer

The decision is not permanent. You start with Layer 1 and add integration components incrementally when a specific requirement surfaces. There is no migration, no data re-modelling, and no service to deploy. You register a Pipeline or an Aggregator in your application startup code and the engine starts using it.

6. The scale story

REST polling has a real ceiling. At tens of thousands of concurrent workers with sub-second polling intervals, the activate endpoint becomes a bottleneck. This is acknowledged honestly: for extremely high-frequency job throughput, a message queue (Kafka, RabbitMQ, SQS) in front of the activate endpoint is the right call.

But that ceiling is much higher than most assume, and the integration layer raises it further. Consider:

• A worker polling every second with an average job duration of 10 seconds generates approximately one activate request every 10 seconds while busy.
• With an Aggregator batching completions, you can reduce round-trips significantly for high-frequency short jobs.
• The MessageFilter drops invalid or duplicate completions before they reach the engine, reducing contention on the instance lock.

For the use cases Priostack is designed for - BPM consultant tooling, enterprise approval flows, architecture-to-execution workflows - the ceiling is never reached. The typical deployment has dozens to hundreds of concurrent instances, not millions. You add a broker when you have a broker-scale problem. Not before.

If you want to see the integration layer in action, the EIP Pipelines article walks through two complete topologies - a big-data corpus-building pipeline and a fast-data anomaly-routing pipeline - built entirely with these nine primitives.