Priostack · Engineering Blog · 10 March 2026 · Updated 21 August 2026 · 14 min read · Companion to Geometric Memory and Process Trajectories

EIP Pipelines and Geometric Memory:
Big-Data Corpus Building and Fast-Data Anomaly Routing

Deep dive Integration Engineering

The companion article explains what geometric memory is: an execution becomes a path through a space, and the distance between paths catches deviations nobody wrote a rule for. It ends on a question it deliberately defers.

A detector is not a system. Knowing how to score one trajectory tells you nothing about where the trajectories come from, how the corpus gets built in the first place, what happens to a verdict once it exists, or what the system should do on its first day when it has seen nothing at all. Those are integration questions, and they are where anomaly detection projects usually fail - not on the mathematics.

This article is about that plumbing, expressed in the vocabulary of the classic Enterprise Integration Patterns catalogue: two distinct topologies, a big-data path that builds the reference corpus from history, and a fast-data path that turns a completed instance into a routed decision in milliseconds.

1. Nine patterns, one vocabulary

Priostack's integration layer is the Hohpe and Woolf catalogue, implemented directly and named the way the book names it. There is nothing proprietary in the vocabulary, and that is deliberate: an integration architecture that invents its own words for Router and Aggregator makes every engineer who joins the project relearn something they already knew.

Conditions and mappings are written in FEEL, the expression language defined by the OMG's DMN specification - the same language your decision tables already use. One expression language across decisions, routing predicates and payload mappings means one thing to learn and one thing to test.

2. The message carries its own history

In most integration frameworks a message is a bag of bytes with some headers. In Priostack a message also carries the trajectory of the execution that produced it: the ordered path through geometric memory that the companion article describes, travelling with the payload rather than sitting in a table somewhere.

This is a small design decision with a large operational consequence. A completion event may pass through several channels, a filter, a translator and a router before anything scores it. If the trajectory lived only on the process instance record, every hop that wanted to reason about it would need a lookup - and by the time the message crosses a service boundary, that lookup is a network call against a database owned by someone else. Attaching the history to the message means any component anywhere in the topology can ask a geometric question without asking anyone's permission first.

3. The two topologies: big data and fast data

Geometric memory needs two different integration shapes depending on where in the data lifecycle you are. They use the same patterns and the same catalogue; what differs is how many components you chain and what the conditions say.

Note the third branch on the fast-data path. It is the one most integration diagrams for anomaly detection do not have, and it is the reason this architecture is safe to switch on before it has learned anything: a verdict of "not enough evidence" gets its own channel and its own handling, and is never quietly merged into either of the other two.

4. Big-data path: building the corpus from what already happened

On the day you deploy against an existing process, the corpus is empty and the detector has no opinion about anything. You do not have to wait months to accumulate one. Most organisations running a process worth monitoring already have years of it recorded in an audit log or a warehouse, and replaying that history through the engine produces exactly the trajectories a live run would have produced - because a trajectory is derived from execution state, and the same path yields the same positions whenever it is walked.

The topology is a straightforward bulk-ingest shape. A Splitter turns one batch message into one message per historical instance. A Filter on each drops records already present, which is what makes an incremental re-import safe: overlapping a batch you have already loaded should be a no-op, not a distortion of what the system believes normal to be. An Aggregator groups the acknowledgements by batch and emits a single "corpus ready" event when the last one lands, published to a subscription channel so every consumer that was waiting on a usable corpus finds out at once.

One discipline governs this entire path, and no amount of infrastructure substitutes for it: register only outcomes you have confirmed were legitimate. A corpus built by replaying an audit log wholesale learns that whatever fraud is in that log is normal. Filtering confirmed-good outcomes is a business decision made by people who know the history, and it is the single highest-leverage step in the whole pipeline.

5. Fast-data path: routing a verdict in real time

When an instance completes, the engine emits an event carrying its trajectory. Three things then have to happen, and they have to happen without adding perceptible latency to a user-facing flow:

1. Score the trajectory against the corpus.
2. Route the message to a channel chosen from that verdict.
3. If it is anomalous, correlate it with the case that will investigate it.

A Content-Based Router evaluates its routes in priority order and the first matching condition wins. Because conditions are FEEL evaluated against the message payload, they can combine several signals rather than thresholding one:

route  fraud-investigation   when   novelty > 0.8 and divergence > 20
route  manual-review         when   novelty > 0.5
route  insufficient-evidence when   territory = "unanswerable"
route  normal               otherwise

Gating on evidence, not on a threshold alone

The single most useful operational pattern here is refusing to route to the fraud queue at all until the corpus can support the claim. A condition that also requires a minimum corpus size, or that checks the territory verdict before trusting a novelty score, is what lets you switch the system on in production on day one without flooding an investigations team with alarms generated from three reference points.

The companion article explains why this is a real risk rather than a theoretical one: a score computed against a thin corpus is not merely uncertain, it is arbitrary. Encoding that as a routing condition rather than as a comment in a runbook means the system behaves correctly during warm-up whether or not anyone remembered the caveat.

6. Declared, not coded

Individual patterns are useful in isolation; the Pipes and Filters pattern is where they compose. A pipeline is an ordered list of components. A message enters at step one and is handed down the chain; if a filter rejects it, execution stops there and nothing downstream ever sees it.

The important property is not the composition, though - every integration framework has that. It is that each step is a declaration evaluated by the engine, rather than a piece of application code that happens to run. A routing rule is a stored condition, not a branch buried in a handler. That buys three things worth having:

• Auditability. Every transformation and every routing decision in the system is enumerable. You can answer "what could possibly send a message to the fraud queue?" by reading the catalogue, rather than by grepping a codebase and hoping you found all of it. In a regulated workload that question gets asked by someone with the authority to stop your deployment.
• Change without redeploying consumers. When the event schema gains a field, you update the mapping in one place instead of touching every service that reads it.
• Testability. A condition or a mapping can be evaluated against a fixture on its own, with no engine running and no message bus involved.

7. Translation as a first-class declaration

A completion event has a rich schema: definition, instance, a snapshot of variables, timestamps, deadlines, compliance flags. A scorer wants almost none of that - a trajectory and a correlation key. The Message Translator declares that narrowing as a mapping expression rather than implementing it as a function in whichever service happened to need it first.

Translation deliberately does not touch the trajectory. The path is carried through verbatim, so the scorer receives the geometry exactly as the engine recorded it, and no schema mapping anywhere in the topology can accidentally reshape the thing being measured. Payload schemas are the integration layer's business; the geometry is not.

8. When scoring cannot be synchronous

Scoring inline is the right default and stays viable further than people expect, because the expensive comparison is not run against every corpus entry: novelty is evaluated against a local neighbourhood, and precedent lookup is index-backed rather than a linear scan. Whole-path Fréchet comparison is quadratic in the lengths of the two paths, which is nothing for a process of five to twenty steps.

Two situations break that, and they are worth designing for before you meet them. One is a genuinely long-running process - hundreds of steps, a case open for weeks - where the pairwise comparison stops being free. The other is any deployment where the scoring service is across a network boundary you do not control. In both, holding a synchronous connection open is the wrong shape.

The correlation match is one-shot: a verdict resumes exactly one waiting instance and the entry is consumed, which is what keeps the pattern correct under concurrency. Expiry is the application's responsibility, and the conventional answer is a boundary timer on the receive task - if no verdict arrives inside the deadline, the instance takes its exception path, which almost always means manual review. A detector that has gone quiet must not be able to advance anything by default.

There is a third option worth knowing about, which is neither of the above: score the trajectory while the case is still running, over a bounded trailing window. For a case that stays open for weeks, a verdict that only arrives at completion is a verdict that arrives too late to act on. Scoring in flight is how a long-running workflow gets judged while there is still something to do about it.

9. One catalogue, consulted at dispatch

Every declared component lives in a single catalogue. It is not a runtime bus - it does not move messages - it is the configuration the engine consults when deciding where a message goes. Registration refuses duplicates rather than silently overwriting them, which turns a misconfigured restart into an error at startup instead of a routing mystery three days later.

Because it is configuration rather than code, the whole catalogue serialises. It can be restored at startup from a file, a database or a remote config service, which means the integration topology is a deployable artefact you can diff, review and roll back like anything else. One caveat worth stating plainly: the catalogue and your process definitions are not cross-validated at registration time. A channel named in a process but never declared, or an endpoint pointing at a service that does not exist, surfaces when a message is dispatched and the lookup comes back empty. Validate the pair in CI; the engine will not do it for you.

10. Operational guidance

Where the cost actually is

The catalogue itself is negligible - a typical production topology is tens of channels, a handful of endpoints, a few routers and pipelines, and lookups are constant-time. Condition evaluation is a lightweight interpreter: simple comparisons are constant, compound conditions are linear in the number of conjuncts. The one thing to avoid in a hot path is a condition that iterates a list; pre-compute that value before the message enters the pipeline and compare a scalar.

The real cost is where the companion article measures it: the corpus and the geometry, not the plumbing. At 100 000 instances on a single node the entire pipeline - process execution, decision evaluation, case handling, geometric scoring and narrated findings - sustains roughly 443 instances per second at about 220 MB resident.

Fan-out and back-pressure

A Splitter has no built-in fan-out limit. For batches beyond a few thousand records, chunk at the source and let each chunk traverse the pipeline independently. This keeps the Aggregator's correlation table bounded, and it gives you a natural restart granularity when a bulk import fails halfway.

Choosing the right shape

Conclusion

Detection is the easy half. What determines whether an anomaly detection project survives contact with production is everything around it: how the corpus is bootstrapped, what the system does before it has learned anything, where a verdict goes once it exists, and whether a human can find out why.

Two topologies cover that lifecycle. The big-data path fans historical records out, deduplicates them, registers them in parallel and announces when the corpus is usable. The fast-data path filters, translates, scores and routes a live completion - down three branches, not two, because "not enough evidence" is a real answer that deserves real handling. Both are declarations in one catalogue rather than code in a dozen services, which is what makes the question "what could send a message to the fraud queue?" answerable by reading rather than by grepping.

Start with the agentic credit tutorial to see the process, decision and case APIs this is built around, and read the companion article for what the scoring step actually measures - including, importantly, the class of anomalies it cannot see at all.