Skip to main content

Data Routes & Pipelines

This section explains the core Stream concepts of Data Routes & Pipelines. There are no interactive sections, so this section is optional.

After we receive an event, we run it through a series of Data Routes to determine how to process that event. Let's look at Cribl's data flow.

Routes & Pipelines

Data Routes​

Routing runs each event through a series of filter expressions to determine how to process the event. The event can optionally be sent down multiple Pipelines, for sending to multiple Destinations – potentially processed differently for each Destination.

Routes

Events proceed linearly down the configured set of Data Routes, matching against each filter expression serially. If a given filter expression evaluates to true, the event is sent down the configured Pipeline for processing, and sent out the configured output for that Route. If a Route is marked Final, the event is consumed, and will only go out that output. If Final is set to No, a copy of the event is sent down the Pipeline, and the original event continues down the Route list for matching against further Data Routes.

Routes

A final rule – called default, with a Filter expression of true – ensures events go out the default output if no other Route matches. If we insert Stream into an existing Pipeline, we want to be configured – by default – to send everything out the same as it came in, so that we are merely a bump in the wire.

Pipelines​

After matching a Route, events go down a Pipeline. Pipelines, structured similarly to Data Routes, are composed of a series of Functions that work on the data. Each Function executes for a given event if its filter expression evaluates to true.

Pipeline

We ship with a number of Functions, many designed to work specifically with logs, like our Parser Functions. Functions can aggregate, enrich, mask, sample, suppress, or perform a number of other operations on the data. Stream can also be extended with custom Functions, written in JavaScript. Custom Functions have all the same functionality as shipped Functions, including a rich configuration definition language based on JSON Schema.

Next​

Next, we'll route an event to multiple Destinations.