Skip to main content

Connecting Sources & Destinations

A Data Route specifies the path that data should take within Stream. In simpler terms: A Data Route connects Sources and Destinations.

important

Select the Routing submenu, and click Data Routes

As we can see, Stream comes with its own default Route. Expanding it (by clicking the little arrow on the left) reveals a few details about this Route that we can use to make our own Route. However, it’s easier to learn by doing!

Create a new Route
  1. In the top right, click Add Route
  2. Name the Route raw2archive
  3. Leave the Filter set to its true default
  4. Select passthru under Pipeline
  5. Leave the Enable Expression field set to its No default
  6. Select cribl_lake:cold_storage for the Output
  7. Fill out the Description with something… descriptive
  8. Toggle Final to No
  9. Click the top left of the Route (next to the minimized arrow) and drag the new Route to position 1
  10. Click Save
  11. In the top right, click Commit.
  12. In the resulting modal, click Commit & Deploy at the bottom right.

What makes a Data Route​

Every Data Route starts with a Name. By now you probably sense a pattern; everything starts with a Unique Identifier (Name) so that it can be called elsewhere in the config or internal code.

After the name we get to the Filter. Filters define when a route is used. When a Filter evaluates to true, data will go down that Route. In our use case, we want this to apply to all data coming through Stream. So, we can leave the default as true. We’ll get into more advanced Filters next time.

A Pipeline is a group of Functions used to transform, enrich, reduce, or redact data. If we want to send all data unaltered to Cribl Lake for archival purposes, we want to simply let it pass through. In other words, we want no Functions in the Pipeline. Stream has a built-in passthru Pipeline that is aptly named.

Next we move on to Output. This is our Destination. Our Route's purpose is to put our data into object storage, so let’s choose cold_storage.

Lastly, we need to add a Description because we want everyone to be able to understand what we were trying to accomplish.

If we read the route as a sentence it would go something like this, ‘Every piece of data coming into Stream gets passed through to cold_storage. Good work!

Final?​

Before we finish, we can cover the Final toggle. This toggle determines whether or not the data continues to any other Routes we have further down. If Final is enabled, the data stops here. Otherwise, it goes back into the processing engine and hits the next Route that evaluates to true.

At the moment this doesn’t matter because we have one real Route and don’t care if the data continues on, but since we plan to build more Routes, let’s leave Final disabled.

Fin!​

Done! Well kind of. If you believe everything works, we’re done. If you prefer proof, we can go check that everything is flowing.