Connecting Sources & Destinations

Course Map

A Data Route specifies the path that data should take within Stream. In simpler terms: A Data Route connects Sources and Destinations.

important

Select the Routing submenu, and click Data Routes

As we can see, Stream comes with its own default Route. Expanding it (by clicking the little arrow on the left) reveals a few details about this Route that we can use to make our own Route. However, it’s easier to learn by doing!

important

Create a new Route

1. In the top right, click Add Route
2. Name the Route raw2s3
3. Leave the Filter set to its true default
4. Select passthru under Pipeline
5. Leave the Enable Expression field set to its No default
6. Select s3:archives3 for the Output
7. Fill out the Description with something… descriptive
8. Toggle Final to No
9. Click the top left of the Route (next to the minimized arrow) and drag the new Route to position 1
10. Click Save

What makes a Data Route

Every Data Route starts with a Name. By now you probably sense a pattern; everything starts with a Unique Identifier (Name) so that it can be called elsewhere in the config or internal code.

After the name we get to the Filter. Filters define when a route is used. When a Filter evaluates to true, data will go down that Route. In our use case, we want this to apply to all data coming through Stream. So, we can leave the default as true. We’ll get into more advanced Filters next time.

A Pipeline is a group of Functions used to transform, enrich, reduce, or redact data. If we want to send all data unaltered to S3 for archival purposes, we want to simply let it passthrough. In other words, we want no Functions in the Pipeline. Stream has a built-in passthru Pipeline that is aptly named.

Next we move on to Output. This is our Destination. Our Route's purpose is to put our data into S3, so let’s choose archives3.

Lastly, we need to add a Description because we want everyone to be able to understand what we were trying to accomplish.

If we read the route as a sentence it would go something like this, ‘Every piece of data coming into Stream gets passed through to archives3. Good work!

Final?

Before we finish, we can cover the Final toggle. This toggle determines whether or not the data continues to any other Routes we have further down. If Final is enabled, the data stops here. Otherwise, it goes back into the processing engine and hits the next Route that evaluates to true.

At the moment this doesn’t matter because we have one real Route and don’t care if the data continues on, but since we plan to build more Routes, let’s leave Final disabled.

Fin!

Done! Well kind of. If you believe everything works, we’re done. If you prefer proof, we can go check that everything is flowing.