Data Routes

Before we end this tutorial, we'd like to cover one more topic: Routing. As we've gone through creating three distinct Pipelines, configuring a Source, and configuring a default Destination, you might have been asking yourself how those three components come together to take events through the Cribl Stream process. The answer is Routing. Routing has two forms in Stream: Data Routes and QuickConnect.

In this first section, we'll look at Data Routes. A Route has three key components:

• A filter to identify which events will be included. Usually, that filter is based on an InputId, a sourcetype, or some other key-value pair found in the incoming event.
• A Pipeline to apply to that event.
• A Destination.

Data Routes also provide the option to clone an event. The clone will maintain the original format of the event, and will be applied to the next Route in the list.

Configure the Filesystem Destination

To demonstrate, let's first create a new Destination, and then create a Route to send events to that Destination.

important

1. With Manage active in Stream's top menu, select the Data submenu then click Destinations.
2. From the resulting tiles, locate and select Filesystem.
   You can use the search box to find it.
3. On the Manage Destinations / Filesystem page, click Add Destination in the upper-right corner.
4. Enter local in the Output ID field.
5. Enter /tmp/out in the Output Location field.
6. Enter /tmp/staging in the Staging Location field.
7. In the Partitioning Expression enter the below:

   `${__inputId}/${sourcetype}`
8. The configuration should look like this (click to enlarge):
9. Click Save.

Configure the Route

Finally, we can create the Route that will tie this Destination together with the Source and Pipeline we created with our first lookup.

important

1. From Stream's top nav, select the Routing submenu and click Data Routes.
2. Click Add Route.
3. In Route Name, enter access_common.
4. Paste the following into the Filter field, replacing the default true entry:

   __inputId=='datagen:apache_common'
5. Set the Route's Pipeline to access_common_lookup.
6. Set the Output to filesystem:local.
7. Drag the Route you've created to the top of the list. When complete, the interface should match this. (click to enlarge)
8. Click Save.

View Routed Events

Now, to check your work, let's view the events as they're being saved to your local filesystem.

important

1. If the terminal is not already open, toggle it open on the bottom right.
2. At the command prompt, type or paste:

   ls -l /tmp/staging/local
   You should see a datagen:apache_common directory where our data is getting pushed.
3. At the prompt, now type or paste:

   ls -lR /tmp/staging
   You should see several new .tmp files created by Stream as out data flows through the route we just created.