Adding a Route

In this section, we'll add a second Destination to route a copy of all events to the filesystem. We'll learn a bit more about backpressure, and will show interactively how Stream behaves when one Destination is online but another is not.

Writing events to the filesystem is a very handy capability, and we often use it to interconnect to cheap storage like a NAS with NFS mount, or to systems like Hadoop which have a filesystem driver. After adding the Destination, our setup will look like this:

Output Diagram

important

Add fs Destination

  1. With Manage active in the top nav, select the Data submenu and click Destinations.

  2. Locate and click on the Filesystem tile.
    You can use the search box.

  3. At the upper right, click + Add New.

  4. In Output ID, put fs.

  5. In Output Location, put /tmp/out.

  6. In Staging Location, put /tmp/staging.

    Your fs Destination should look like this:

    FS Output

  7. Click Save.

One other thing to note on this UI page is the Partitioning Expression. The Partitioning Expression is another JavaScript expression, used to determine the path to prepend to the filename. The entire directory structure will be created if it does not exist. This is also used in S3 and other outputs where the eventual reader of the data would like events segmented by the metadata available in the event.

In this case, we've used the default, which will put events into a file per unique combination of ${host}/${sourcetype}. This can get much more sophisticated, and usually we'll partition by time, as well as by a number of other metadata fields. Partitioning allows consuming systems to quickly rule out files they do not need to query, by looking at the directory structure before opening and scanning files.

We've added the output Destination, but next we need to add it to a Route so events will begin flowing to this output.

important

Add Route

  1. Select the Routing submenu at the top and click Data Routes.
  2. In the default Route, click the ... menu.
  3. Click Insert Route Above.
  4. In the new Route's Route Name field, put fs.
  5. Set the Pipeline drop-down to passthru.
  6. Set the Output drop-down to filesystem:fs.
  7. Click Save.

Your Route should look like this:

FS Route

This should be pretty straightforward. We're creating a new Route, matching everything (with Filter set to true), and sending it unmodified – via the passthru Pipeline – to the filesystem.

The setting I want to focus on here is Final. Right now, Final is set to Yes, so all events are now heading out this output. You'll notice in the terminal below, assuming you're still running nc, that events have stopped flowing to the tcpjson output. If you click on Quick Stats and then Outputs in the right pane, you'll notice that events are now heading out the fs output and not the tcpjson output.

Output Stats

Let's validate that events are indeed flowing to the filesystem.

important

Run tail against files

  1. In the terminal, type ^C or (Ctrl-C) to kill anything running.
  2. Once you get the command prompt, run this tail command: 
    tail -qf /tmp/staging/*/*/*/*/CriblOut*.json.tmp | jq .

You should see output similar to what you were seeing before with nc. However, by setting the fs Route's Final toggle to Yes, we are no longer getting output through our tcpjson output. This is because we placed its default Route below fs in the UI. Let's change that Final setting.

important

Disable Final

  1. Under Routing > Data Routes, make sure your new fs Route is expanded.
  2. Set Final to No.
  3. Click Save.

After changing this, you should notice no data is being written to the file or to tcpjson. Why is this?

Once again, we're hitting "backpressure." Because we cannot push out the tcpjson output, events are failing to push to the filesystem as well. If you begin receiving the output from the tcpjson Destination again, you'll start to see events flow again.

important

Listen for TCP JSON Traffic

  1. Type ^C in the terminal.
  2. Run nc with these options: 
    nc -lkp 42000 > /tmp/nc.log &
tail -f /tmp/nc.log | jq .
  3. Wait, it can take a minute for traffic to flow.
  4. (Optionally:) Hide the terminal.

In the right pane, if you select Quick Stats (from the ... menu), then select the Outputs tab, you should see that events are now flowing out both Destinations.

What if we want to not require tcpjson to be available? We can change that preference by changing the output's Backpressure Behavior in Destinations.

important

Change tcpjson Backpressure Behavior

  1. Select the Data submenu and click Destinations.
  2. Find and click TCP JSON tile.
    You can use the search box or the Confgured Only control.
  3. Click the tcpjson output to expand it.
  4. Change Backpressure Behavior from Block to Drop Events.
  5. Click Save.

Events should now flow, whether or not we're listening for events on port 42000.

Next, in the Find & Replace section, we're going to create a Pipeline that will allow us to parse and transform events.

← Data Routes & PipelinesFind & Replace →