Skip to main content

The Lookup

At long last, we're ready to configure the lookup. First, let's create the Lookup table we'd like to use.

Getting the goods

For this next portion you will need a CSV file. Download the CSV of status codes here: HTTP Status Codes. The CSV will be downloaded to the default location configured for your browser. This is usually your Downloads folder.

Add the Lookup Table​

important

  1. With Manage active in Stream's top nav, select the Processing submenu then click Knowledge.

    Note: Depending on the size of your window, the top nav will consolidate items that won't fit in a pulldown represented by an ellipsis (...) - if so, click on the ellipsis and then select Processing and click on Knowledge.

  2. If Lookups is not already selected in the left sidebar, click to select it.

  3. At the upper right, click Add Lookup File. This will allow you to create a lookup in one of two ways (choose whichever you prefer):

    Method 1: Upload a File
    1. Select Upload a New File.
    2. Navigate to the HTTP Status Codes CSV file that you've downloaded, then select it and click Open.
    3. Type or paste http_status_codes.csv into the Filename field.
    4. Click Save.
    Method 2: Create within Stream
    1. Select Create with Text Editor.
    2. Open the HTTP Status Codes CSV file that you've downloaded, then select all the text, up to the last character, and copy it to your clipboard.
    3. Back in Stream's Add Lookup File modal, paste the clipboard contents into the large text field.
    4. Type or paste http_status_codes.csv into the Filename field. Your interface should now look like this (click to enlarge): Pipelines
    5. Click Save.

View the Lookup in the Terminal​

In Stream, Lookup tables are simple CSV-formatted files that can be modified through Stream or through other scripts and programs. This flexibility enables a wide variety of use cases. For example, a script could update a Lookup table, on a daily basis, with the 1,000 most commonly accessed websites. This way, events containing requests to those popular sites could be filtered out, dramatically reducing data volume while improving your ability to identify anomalies.

important
  1. If the terminal is not already open at the bottom right, display it.
  2. At the command prompt, type or paste: 
    ls -l /opt/cribl/data/lookups

You should see a list of this Stream instance's Lookup tables, including the one we just created.

  1. At the prompt, type or paste: 
    head /opt/cribl/data/lookups/http_status_codes.csv

You should see the first several lines of the Lookup table we just created.

Configure the Lookup​

Now that the Lookup table is saved, we are ready to configure the Lookup Function.

important
  1. Select the Processing submenu, then click Pipelines.
  2. Choose the access_common_lookup Pipeline from the list of Pipelines on the left.
  3. As before, in the right Sample Data pane, click Simple next to the apache_common sample. Your interface should look like this (click to enlarge): Pipelines
  4. To add the Lookup Function, click Add Function, then Standard, then Lookup.
    Be sure to select Lookup, not DNS Lookup.
  5. Configure the new Function's Lookup file path, Lookup field(s), and Output field(s) to match this screenshot (click to enlarge). Use the +Add field(s) buttons as needed: Pipelines
  6. Click Save.
  7. In the right Preview pane, toggle between IN and OUT. When selecting OUT, you should see the lookup values added to each event. Pipelines

Configure Sampling​

Finally, for the payoff: Let's sample events that are of type Successful, in order to reduce indexed volume without reducing your ability to identify anomalies.

note

Skip these steps if you're continuing from the preceding steps:

  1. Select the Processing submenu, then click Pipelines.
  2. Choose the access_common_lookup Pipeline from the list of Pipelines on the left.
  3. As before, click Simple next to the apache_common.log sample on the right. Your interface should look like this (click to enlarge): Pipelines
important
  1. To add the Sample Function, click Add Function, then Standard, then Sampling.
  2. Configure the new Sampling Function to match this screenshot (click to enlarge). This sets up a 20% sampling rate for events of type Successful: Pipelines
  3. Click Save.
  4. In the right Preview pane, toggle between IN and OUT. When selecting OUT, you should see that some events are faded and crossed out. Stream will omit those events when sending to destinations.
  5. Finally, in the Preview pane, click on the Pipeline Diagnostics icon () Note: If your particular data sample didn't include events of type Successful, you will see no events faded and crossed out. You can remedy this by taking another sample of the data.