Regex. Regex everywhere.

Part of Ed's request was to add some data to the logs to make searching their SIEM better. To do this, we will also need to extract some data from the payload and label it.

important

Add a Regex Extract function

  1. Click + Function in the top right
  2. Type Regex Extract into the search (alternatively, you can click to expand Standard and clickRegex Extract)
  3. Click Regex Extract to add the Function to our Pipeline

There are a lot of Functions available in Stream. To help you locate the Function you need, a Search box is available when you click + Function.

Regex Extract is a well named Function, it extracts data from our events based on a regular expression. We are going to use it to grab Source and Destination IP addresses from the Palo Alto firewall logs passing through Stream.

First, we want all the data flowing through this particular Pipeline to pass through our Function. Therefore, we can leave the Filter as true. We will also be leaving Final toggled to No, because we want the data to continue down to the rest of the Functions we are going to add.

important

Fill out the Function details as follows:

  1. Under Description, enter extract src_ip and dst_ip from event
  2. In the Regex field, type 
    (?:[^,]*,){7}(?<src_ip>[^,]*),(?<dst_ip>[^,]*)
  3. Click Save

If you are not familiar with Regex, this expression grabs two pieces of data and labels them src_ip and dst_ip. It's instructed to pick out whatever comes after seven commas. We know how to construct the logic, because a quick internet search returns that Palo Alto logs follow a specific format.

Want to see the Regex in action?

important
  1. Click the Expand icon (Advanced Mode) on the right of the Regex field
  2. Under Sample input select pan_firewall_traffic.log
note

Stream is unique in that you can check the results of your configurations (and Regular Expressions) with real data in the UI. There is no need to guess, push the config, restart the system, and check by capturing new data every time. With Stream, you can capture your data once (easily) and check all your Functions and Regexes against it without restarting or pushing a configuration.

Looking at this window, we can see:

  • the Regex on the top left
  • matches (what the Regex grabs out of the sample) on the top right
  • our sample event on the bottom

Stream automatically loaded up our pan traffic sample and we can see how the Regex is working. On the right, we can see it is grabbing the IP addresses we were looking for and applying our label. Sweet.

← Adding a PipelineLookup - Source →