Regex Challenge

Now let's see how Stream can help us meet our next challenge:

Sort events in the same log into different sourcetypes, even if they have different structures.

With our last example, we could use an "exact match" lookup because, while each event in that Source had different values, the overall structure of all events was the same. That meant that we could extract fields from every event with the same regular expression and be confident we were capturing the right fields and working with the right key-value pairs.

With this next example (click to enlarge), we have no such luxury:

Our sample data comes from firewalls, where "threat" events (comma-separated events with 41 fields) are intermingled with "traffic" events (comma-separated events with 46 fields). No single regular expression could capture the correct values from both types of events, so we'll need to leverage the regular expression option in Cribl Stream's Lookup Function.

In this scenario, we want to assign the correct sourcetype to each event before we send it along its way. This reproduces some of the event modification that customers of popular SIEMs usually perform with a middle layer transformer and a technical add-on. Using Cribl Stream for this kind of event modification can – for many customers – enable them to retire some or all of their middle layer transformers, simplifying their SIEM architecture and reducing their overall infrastructure costs.