Skip to main content

Cribl/Regex Enchantment Tools

As we mentioned earlier, Cribl Stream ships with a Regexes Library that contains a set of pre-built, common regex patterns. This allows the muggles among you to pass as powerful witches and wizards, with an easily accessible repository of regular expressions (a.k.a., spells). The Library is searchable, and you can assign tags to each pattern to organize and categorize them.

Let's look at the Library.

important
  1. In the Stream UI's top nav, make sure Manage is active.
  2. From the submenu, select Processing > Knowledge
  3. From the Knowledge Library's left tabs, select Regexes.

Now let's see regex in action.

important
  1. In the Stream UI's top nav, make sure Manage is active.
  2. From the submenu, select Processing > Pipelines.
  3. On the Pipelines page, find and click the palo_alto_traffic Pipeline.
    (To display this first column's header and contents, you might need to drag the pane and column dividers toward the right.)
  4. On the palo_alto_traffic Pipeline page's left edge, click the Regex Extract Function's accordion to expand its configuration.
  5. To open the Regex validation modal, click the Advanced mode button on the right edge of the Regex field.

In the resulting Regex validation modal, regex patterns from the Regexes Library will appear in the Regex search field as typeahead options. Click a pattern to paste it into the field below. You can then use the pattern as-is, or modify it as necessary.

You can also add new, custom patterns to the Library. In the same modal, once you've built your pattern, click the Save to Library button at the top.

You also have the option of setting flags for your regex search patterns by clicking a flag button. There's one in the Regex validation modal (shown below), and another one back in the Regex Extract Function's configuration (on the right side of the Regex field). Cribl adopts the ECMAScript flavor of regex, which supports the six flags shown below.

To visually inspect your sample data, enable Render Whitespace in the modal: Click the gear button at the right, to toggle between displaying special characters (carriage returns, newlines, tabs, and spaces) as white space, versus as the respective symbols ␍, ↵, →, and ·.

Enabling Render Whitespace can be especially helpful when you're working with Windows events, which might contain hidden characters that will throw off your regex logic.

important
  1. Click Cancel to close the Regex validation modal. This will return you to the palo_alto_traffic Pipeline page.

The Regex Cheat Sheet​

Here's a non-exhaustive regex cheat sheet – a character dictionary to build your book of spells. You can concoct different characters into an incantation (or pattern, for you muggles) that matches the data you want to capture.

Word of caution: Don’t start out attempting to string together a complex regex pattern in one go. Break up your syntax into small chunks, and iteratively test it out with sample data.

note
  • * 0 or more matches
  • + 1 or more matches
  • ? 0 or 1 match
  • ^ matches the start of the string or line
  • $ matches the end of the string or line
  • \ signifies an escape sequence
  • . matches any single character except newline
  • ( ) capturing group; save to reuse later
  • | used as a logic OR inside a capturing group
  • [abc] character set; matches one of the things in the brackets
  • [^abc] negated character set; matches anything except...
  • [a-zA-Z] all uppercase and lowercase letters; you can specify any range.
  • {1} exact number of matches; comes after a set or group
  • {1,5} inclusive range for number of matches
  • {1,} minimum number of matches
  • {,5} maximum number of matches
  • \s any whitespace character
  • \S any NON-whitespace character
  • \d any digit; same as [0-9]
  • \D any NON-digit
  • \w any word character; same as [a-zA-Z0-9_]
  • \W any NON-word character; same as [^a-za-z0-9_]
  • \b word boundary
  • \B NON-word boundary
  • [\b] - escape sequence for a backspace character
  • \u0404 - 4-digit unicode hex value for a character

To learn more about regex, check out https://regex101.com/. This is a rich (and free) resource for beginners and advanced users alike. You can create, test, and save your regex, as well as engage the community to ask questions, in real time.

Next, you'll see how we turn these characters into spells, as we apply them in our scenarios.