Skip to main content

Conjuring Tricks to Create New Fields

We can also use regex to conjure (or extract) fields from an event. This is serious magic, folks, so exercise it with prudence. For example, some syslog events don't have the required host field, and missing or incorrect host fields will break some reports and analysis.

In this example, we'll create a Pipeline that extracts host fields from the sample log file's events, using the Regex Extract Function.

important
  1. In the Stream UI's top nav, make sure Manage is active.
  2. From the submenu, select Processing > Pipelines.
  3. On the Pipelines page, find and click the Beginner_Extract_Host Pipeline.
    (To display this first column's header and contents, you might need to drag the pane and column dividers toward the right.)
  4. In the right Preview pane, make sure Sample Data is selected.
  5. Click the Simple link at the lower right beside the sample.log file.
  6. At the end of any event's _raw field, notice the string from host, followed by a space and the host's name.
  7. Click Add Function near the top of the left pane, and either find Regex Extract in the Standard submenu, or type Regex Extract into the search box to locate it. Then click Regex Extract to add this Function to the Pipeline.
  8. Leave the Filter field default at its default true value.
  9. Enter a simple Description for the Function.
  10. In the Regex field, paste the following: 
    (host)\s(?<host>.*)
  11. Leave the Source_Field at its default _raw value.
  12. Click Save to save the Regex Extract Function to your Pipeline.
  13. Click the OUT button, near the top of the right Preview pane, to see how your regex has transformed the events: Each event now contains a host field (and value), extracted to the top level, parallel to _raw.

Let's unpack this. With the Regex Extract Function, you get to flex your magic powers by extracting and conjuring up new fields from an event. In this example, each event's _raw field had this format:

2021-10-26 17:00:12 PDT sample log data from host eagle1

Our regex literal needs to capture both the name and value of the field, in the following format:

(?<_NAME_0>host)\s(?<_VALUE_0>.*)

In our example, that translates to: (host)\s(?<host>.*)

We captured (host), and addressed the white space between the host and its name by using the character: \s.

Next, let's discuss filtering real-time events (or making things disappear) in Cribl using...drum roll, please...regex! Or magic! Or both!