Make Things Go Poof!

How about making things simply disappear, you ask? Rest easy, we goat you there, too. To filter events in real time (data in motion), we can use the out-of-the-box Regex Filter Function. This is similar to nullqueueing with TRANSFORMS in popular SIEMs, but the matching condition is way more flexible.

In this example, we'll filter out any client_port=='2021' events with a _raw field containing the pattern Removed.

important

1. In the Stream UI's top nav, make sure Manage is active.
2. From the submenu, select Processing > Pipelines.
3. On the Pipelines page, find and click the Beginner_Regex_Filter Pipeline.
   (To display this first column's header and contents, you might need to drag the pane and column dividers toward the right.)
4. In the right Preview pane, make sure Sample Data is selected.
5. Click the Simple link at the lower right beside the regex_filter.log file.
6. Click Add Function near the top of the left pane, and either find Regex Filter in the Standard submenu, or type Regex Filter into the search box to locate it. Then click Regex Filter to add this Function to the Pipeline.
7. In the Filter field, paste the following:

   client_port=='2021'
   (You can add more conditions to filter on, as needed.)
8. In the Regex field, paste the following:

   Removed
9. In Field, leave the default _raw value unchanged, to apply this regex to the _raw field.
10. Select Save.

To check the results, in the right Preview pane, select the OUT button. Then scroll down through the events. Any event whose _raw field contained the text Removed should now be struck out, indicating that the event has been filtered out.

You get to flex your magic powers by filtering incoming events in real time – or by making them disappear – based on virtually any arbitrary conditions.