Skip to main content

The File Header Breakers

One challenging type of data stream to work with consists of events that originate inside a file with this structure: field names in the first row, and field values in the subsequent rows. Some examples of this are Microsoft IIS Logs and Bluecoat Proxies. Both systems use the W3C format for their logs.

In this module, you'll work with a sample of this data to correctly extract the field names.

The File Header Event Breaker​

important
  1. Close any modals, if open.
  2. Navigate to Processing > Knowledge.
  3. In the left nav, select Event Breaker Rules.
  4. Click the blue Add Ruleset button at top right.
  5. Click the Add Rule button.
  6. Try out this Event Breaker, by copying and pasting the JSON array into the input section. Click anywhere in the input pane, outside the field labeled Paste your events here or upload a sample file.
#Software: Microsoft HTTP Server API 2.0
#Version: 1.0
#Date: 2002-05-02 17:42:15
#Source: https://learn.microsoft.com/en-us/windows/win32/http/w3c-logging
#Fields: date time c-ip cs-username s-ip s-port cs-method cs-uri-stem cs-uri-query sc-status cs(User-Agent)
2022-05-02 17:42:15 172.22.255.255 - 172.30.255.255 80 GET /images/goats/pygmy.jpg - 200 CRIBL/4.0+(compatible;+GOAT+4.0)
2022-05-02 17:42:16 172.22.255.255 - 172.30.255.255 80 GET /images/goats/fainting.jpg - 200 CRIBL/4.0+(compatible;+GOAT+4.0)
2022-05-02 17:42:17 172.22.255.255 - 172.30.255.255 80 GET /images/goats/nubian.jpg - 200 CRIBL/4.0+(compatible;+GOAT+4.0)
2022-05-02 17:42:18 172.22.255.255 - 172.30.255.255 80 GET /images/goats/kinder.jpg - 200 CRIBL/4.0+(compatible;+GOAT+4.0)
  1. Click the Out tab and observe that the events, while correctly broken, do not contain the field names.
  2. From the Event Breaker Type drop-down, select File Header .
  3. Change the Field Delimiter to \s

Now the preview shows the events with all fields extracted correctly. Very useful!

Fixing Extracted Event Fields​

The cs_User_Agent_ field shows a few + signs for spaces. You can change this by adding a field to the event.

note
  1. Scroll to the bottom of the Event Breaker Rule configuration.
  2. Click the Add Field button.
  3. Enter the following configuration:
    • Name: cs_User_Agent_
    • Value Expression: cs_User_Agent_.replace(/\+/g, ' ')
important

Close all modals without saving. We will not be using this data or Event Breaker later in the course.

Conclusion​

In this module, you learned how to configure a sample File Header event breaker for logs from a (fictitious) IIS server. In the next module, you'll learn how to break multi-line events.