Skip to main content

What's in the Box?

The first group is well-named: Extraction and Reduction. Oh, and they used the descriptions for good documentation! Here we’ll go over the broad strokes and if you would like, you can expand the specific Functions to see what exactly is going on. The line numbers below correspond to the Function numbers in Stream.

Functions 1 - 5​

  • 1 - This Function is actually a comment! Ed took care to explain the rest of the functions for us! How nice!
  • 2 - Circuit Breaker to keep non-matching data out. If it’s not from the firewall, we don’t want to try enriching it.
  • 3 - Regular Expression Extraction to pull out relevant fields. This looks into the _raw and pulls out certain values that we care about, like log_subtype.
  • 4 - Drop events of log_subtype==start, as they often have incorrect app assignments. This will help lower SIEM usage.
  • 5 - (disabled) Sampling: Sample traffic with 0 bytes received at 5:1, Sample traffic from trusted-trusted & action allowed at 10:1. Also meant to help lower SIEM license, but still tinkering with the sampling rates.
    • This function is disabled because Ed is still working on it.

Take special note of number 4. Stream Packs allow for users with high levels of domain knowledge to transfer that knowledge easily. Here we see that Ed, being a Palo Alto expert, knows certain logs are not useful to keep. Thanks Ed!

The next group of Functions looks up where the IP addresses inside the logs are geographically located. It has a few extra steps to make sure things stay clean.

Functions 6 - 9​

  • 6 - Another comment explaining the following functions. Gotta love good documentation!
  • 7 - Look up source or destination IP against a GeoIP file. The file is located in the pack Knowledge tab.
  • 8 - Get rid of extra fields that the GeoIP lookup added.
  • 9 - Take the rest of the fields that GeoIP added and flatten them all into one field for brevity.
note

This Pack includes a sample GeoIP file for demonstration purposes only. To apply GeoIP services in your actual Stream deployments, use a ‘real world’ database.

The final group assigns labels (e.g. Home Woker Nets) to Source and Destination IPs against data given by IT using the Lookup Function.

The rest of the Functions are as follows:

Functions 10 - 16​

  • 10 & 11 - Check if any of the IPs included in the list of IPs included in the Knowledge tab of the pack.
  • 12 - And one more comment. If you couldn't tell, these descriptions are basically copy pasted from the comments.
  • 13 - Check if any IPs appear in a list of compromised IPs also included in the Knowledge tab of this pack.
  • 14 - Get rid of message field as it is a duplicate of the _raw field.
  • 15 - Rename the leftover GeoIP fields to names that are easier to read.
  • 16 - (disabled) Push all the new fields back into raw. Disabled for now, Ed will check how it works with Elastic once in production.