Skip to main content

🎶 In your wildest Streams 🎶

The whole reason we made this sandbox was to showcase the power of Cribl Stream as a replacement for your Syslog-NG deployment. Understandably, you’re not going to go in and rip out Syslog-NG for Cribl Stream overnight. As syslog needs scale up, Cribl Edge helps stopgap some of the issues we've seen arise, giving you time to deploy Cribl Stream for your toughest projects.

Cribl Stream is our endgame, however. Here, we’ll configure Stream to:

  • Receive our syslog (rather than Syslog-NG).
  • Send the syslog data off to the SIEM and low-cost object storage (rather than using a secondary agent).
  • Use pre-made Packs to help reduce event sizes.

Buckle up, it’s gonna be a...pretty straightforward ride, actually. But you should always wear a seatbelt when in a moving vehicle...or a sandbox? The point is, Stream’s UI makes these things rather simple.

Our Syslog, Comrade​

In a real-world environment, you would install Stream on the Syslog-NG server and avoid changing the IPs of all your syslog devices. Kind of tough to do that with how our sandbox is architected, however. So we have configured syslog data to send to both Syslog-NG and Stream. As a result, we can skip installing and get right to configuring.

Navigate to Stream
  1. Navigate to Cribl Stream
    1. On the top tabs, click Cribl.
    2. At Edge's top left, hover over Edge to unroll the product switcher.
    3. From the product switcher, click Stream.
  2. Click into the goatfarm Worker Group.
  3. Once in Stream, click the large QuickConnect tile on the left.

Look familiar? It's the same UI we used in Cribl Edge to quickly connect our Sources and Destinations. This time, we'll do two things differently: Use a syslog Source, and a Pack to condition our syslog data.

Luckily, someone has achieved their Cribl Certified Observability Engineer User certification, and was already in here configuring a stream-syslog Source, an s3 Destination, and our splunk Destination. All that's left for us to do is add our Pack and connect our Destinations.

A Pack a Day​

Remember when we said that Stream comes with pre-made Packs that can help us reduce event sizes? Let’s see how simple they are to deploy, how much data reduction they can offer, and [need a joke here]. (See what we did there? We reduced the size of the joke.)

Condition the Syslog
  1. With the Manage tab active, click Processing from the submenu, and then select Packs. If the Processing link is not visible, click the submenu's ••• overflow menu to reveal it.
  2. Click Add Pack and select Add from Dispensary.
  3. Search for and click on Syslog Pre‑Processing.
    • Feel free to take a minute and read what the Pack will do for us.
  4. From the Syslog Pre‑Processing drawer, click Add Pack.
  5. Click X to close the Syslog Pre‑Processing drawer.
  6. Click X to close the Add Pack from Dispensary drawer.

This Pack contains a CriblSyslogPreProcessing Pipeline with over 60 Functions in it, the content of which is beyond the scope of this sandbox. Suffice to say, it does great things for syslog data. Let's go add it to our stream‑syslog Source and finish out this journey.

From Stream to Shining SIEM​

We have our Sources and Destinations configured, as well as our Syslog Pre‑Processing Pack. To finish out, we need to mirror the initial Syslog-NG + Splunk UF configuration. Or rather, since Security Operations enjoyed having the low-cost, full-fidelity archive, we’ll be mirroring the Syslog-NG + Cribl Edge configuration, to send syslog data to both the SIEM and S3.

Condition the Syslog
  1. Navigate back to QuickConnect: Manage > goatfarm > Routing > QuickConnect.
  2. Hover over the stream-syslog Source and click Configure.
  3. In the resulting drawer, click the Pre‑Processing left tab.
  4. For the Pipeline, select PACK cribl‑syslog‑input (Syslog Pre‑processing).

Awesome! This pack will clean up our syslog data and reduce its volume somewhat. However, it does set the sourcetype to syslog. In the real world, this is fine. In our sandbox, though, we have been using sourcetype to illustrate that we are, in fact, sending data to the SIEM. So, real quickly, let's add another JavaScript expression to label our Fields in order to set our sourcetype. This one will be more for checking our work than actually being usable in a real environment.

Set the sourcetype
  1. Just above the Pre‑Processing left tab, click Fields.
  2. Click Add Field.
    • In the Name column, put sourcetype.
    • In the Value column, put: 
      host.includes("syslog") ? "stream-pan" : "stream-apache"
It's not pretty, but it works

Like we said above, this conditional expression is here merely to get the job done. Our syslog data coming from the Palo Alto firewall includes syslog in the hostname. Our Apache web server sends from hosts web*.cribl.io. This conditional simply looks for syslog and labels it as pan traffic.

  1. Click Save

Now that our input is being pre-conditioned with a few nice-to-haves (e.g., automatic sourcetype and index addition), let's configure our Routes, Commit and Deploy, check our work, and close out this sandbox.

Send the Syslog
  1. Connect stream-syslog to both s3 and splunk. (Drag the + on stream-syslog over to each of the two Destinations.)
  2. In each resulting pop-up, leave the default Passthru selection and click Save.
Can we use more Packs?

Some of our syslog input for this sandbox comes from Palo Alto Networks firewall logs. We do have a Pack to help reduce unnecessary fields (and more), but that isn't in the scope of this sandbox. We actually have a different sandbox to go over that.

  1. At the Stream UI's top right, click Commit .
  2. In the resulting Git Changes modal, enter a descriptive message, and click Commit and Deploy.

Once again, that's it! With just a little bit of configuration, you were able to configure our Stream deployment to receive syslog directly, and then pass it on to S3 for low-cost, full-fidelity retention, as well as on to the current SIEM. In the future, if you need to switch SIEM vendors or investigate a data breach, you have the infrastructure in place.

Let's wrap up.

Check your work

Well sure, we configured it, but how do we know it all works?! Don’t take our word for it! Feel free to go back to the Splunk UI (Splunk top tab) and search for the syslog sourcetype.

index="syslog" sourcetype="stream-pan"
index="syslog" sourcetype="stream-apache"