Skip to main content

🎶 Look Around! Look Around! 🎶

Let's take a look at our common deployment for syslog collection in a standard IT environment. In case you forgot, the architecture looks like this:

architecture again

How can we look at this? Well, we can verify it in a few ways. First, let’s check the SIEM deployment to see if data is flowing in.

Run a quick Search
  1. On the top tabs, Click Splunk.
  2. Once Splunk's UI spins up, which could take 20-30 seconds, click Search & Reporting.
  3. Syslog data from the Splunk UF should be flowing into the SIEM under our syslog index. Let’s copy/paste the expression below into the search box. Then accept the default time window, and click the magnifying-glass button to search: 
    index=syslog
  4. Observe the results showing (unaltered) syslog messages.
  5. Below the events, note the sourcetypes associated with our logs (pan && apache).
There is no data?

When the sandbox first spins up, it can take about a minute for data to flow into Splunk. If you don't see data, wait a minute and search again.

You can also try a wildcard search using *.

OK, great, the SIEM is receiving the messages. Let’s check how they’re being written to our Syslog-NG server. We have two ways of doing this:

  • Click the Terminal top tab, then enter commands of this form to list the log file and its contents (details below): 
    cat /var/log/*/*
  • Alternatively, we can install Cribl Edge on the Syslog-NG server, in order to help us:
    • Verify that our syslog events are received and recorded to disk, by inspecting the file system.
    • Monitor the files, and route events to alternative destinations (e.g., low-cost object storage).

Why Not Both?​

Hooray for sandboxes! We have configured the environment to handle both options. This also benefits us (Cribl), since we can demonstrate the advantages of yet another product, Cribl Edge.

Using the Terminal​

Let’s start with the simpler of the two methods: showing the data on disk by copy + pasting the commands below.

Read a wall of text
  1. On the top tabs, click Terminal
  2. In the terminal, copy and paste the following commands serially: 
    cat /var/log/apache/*
    cat /var/log/pan/*
How did you know that?

How do we know to cat /var/log/*/*? Well, for one, I wrote the course! Jokes aside, our Syslog‑NG server config writes out to two directories: /var/log/apache && /var/log/pan. Inside each, we write to a file named ${YEAR}-${MONTH}-${DAY}.log. Hence, the instructions just include a * for the YMD prefix. I don't know when you'll be doing this sandbox!

The syslog-ng.conf file we're using is available here if you want to take a closer look:
@version: 4.2
@include "scl.conf"

source s_network {
  default-network-drivers();
};

filter f_pan { 
  match("syslog", value("HOST")); 
};

filter f_apache { 
  match("cribl.io", value("HOST")); 
};

destination d_apache {
  file("/var/log/apache/${YEAR}-${MONTH}-${DAY}.log" perm(0644) create_dirs(yes) dir-perm(0755));
};

destination d_pan {    
  file("/var/log/pan/${YEAR}-${MONTH}-${DAY}.log" perm(0644) create_dirs(yes) dir-perm(0755));
};

log {
  source(s_network);
  filter(f_pan);
  destination(d_pan);
};

log {
  source(s_network);
  filter(f_apache);
  destination(d_apache);
};

The files you saw after the above cat commands are where Syslog-NG writes all events received. If they look like an unreadable mess, that’s because they are.

Using Cribl Edge​

Let’s see how long it takes to launch Cribl Edge on the Syslog‑NG server and read those same files. Then we’ll send them to the SIEM and low-cost object storage at the same time. (You know, in case you want some extra data retention for compliance purposes without the huge cost of storing all that extra data in your SIEM.)

Install Cribl Edge
  1. On the top tabs, click Cribl.
  2. On the right Edge tile, click Manage.
  3. At the upper left of the Edge UI, click default_fleet.
  4. On the resulting Fleet page, click Add/Update Edge Node at the upper right.
  5. From the drop-down, hover over Linux and click Add.
  6. In the resulting Add Linux Node modal, we'll make a small change due to our sandbox architecture. In the Installation Directory field (lower left), change the value to /opt/edge.
  7. At the modal's lower right, click Copy Script.
  8. Click back over to the Terminal tab up top.
  9. Paste the script into the terminal and press Return or Enter.
  10. Look for a Cribl started confirmation message.

All done! You just installed Cribl Edge on the Syslog-NG server. We can now go see all the running processes and active files! Shall we?

Check out the files using Cribl Edge
  1. On the top tabs, click Cribl.
  2. If still open, close out the Add Linux Node modal.
  3. Navigate to Manage > Fleets > default_fleet.
  4. On the submenu, click Explore, and then click the Files lower tab.
Can't see any files?

It may take about a minute for the Edge Node to send data in. If you see an error when trying to view the Files tab, click between Processes and Files a couple of times.

  1. In the Files submenu, select the Manual discovery mode.
  2. Click on /var/log/apache/yyyy-mm-dd.log.

Wow, that was quick. Well, if you have installed a log shipper before, then you’ll know that was quick. If you haven’t, you’ll just have to trust us. Cribl generated an installation script for us. When we ran the script, the agent came up and connected to our Edge Leader automatically (in under a minute, I might add), and Cribl Edge automatically discovered our syslog files!

Why did we click Manual?

As noted on our excellent docs page, Cribl Edge offers three modes of file discovery:

  • Auto: Tells Cribl Edge to automatically discover files that are open for writing on currently running processes.
  • Manual: Tells Cribl Edge to discover the files within the Path (directory) and Allowlist that you specify, down to the Max depth that you specify.
  • Browse: Displays a tree view of all of your directories and files.

The reason we needed to use Manual mode is because of the architecture of our sandbox: A separate pod runs Syslog-NG and mounts the persistent volume of Syslog-NG in our terminal. As a result, Cribl Edge can't monitor the processes running from our Syslog-NG pod. (Real-world deployments would most likely have Edge running on the same host as Syslog-NG, so this shouldn't be a problem.)

Notice, however: By default, even in Manual mode, Edge monitors /var/log/* for active files as well, meaning we were able to find and read our syslog file with minimal effort.

It takes a certain person to admit this, but that was fun! We saw that data is in fact entering the SIEM. We verified this by showing the files in our terminal. And lastly, we installed Cribl Edge (quickly!) and verified our files' placement.

Let's now explore how Cribl can help us handle these syslog events at scale. And level up our deployment and make it better, faster, and stronger!