Skip to main content

Unrolling the JSON

Our first step will be to break up the events, which each contain multiple "employee" records, into individual employee events.

First, we need to parse the record.

important

Add the Unroll Function to the Pipeline

  1. From the Capture step, we should already be on the Pipelines page.
    If you are not in Pipelines, select Processing on the top navigation and click Pipelines.
  2. If you don't see sample data in the right pane, click Simple to the right of the file name that you saved in the Capture step.
  3. In the Pipelines column, click on the employee Pipeline. (To display this column's header and contents, you might need to drag the pane and column dividers toward the right.) This should bring you to the Pipeline editor page, with no Functions in the list yet.
  4. Click Add Function near the top of the page.
  5. In the resulting search box, type unroll, and click JSON Unroll in the results.
  6. In Filter, replace the default true entry by pasting in the following expression: 
    sourcetype==='employee'
  7. In the Path field, type Employees.
  8. Click Save to save this Function.

In the right pane, you should see the data change after a brief delay. Click the Basic Stats button (to the left of the "Select Fields" box, towards the top of the right pane), and a table of statistics about the sample data will pop up.

If you look at the last column of that table, you'll see the number of events in the capture (on the IN row) and significantly more events in the OUT row. In this example, it shows that we had 10 events in our capture, which were unrolled into 54 events.

Unrolled Basic Stats

If you look at the data in the right pane, you'll see that each event's _raw field now contains a new Employee record, like this:

Unrolled Data

Now, we need to extract the _raw field's contents into separate fields in the top level of the event. We'll use the Parser Function to do that...

important

Extract the JSON from the _raw field

  1. Click Add Function near the top of the page.
  2. In the search box, type parse, and click Parser in the results.
  3. Scroll down to click into the new Parser Function, below the JSON Unroll Function.
  4. In Filter, replace the default true entry by pasting in the following expression: 
    sourcetype==='employee'
  5. in Source Field, if it's not already there, enter _raw.
  6. From the Type drop-down menu, select JSON Object.
  7. Click Save to save this second Function.

Note: Even though we're already filtering the data in the JSON Unroll Function, it's a good practice to still add the filter in each Function. This way, if the first Function is deleted, the Pipeline will still properly filter the data.

At this point, the records in the right pane should change a bit, and you should see new fields for FirstName, LastName, Id, and a nesting container called Role in each record, like this:

Parsed Event

Now that we have the records split out, and parsed into fields, we need to do a little cleanup on the events...