Event Cleansing

We still have some nested values in our events, and we want to get rid of the _raw field. Let's do a little housecleaning...

important

Flatten the Event

1. Still on the employee Pipeline page, again click the Add Function button at the top right.
2. In the search bar, type flat, and select the Flatten Function. It will be added at the bottom of our Pipeline.
3. Scroll down to click into the new Flatten Function.
4. As with our other Functions, in Filter, replace the default true entry by pasting in the following expression:

   sourcetype==='employee'
5. In the Fields box, type Role.
6. Click Save to save this third Function.

You should now see two new fields in your events: Role_Department and Role_Type, as in this image.

Now, we want to discard the original _raw field. (Because we're sending this data to plain text files in a filesystem, we no longer need this field. If we were sending the data to a destination that expects a _raw field – like popular SIEMs – we'd retain a _raw field and skip this step.)

important

Discard _raw Field

1. Click the Add Function button at the top right.
2. In the menu's search bar, type Eval, and select the Eval Function. It will be added at the bottom of our Pipeline.
3. Scroll down to click into the new Eval Function.
4. In Filter, as with our other Functions, replace the default true entry with the following expression:

   sourcetype==='employee'
5. In Remove Fields, type _raw
6. Click Save to save this fourth Function.

You should see the _raw field get "redacted" in the right pane, like this:

This removes the field, as we wanted. We're almost done...