Event Breaking

There were a few examples in this module that returned information, but didn't display particularly well - or even had one record that was broken across multiple records.

In other cases you might have been thinking to yourself, "Do I really have to collect ALL of this sh...tuff? Isn't there a way to just get the data elements I want?

What you're asking for is called "Event breaking" in Cribl, and that's what this module is all about. We're going to configure a custom Event Breaker to automatically unroll the JSON arrays into individual events. This is useful when you're obtaining data from REST endpoints and forwarding individual events through Cribl Stream's Routes and Pipelines.

Run the following command in your sandbox terminal:

curl 'https://dummyjson.com/recipes' | jq

You'll see a result similar to the following:

{
  "recipes": [
    {
      "id": 1,
      "name": "Classic Margherita Pizza",
      "ingredients": [
        "Pizza dough",
        "Tomato sauce",
        "Fresh mozzarella cheese",
        "Fresh basil leaves",
        "Olive oil",
        "Salt and pepper to taste"
      ],
      "instructions": [
        "Preheat the oven to 475°F (245°C).",
        "Roll out the pizza dough and spread tomato sauce evenly.",
        "Top with slices of fresh mozzarella and fresh basil leaves.",
        "Drizzle with olive oil and season with salt and pepper.",
        "Bake in the preheated oven for 12-15 minutes or until the crust is golden brown.",
        "Slice and serve hot."
      ],
      "prepTimeMinutes": 20,
      "cookTimeMinutes": 15,
      "servings": 4,
      "difficulty": "Easy",
      "cuisine": "Italian",
      "caloriesPerServing": 300,
      "tags": [
        "Pizza",
        "Italian"
      ],
      "userId": 166,
      "image": "https://cdn.dummyjson.com/recipe-images/1.webp",
      "rating": 4.6,
      "reviewCount": 98,
      "mealType": [
        "Dinner"
      ]
    },
    {
      "id": 2,
      "name": "Vegetarian Stir-Fry",

Configure the Cribl Stream REST Collector

Now we're going to set this up in a Cribl REST collector, but - fair warning - the results aren't going to be what you might expect.

important

If necessary, navigate back to the REST Collector Source page. From the top nav of your Cribl Stream Sandbox, select Manage > Data > Sources, then select Collectors > REST from the Data Sources page's tiles. Click Add Collector to open the REST > Add Collector modal, which provides the following options and fields.

1. In the Collector ID field, enter array_event_breaker.
2. Copy the following URL to the Collect URL field.

   'https://dummyjson.com/recipes'
3. At the bottom left, click ► Save & Run. In the Run configuration modal, click Run again.

The Preview modal should display a single event. Why only one? When we displayed it in our terminal there were 30 recipes showing! The reason is that, even in our terminal, what we really got was a single array that contained 30 recipes. The secret is here:

{
  "recipes": [
    {
      "id": 1,

What we'll need to do is break up that array into the parts we want.

Configure the Custom Event Breaker

Now you'll configure a custom Event Breaker, to automatically unroll the array of items into individual events. There are two parts to this:

• First, creating the event breaker rule.
• Second, selecting and applying it within a REST collector.

important

Close the Preview modal (if open).

1. From the top nav of your Cribl Stream Sandbox tab, select Manage > Processing > Knowledge.2. From the left tabs, select Event Breaker Rules.

note

There are A LOT of event breaker rules already set up. At some point (maybe on a rainy day when the network is quiet, the servers aren't busy, and the users are on vacation) you should take a look through them to see what they have to offer. Even if none of them are EXACTLY what you need, there might be something that's close and you can clone it and make a few changes, instead of creating your own from scratch.

2. On the Event Breaker Rulesets page, click New Ruleset to create a new Event Breaker ruleset.
3. In the ID field, enter REST API Array Breaker.
4. Click + Add Rule.
5. In the Rule Name field, enter Array.
6. In the Event Breaker Type drop-down, select JSON Array.
7. In the Array field field, enter recipes
8. Try out this Event Breaker by copying and pasting the JSON array below into the input section. Click anywhere in the input pane, outside the box labeled Paste your events here or upload a sample file.

Copy this:

{"recipes":[{"id":1,"name":"Classic Margherita Pizza","ingredients":["Pizza dough","Tomato sauce","Fresh mozzarella cheese","Fresh basil leaves","Olive oil","Salt and pepper to taste"],"instructions":["Preheat the oven to 475°F (245°C).","Roll out the pizza dough and spread tomato sauce evenly.","Top with slices of fresh mozzarella and fresh basil leaves.","Drizzle with olive oil and season with salt and pepper.","Bake in the preheated oven for 12-15 minutes or until the crust is golden brown.","Slice and serve hot."],"prepTimeMinutes":20,"cookTimeMinutes":15,"servings":4,"difficulty":"Easy","cuisine":"Italian","caloriesPerServing":300,"tags":["Pizza","Italian"],"userId":166,"image":"https://cdn.dummyjson.com/recipe-images/1.webp","rating":4.6,"reviewCount":98,"mealType":["Dinner"]},{"id":2,"name":"Vegetarian Stir-Fry","ingredients":["Tofu, cubed","Broccoli florets","Carrots, sliced","Bell peppers, sliced","Soy sauce","Ginger, minced","Garlic, minced","Sesame oil","Cooked rice for serving"],"instructions":["In a wok, heat sesame oil over medium-high heat.","Add minced ginger and garlic, sauté until fragrant.","Add cubed tofu and stir-fry until golden brown.","Add broccoli, carrots, and bell peppers. Cook until vegetables are tender-crisp.","Pour soy sauce over the stir-fry and toss to combine.","Serve over cooked rice."],"prepTimeMinutes":15,"cookTimeMinutes":20,"servings":3,"difficulty":"Medium","cuisine":"Asian","caloriesPerServing":250,"tags":["Vegetarian","Stir-fry","Asian"],"userId":143,"image":"https://cdn.dummyjson.com/recipe-images/2.webp","rating":4.7,"reviewCount":26,"mealType":["Lunch"]},{"id":3,"name":"Chocolate Chip Cookies","ingredients":["All-purpose flour","Butter, softened","Brown sugar","White sugar","Eggs","Vanilla extract","Baking soda","Salt","Chocolate chips"],"instructions":["Preheat the oven to 350°F (175°C).","In a bowl, cream together softened butter, brown sugar, and white sugar.","Beat in eggs one at a time, then stir in vanilla extract.","Combine flour, baking soda, and salt. Gradually add to the wet ingredients.","Fold in chocolate chips.","Drop rounded tablespoons of dough onto ungreased baking sheets.","Bake for 10-12 minutes or until edges are golden brown.","Allow cookies to cool on the baking sheet for a few minutes before transferring to a wire rack."],"prepTimeMinutes":15,"cookTimeMinutes":10,"servings":24,"difficulty":"Easy","cuisine":"American","caloriesPerServing":150,"tags":["Cookies","Dessert","Baking"],"userId":34,"image":"https://cdn.dummyjson.com/recipe-images/3.webp","rating":4.9,"reviewCount":13,"mealType":["Snack","Dessert"]}],"total":50,"skip":0,"limit":3}

..and you should see this:

Notice how the red, yellow, and blue color-coding illustrates the way each discrete element is broken up (or deleted).

9. Click the Out tab at the top to view the array displaying 3 events with individual objects.

Extracting only some data from each record:

What if you don't want all the data in each record? You can extract a specific field from each record, and the event breaker will treat them as separate records.

1. Go back to the "In" side of the display
2. In the Array field add this parameter: recipes.ingredients.

You will see the color-coding immediately change so only the individual ingredients in each recipe are highlighted. Click the "Out" and you'll see 24 discrete records, one for each ingredient.

Removing some data from each record

On the flip side of just extracting certain information is removing only certain parts of the record.

1. First, click the JSON extract fields slide. This tells Cribl to extract each JSON field into a separate data element in the record.
2. Next, add the following items to Fields to remove: image,caloriesPerServing,*TimeMinutes

If you click the Out tab, you'll notice a few things:

• Each record contains multipl data elements from the API data
• The fields we identified are no longer present (you can check by clicking the "x" next to them in Fields to remove to see them pop back up)

Use your new breaker rule in the REST collector

1. At the bottom right, click OK to exit.
2. In the warning dialog, click Yes to ignore the warning that recommends using a Filter Condition other than the default true. (We are applying the Event Breaker only to the REST Collector, so here, the default is fine.)
3. At the bottom, right, click Save.

Apply the Event Breaker to Data Collection

Now let's apply the Event Breaker to the array_event_breaker Collector configuration.

important

WARNING WARNING - YOU NEED TO READ THIS If you have done any other Cribl training, you know how often we tell you to click the friendly Commit & Deploy button in the upper right corner of darn-near every single screen. And if you have, you might have noticed how consipicously absent that instruction has been so far. The fact is, the work we've been doing has been remarkably forgiving in that respect.

BUT NOT ANY MORE.

You need to stop what you're doing and click that friendly Commit & Deploy button now. If you don't, none of the instructions below will work.

You have been warned.

important

Navigate back to the REST Collector Source page. (From the top nav of your Cribl Stream Sandbox, select Manage > Data > Sources, then from the Data Sources page's tiles or left nav, select Collectors > REST.)

1. On the Manage REST Collectors page, click the array_event_breaker Collector row to open its configuration.
2. From the config modal's left Result Settings section, select the Event Breakers tab.
3. Click + Add ruleset and, from the drop-down, select REST API Array Breaker. You may need to scroll to the bottom of the list.
4. At the bottom left, click ► Save & Run. In the Run configuration modal, click Run again.

The Preview modal should display the expected 30 recipes.

Conclusion

You've now configured a simple Event Breaker to automatically unroll JSON Array responses from REST API endpoints!

In the next module, we'll explore how to collect data from a REST API endpoint on a schedule.