Discovering Data

In this module, we'll work with the REST Collector's Discover section to dynamically collect events.

Cribl Stream runs collection jobs in five phases:

1. Authentication (optional)
2. Discovery (optional)
3. Collection
4. Event Breaking (optional)
5. Filtering (optional)

In the previous module, you performed only step 3 (Collection). However, there's a technicality here: All collection jobs have an implicit discovered object – if one is not defined – to seed the collection phase. Without this, Cribl Stream would not run the collection job.

Now, we'll add Discovery to REST Collectors with different configurations.

Item List

The Item List discovery mode is the simplest way to configure a Collector to run multiple collection tasks in a single job.

important

If necessary, navigate back to the REST Collector Source page. From the top nav of your Cribl Stream Sandbox, with Manage active, select Data > Sources, then select Collectors > REST from the Data Sources page's tiles. Click + Add New to open the REST > Add Collector modal, which provides the following options and fields.

1. In the Collector ID field, enter discovery_list.

2. Expand the Discover accordion header, then from the Discover Type drop-down, select Item List.

3. In the Discover items field, enter 1,2,3 and press your space bar. You will see the comma list convert into individual tags.

   Note: The Discover items entries generate 3 individual collection tasks. You can now use the id variable to reference the item's value anywhere in the Collect URL, parameters, or headers inputs.

4. Configure the Collect URL to reference the id value in the URL path.

   `http://rest-server/collect/object/${id}`

   Note: The backticks above allow you to reference the variable, by wrapping it in curly braces preceded by a dollar sign – i.e., ${id}

5. At the bottom left, click ► Save & Run. In the Run configuration modal, click Run again.

The Preview modal should display three events.

JSON Response

important

Close the Preview modal and open a new REST Collector configuration modal.

1. In the Collector ID field, enter discover_json_array.

2. Expand the Discover accordion header, then from the Discover Type drop-down, select JSON Response.

3. Copy and paste the following JSON into the Discover result box:

   [{"id":1},{"id":2},{"id":3}]

4. Leave the Discover data field empty. We'll explore when to configure this in the next section.

   Note: The Discover items entries generate 3 individual collection tasks. You can use the id variable to reference the item's value anywhere in the Collect URL, parameters, or headers inputs.

5. Configure the Collect URL to reference the id value in the URL path.

   `http://rest-server/collect/object/${id}`

   Note: The backticks above allow you to reference the variable with ${id}.

6. At the bottom left, click ► Save & Run. In the Run configuration modal, click Run again.

The Preview modal should display three events.

JSON Response using Attribute

important

Close the Preview modal and open a new REST Collector configuration modal.

1. In the Collector ID field, enter discover_json_array_attribute.

2. Expand the Discover accordion header, then from the Discover Type drop-down, select JSON Response.

3. Copy and paste the following JSON into the Discover result box:

   {"items":[{"id":1},{"id":2},{"id":3}]}

4. In the Discover data field, enter items.

   Note: Within the response, this is the name of the field that contains discovery results.

5. Now, in the Collect section, configure the Collect URL to reference the id value in the URL path. Here again, the backticks are essential:

   `http://rest-server/collect/object/${id}`

6. At the bottom left, click ► Save & Run. In the Run configuration modal, click Run again.

The Preview modal should display three events.

Conclusion

In the next module, we'll explore how to use responses from HTTP requests to dynamically discover what data you can collect from REST APIs!