Discovering Data using HTTP Responses

In this module, we'll configure Cribl Stream to discover data using HTTP requests' JSON Response body. Common response formats include objects and arrays. Let's explore both.

HTTP Request – Object Response

In the previous examples, all the data we need is in a single API request URL. But sometimes (actually, more than "sometimes") we first need the information in one set of data (say, a specific group of users) before we can get the data we want (all the items in their shopping carts, for example).

For those situations, there's the HTTP Response Discovery option. At a high level, here's how it works

First, we provide an API URL that generates a list of records that has a related data point - in this case, the User ID.
Next, we give the API URL that has the data we want, using a field from the first set to limit the data returned

To spell it out, we're going to get a list of users who's last name is "Brown", and then get the shopping cart data for those users by linking the User ID.

note

You can see the user records by running the following command in your terminal / command prompt window:

curl https://dummyjson.com/users/search?q=Brown | jq

And here's the data you should see as a result:

{
  "users": [
    {
      "id": 3,
      "firstName": "Sophia",
      "lastName": "Brown",
      "maidenName": "",
      "age": 43,
*** MORE INFORMATION HERE *** 
   },
      {
         "id": 103,
         "firstName": "Emily",
         "lastName": "Brown",
         "maidenName": "Taylor",
         "age": 43,
*** MORE INFORMATION HERE ***
    }
  ],
  "total": 2,
  "skip": 0,
  "limit": 2
}

So let's set this up:

As with the previous modules, you create a new REST collector by selecting Manage > Data > Sources, then select Collectors > REST from the Data Sources page's tiles. Click Add Collector to open the REST > Add Collector modal.
In the Collector ID field, enter discover_http_object.
Expand the Discover accordion header, then from the Discover Type drop-down, select HTTP Request.
In the Discover URL field, enter the following URL:
```
'https://dummyjson.com/users/search?q=Brown'
```
note
NOTE: Those are regular single quotes we're using, and you need to include them!
In the Discovery data field enter users
In the Collect section, configure the Collect URL to reference the id value at the end of the URL path.
```
`https://dummyjson.com/carts/user/$\{id\}`
```
note
NOTE: Those are backticks, not single quotes. Make sure you have the right ones or this example won't work!
At the bottom left, click ► Save & Run. In the Run configuration modal, click Run again.

You should see two records with the cart items for Sophia Brown (userID 3) and Emily Brown (userID 103).

important

IMPORTANT NOTE: "How do I know when I need a Discovery data field and when I don't?" Take a look at the output in your terminal window again. Specifically the very top:

{
  "users": [
    {
      "id": 3,
      "firstName": "Sophia",
      "lastName": "Brown",

"users": [ tells us that this JSON response is an array. In some cases, there may be more than one top-level array like this. Therefore you need to indicate the name in Discovery data field. If the returned data had looked like this:

[
    {
      "id": 3,
      "firstName": "Sophia",
      "lastName": "Brown",

...then you would NOT have needed the Discovery data field.

Conclusion

You now know how to discover data from one REST API endpoint and use those results to collect data from a different REST API endpoint!

In the next module, we'll explore ways to focus our REST queries by using parameters.

HTTP Request – Object Response​

Conclusion​

HTTP Request – Object Response

Conclusion