The Externaldata Operator
Another saber in you toolbelt will undoubtedly, be the externaldata operator, which allows you to query any http(s) api endpoint directly without the need to set up a dataset or dataset provider (cue the oooo's and ahhhh's).
The externaldata operator also has a few parameters that allow you to alter aspects of the request to the API endpoint and/or manipulate the way the returned data is processed.
| Property | Type | Description |
|---|---|---|
| dataField | string | The name of the field (in the response JSON) to pull data from. Leave blank if the result is an array. |
| datatype | string | The data type to use to parse the data. If none is specified, tries to parse the data as a JSON array. |
| headers | string | A JSON object containing the headers to send with the request. |
| method | GET or POST | The HTTP method used when making the API request. |
If you find yourself querying the same API frequenlty it may be better to create a Generic HTTP API dataset provider. Additional guidance on how to do so can be found in the Cribl Search Data Sources sandbox.
Have I Been Pwned?
Let's get some practice using the externaldata by querying the Breaches v3 API from haveibeenpwned.com.
-
Clear the
query bar. -
Enter the following query:
externaldata ["https://haveibeenpwned.com/api/v3/breaches"] -
Click
SEARCH.
Upon running the search, you can see that the data returned by the API magically returns with properly applied event breaking, parsing, and is ready to be searched.
This is because by default, Cribl Search assumes (correctly in this case) that the results are in a JSON Array. If results are in a different format or there is a specific way that you would like the data to be processed, you could use the datatype property of the externaldata to choose another configured datatype to process the data.
Additional guidance on configuring datatypes can be found in the Cribl Search Data Sources sandbox.