Processing
Now that you've gotten everything "scoped out" (see what I did there?), we'll move on to the next phase of using the Force searching which we like to call processing. In this phase, we can optionally transform, format, evaluate, and filter the scope using additional operators in conjunction with context, cribl, statistical, and scalar functions. Operators (and their associated functions) are separated by the | (pipe) symbol.
In the processing phase you can use as many or as few operators as needed to achieve your desired outcome. The same operators can even be used more than once!
Just keep in mind that the order of the operators will affect both the processing speed and the query results. There are many ways to achieve the same outcome. There are no wrong answers (unless you don't get your desired result, those answers are most definitely wrong).
Processing Operators
Operators are like Midi-chlorians, the more you've got the stronger you are with The Force. If you truly wish to be a data Jedi then you'd best get familiar with them. The full list of operators is pretty extensive, so we'll focus our training on a few that every data Jedi should know. Operators that we are going to cover further in your training are denoted with a ✔️.
| Operator | Description | Incl. |
|---|---|---|
eventstats | Enriches events with aggregated data. | ✔️ |
export | The export operator does one of the following:Cribl Lake dataset. (Mind that you need to create the dataset first).lookup table from search results. | ✔️ |
extend | Appends fields created by expressions. | ✔️ |
extract | Extracts data. | ✔️ |
ip-lookup | Enriches events with IP address data. | |
join | Merges datasets. | ✔️ |
limit | Limits the number of events (same as take). | |
lookup | Enriches events with lookup files. | ✔️ |
print | Outputs expression results. Used in the scope of a query, before a pipe |. | |
project | Configures fields to return. | ✔️ |
project-away | Configures fields to exclude from results. | ✔️ |
project-rename | Renames fields. | ✔️ |
render | The render operator enforces a specific visualization of the search results. It’s useful when you want to override the default rendering of a query.The render operator supports the following modes: Events tab.Chart tab. | |
send | Sends search results to Cribl Stream. | |
sort | Arranges events (same as order). | |
summarize | Aggregates your data. | ✔️ |
timestats | Aggregates by time periods or bins. Supported in the scope of a query, before a pipe |. | ✔️ |
top | Returns the first N events sorted by the specified fields. | ✔️ |
top-hitters | Counts the most frequent values. | ✔️ |
where | Filters specific events. |
Functions, Functions Everywhere!
Functions are used in conjunction with operators to process data. If Operators are instructions for what to do to the data, think of Functions as instructions for how to do it.
Take a look at some of the different functions that Cribl Search has to offer. Many of them will be used in some of our examples throughtought the rest of the sandbox.
Cribl Functions
Cribl Functions are used in conjunction with the summarize, eventstats, and timestats operators to aggregate your data.
Cribl Search supports the following additional functions:
- findearliest
- findearliestif
- findfirst
- findfirstif
- findlast
- findlastif
- findlatest
- findlatestif
- list
- median
- medianif
- persecond
- persecondif
- rate
- rateif
- sumsq
- sumsqif
- values
Statistical Functions
Statistical Functions are used in conjunction with the summarize, eventstats, and timestats operators to aggregate your data.
Cribl Search supports the following statistical functions:
- avg
- avgif
- count
- countif
- dcount
- dcountif
- max
- maxif
- min
- minif
- percentile
- stdev
- stdevif
- stdevp
- sum
- sumif
- variance
- varianceif
- variancep
Context Functions
Context Functions return contextual information about your search.
Cribl Search supports the following context functions:
- createdTime()
- earliestTime()
- jobID()
- latestTime()
- query()
- user()
Scalar Functions
Scalar Functions
Cribl Search supports the following scalar functions grouped by type.
Binary Functions
- binary_and
- binary_not
- binary_or
- binary_shift_left
- binary_shift_right
- binary_xor
- from_binary_string
- to_binary_string
Conditional Functions
- case
- coalesce
- iif
- max_of
- min_of
Conversion Functions
- bin
- bin_auto
- floor
- tobool
- todouble
- toint
- tolong
- toreal
- tostring
DateTime Functions
- ago
- datetime_add
- datetime_diff
- datetime_part
- dayofmonth
- dayofweek
- dayofyear
- endofday
- endofmonth
- endofweek
- endofyear
- format_datetime
- format_timespan
- getmonth
- getyear
- hourofday
- make_datetime
- make_timespan
- monthofyear
- now
- startofday
- startofmonth
- startofweek
- startofyear
- strftime
- strptime
- todatetime
- totimespan
- unixtime_microseconds_todatetime
- unixtime_milliseconds_todatetime
- unixtime-nanoseconds_todatetime
- unixtime_seconds_todatetime
- week_of_year
Hash Functions
- hash
- hash_combine
- hash_many
- hash_md5
- hash_sha1
- hash_sha256
- hash_xxhash64
INET Functions
- ipv4_compare
- ipv4_is_in_range
- ipv4_is_in_any_range
- ipv4_is_match
- ipv4_is_private
- ipv4_netmask_suffix
- ipv6_compare
- ipv6_is_match
- format_ipv4
- format_ipv4_mask
Mathematical Functions
- abs
- acos
- asin
- atan
- atan2
- beta_cdf
- beta_inv
- beta_pdf
- cos
- cot
- degrees
- exp
- exp2
- exp10
- gamma
- isfinite
- isinf
- isnan
- log
- log2
- log10
- loggamma
- not
- pi
- pow
- radians
- rand
- range
- round
- sign
- sin
- sqrt
- tan
String Functions
- base64_decode_toarray
- base64_decode_tostring
- base64_encode_fromarray
- base64_encode_tostring
- countof
- extract
- extract_all
- extract_json
- has_any_index
- indexof
- isempty
- isnotempty
- isnotnull
- isnull
- match_regex
- parse_csv
- parse_ipv4
- parse_ipv4_mask
- parse_ipv6
- parse_ipv6_mask
- parse_json
- parse_url
- parse_urlquery
- parse_version
- replace_regex
- reverse
- split
- strcat
- strcat_delim
- strcmp
- strlen
- strrep
- substring
- tolower
- toupper
- translate
- trim
- trim_end
- trim_start
- url_decode
- url_encode