Skip to main content

Capturing Live Payloads & Routing Data

Lastly, we're going to look at pairing AppScope with Cribl Stream. Stream is an observability pipeline that makes connecting any source to any destination simple. When we configure the scope CLI to send to Stream, we can immediately route metric data to Datadog, New Relic, Grafana Cloud, and many more. Event and log data can be sent to Elasticsearch, Datadog, New Relic, and others. Stream can also route raw full-fidelity data to S3 to create a data lake that has all your historical data.

In addition to routing your data, Stream creates an observable observability pipeline (yes, this is a thing), where you can easily introspect and monitor the data flowing through it. When running observability tooling in production, understanding how that data is flowing, to where, and why, is incredibly important.

Let's get right to sending data from AppScope to Stream. First, we need to turn on the AppScope input. Stream ships with it configured but disabled.

important

Enable Stream Input for AppScope

  1. Click the Cribl tab at the top.
    • Stream should automatically log you in, but if you're idle for too long, or get disconnected, it's possible you'll be prompted for a login. The Username is admin and the Password is cribldemo.
  2. In Stream's top navigation bar, with Manage active, select Data and then click Sources.
  3. Type AppScope into the search box to locate its tile.
  4. Click the AppScope tile to open the Manage AppScope Sources page.
  5. In the Enabled column, toggle the slider from No to Yes. This turns on the input.
  6. Click Yes in the confirmation dialog.

That's it, now the AppScope input is enabled! Let's send some data to it.

important

Send Data to Stream

  1. In Terminal 1, run: 
    scope run -c tcp://appscope:10090 \
        -- curl -so /dev/null https://wttr.in/94105
  2. On the Cribl tab, click Live at the right of the in_appscope source.
  3. In the resulting Manage > Data > Sources > AppScope > in_appscope modal, click the Charts tab.

You should see that some events have been sent through Stream. Let's run a live capture, to see what kinds of events are flowing through. This time, we're going to send in events with full payloads.

important

Send Data to Stream

  1. On the Cribl tab, within the Manage > Data > Sources > AppScope > in_appscope modal, click the Live Data tab.
  2. Click Stop.
  3. Replace the Filter Expression field's contents with: source=='http.resp'
  4. Click Capture.
  5. Set the Capture Time to 60 seconds.
  6. Click Start.
  7. In the Terminal 1 tab, run: 
    scope run -p -c tcp://appscope:10090 \
        -- curl -so /dev/null --http2 https://ifconfig.me
  8. Click back to the Cribl tab.

You should see that events have come in. In the above command, note the addition of -p to scope run, which tells scope to send payload data to Stream. Additionally, note the --http2 parameter to curl. With the current versions of AppScope and Stream, payload extraction is working only on HTTP2, but that should change in forthcoming releases.

Filtering to our http.resp event, we can see the data field has the actual payload from the transaction – captured remotely, from an interactive GUI! With Stream and AppScope, a user can introspect application payloads live and create fundamentally new instrumentation by parsing payloads. Imagine parsing headers and payloads to give response time by session and user, or transactions by geo, and much more.

Routing​

Stream makes it simple to route data to third-party destinations. To see just how easy it is, let's wrap up this course by sending some data from Stream out to netcat running in the terminal.

important

Create a TCP JSON Output

  1. In Terminal 2, if netcat is not still running from its previous launch, restart it using the same command you used previously: 
    nc -lkp 10001
  2. On the Cribl tab, close the in_appscope input's modal by clicking X at its top right.
  3. In Stream's top navigation bar, click Manage > Data > Destinations.
  4. In the search box, type TCP and then select the TCP JSON tile.
  5. Click Add Destination at the upper right to add a new TCP JSON output definition.
  6. In Output ID, put tcpjson.
  7. In Address, put appscope-terminal.
  8. In Port, put 10001.
  9. Select None from the Compression drop-down.
  10. Click Save.
  11. On the resulting Manage > Data > Destinations page, click Default at the top of the left sidebar.
  12. On the resulting Manage Destinations / Default page, click the default row.
  13. In the Default Output ID drop-down, select tcpjson.
  14. Click Save.
  15. In Terminal 1, run: 
    scope run -p -c tcp://appscope:10090 \
        -- curl -so /dev/null --http2 https://ifconfig.me

Now, in Terminal 2, we should see a ton of newline-delimited JSON documents. This is our data that's flowed from AppScope, through Stream, to netcat. On the Cribl tab, if you select Monitoring from the top nav, you can browse around our metrics and see that, indeed, the data is moving. If you look at the TCP JSON output, you can inspect live metrics and data.

Stream makes it this easy to route data to any destination we support. Additionally, you can easily filter, downsample, aggregate, enrich, suppress, and reshape data, using our data–centric user experience. If you'd like to learn more about Stream, check out the Cribl Stream Overview sandbox.