Skip to main content

AppScope also sees all network traffic. When an application requests to open a socket, AppScope sees it. When an application writes bytes to that socket, AppScope sees it. AppScope isn't the only way to collect flow-level information, but it is the only way to collect flows easily with clear text payloads, as we'll see when we get into HTTPS traffic in the next section.

Let's start with a the simplest way of putting data on the wire – another Linux utility called netcat, or nc. For this exercise, we'll need both Terminal 1 and Terminal 2. We'll start by running netcat in Terminal 2. All subsequent commands will be in Terminal 1 – they depend on netcat running undisturbed in Terminal 2.

important

Network Metrics

  1. In Terminal 2, run: 
    nc -lkp 10001
  2. In Terminal 1, run: 
    scope sh -c 'echo "some bytes" | nc -w1 localhost 10001'
  3. In Terminal 1, run: 
    scope metrics -m net.tx -m net.duration --cols

Here, we've scoped another unmodified Linux utility, and we're using scope to display the metric information a little differently, as columns. We can see we transmitted 11 bytes and the connection was open between 500 and 1000 milliseconds. In addition to metrics, we can also capture event data:

important

Network Events

  1. In Terminal 1, run: 
    scope events -t net

Here, we can see there was a network connection, who it connected to, and how much data was transmitted over that connection. This is a good time to point out the difference between metrics and events.

AppScope thinks about metrics and events the same way the industry in general does. Metrics show you summary information about what the application is doing. Events show specifically that an application did something.

Both are important, but different. Investigations often start with a metric showing that some time series is skewing in the wrong direction, and then dive into events to understand what is happening. Metrics can be stored more cheaply in a time-series database, while events might end up in a logging tool or any data lake.

The scope CLI allows you to dive into more data than can be summarized in one line. The value in []'s at the left side of the event is the ID. You can copy that ID and pass it along to scope to dive into the details of that event.

important

Network Event Details

  1. Note the ID of the previous event. E.g., if the event's left side was echoed as [Ux], you're interested in the characters between the brackets: Ux.
  2. In Terminal 1, run the following command, replacing the <id> placeholder with the ID you noted just above: 
    scope events <id>

Flows

In addition to showing metadata about network flows, AppScope can also capture full payloads. Since capturing all payloads will consume a lot of disk, it is disabled by default. First, let's see how the scope CLI presents information, even if payload capture is off.

important

Network Flows

  1. In Terminal 1, run: 
    scope flows

This view uses the same data in events.json, but it presents it in a more consumable format. If we pass in the ID that it returned, we can see more information:

important

Network Flow Details

  1. Note the ID echoed in the previous event's ID column. This might be something like A6Jop3.
  2. In Terminal 1, run the following command, replacing the <id> placeholder with the ID you noted just above (it is case sensitive, so type it exactly as it appears) 
    scope flows <id>

Here, we see the full details of the flow. Note that there is no In File or Out File, because payloads weren't captured. Let's re-run the same nc command, but capture payloads this time:

important

Network Flow Payloads

  1. In Terminal 1, run: 
    scope run -p -- sh -c 'echo "some bytes" | nc -w0 localhost 10001'
  2. In Terminal 1, run: 
    scope flows $(scope flows | tail -n 1 | awk '{ print $1 }') --out

Now, with payload capture on, we can see the contents of the wire data from Step 2 above. From the first command, we should expect to see some bytes, and we do.

Obviously there are tons of use cases for seeing network flows and payloads easily from any application without modifying it. To name a few:

  • Finding potential malicious activity
  • Understanding risk in terms of dependencies
  • Service discovery

AppScope is not unique in these capabilities, as this is the equivalent of packet capture or wire data. eBPF-based instrumentation can see flows and payloads at a kernel level. But, AppScope can see clear text.

Next, we're going to explore HTTP events. HTTP events expand AppScope from infrastructure-level metrics and up to application level. HTTP and payload data are key to making application monitoring ubiquitous.