AppScope is a new type of instrumentation technology. It works on any unmodified Linux binary, no matter the runtime. Our goal with AppScope is to make application monitoring as ubiquitous as infrastructure monitoring.

First, we need to start nginx using scope (we're going to use this a bit later).

important

Start NGINX for Later Use

In Terminal 1:
```
scope nginx
```

Let's dive in to how AppScope gets its data. AppScope is language-agnostic. It gathers APM-like data, but it's runtime-agnostic. AppScope achieves all this by loading a very lightweight library, libscope.so, into every running scope'd process.

It's pretty magical. Let's start by scoping perhaps one of the simplest applications, /bin/echo.

important

Scope /bin/echo

In Terminal 1:
```
 scope /bin/echo "something"
```

Our application ran like normal, but AppScope has collected a lot of information about resources that the application accessed and consumed. AppScope interposes itself between the application and shared libraries, like libc or openssl. This allows it to see HTTP traffic, Filesystem traffic, and Network traffic. AppScope gathers detailed performance metrics about CPU, Memory, Network traffic, and Filesystem IO. The scope CLI makes it simple to work with this data interactively. Also, libscope.so allows sending data to any downstream system of your choice, using open protocols.

Metrics

important

Scope Metrics

Let's see what kind of metrics AppScope collects:
```
scope metrics
```

AppScope collects metrics about all aspects of an application's performance. /bin/echo doesn't do much, but already we can see a lot of information.

We can see:

From proc.start that one process was started, and under the TAGS column
args as a dimension.
Some microseconds and what percentage of CPU was consumed, with proc.cpu and proc.cpu_perc.
How much memory was consumed, with proc.mem.
How many threads, file descriptors, and children are associated with proc.threads, proc.fd, and proc.child.

Lastly, we can see how many files were opened, with fs.open. A more sophisticated process will yield even more metrics for filesystem throughput, network connections and throughput, and even application-level metrics for HTTP.

The above scope metrics command can work with the data in several ways that we'll explore more later, including graphing. In addition to metrics, AppScope collects events like files opened, logs written to log files and the console, network connections, and HTTP-level request information. Let's take a look at what's collected for /bin/echo:

Events

important

Scope Events

Run:
```
scope events
```

/bin/echo doesn't do much, so we only see one console event. Each Scope event has an ID, which can be used to drill down into more detail. The first event has an ID of 0.

important

Scope Events Drilldown

Run:
```
scope events 0
```

Each AppScope event captures a bunch of information about the process itself, the PID, Command Line, Source, Sourcetype, Hostname, and Data, which contains what was collected from that process. In the case of our console event, we see something in Data. If you want to see just the raw data, you can output scope events as JSON, to send along to jq or other tools:

important

Scope Events JSON

Run:
```
scope events -j | jq .
```

Sessions

AppScope collects a lot of information. The scope CLI keeps each scope'd command as a separate session.

important

Scope History

To see the last 20 scope sessions:
```
scope hist
```

scope hist shows you all the sessions that have been run, with summary information about each session: when it was run, how long it lasted, how many events it generated, etc. You should see two sessions, one for nginx, which we'll get to later, and a second for echo, which we just ran.

Most scope parameters take an --id parameter to tell scope which session to reference for metrics and events. scope hist takes an --id parameter to show you the details of the session.

important

Scope History

Our last session, seen from scope hist, should be ID 2. Run:
```
scope hist --id 2
```

scope stores data about each session in files in the WorkDir. You can easily see all these files on disk:

important

Scope WorkDir

This little CLI trick makes it easy to access the WorkDir from any command line. Run:
```
ls $(scope hist -d)
```

Next Steps

As we can already begin to see, scope is a powerful way to look inside at what a process is doing. It works on out-of-the-box Linux processes, no matter the runtime, and even on built-in utilities like /bin/echo. Next, we'll be exploring more applications to scope, and what we can observe of their behavior.