Skip to main content

Copilot Pipeline Editor

TL;DR

One of Copilot's most powerful capabilities is helping you transform data from one format to another, or extract structured data from unstructured logs.

Now that we have our Palo Alto traffic logs flowing through our system, let's explore one of the most common and valuable uses of Cribl Copilot: transforming raw logs into more useful formats. In this example, we'll see how Copilot can help convert these logs into the Open Cybersecurity Schema Framework (OCSF) format.

Scenario: Palo Alto Traffic Logs to OCSF​

Imagine you're receiving raw Palo Alto Traffic logs and need to convert them to the standardized OCSF format for better interoperability with your security tools. Without Copilot Pipeline Editor, this would involve creating a complex Pipeline with multiple Functions and understanding both formats in detail.

Copilot Pipeline Editor to Transform Logs

  1. Navigate to the QuickConnect interface

  2. Go ahead and click the passthru connection we created in the previous section

  3. In the modal click Build with Copilot Editor

  4. In the Copilot Pipeline Editor interface:

  5. select Capture live events from source

  6. select Convert my input events to a specific target schema

  7. select the recommended schema OCSF@1.4-network-activity-4001

  8. A pipeline plan is generated which you can edit and control.

    • You can see the source and target schema

  9. Since everything looks good to you, click Confirm

  10. You can now see the pipeline that has been created and the output of the pipeline

  • You can see the functions that will be used to transform the data
  • You can see the preview of the data before and after transformation
  • We are re-affirming the human in the loop so you have control and flexibility.
Review and Implement
  1. Review the suggested Pipeline to understand the transformation
  2. Copilot will create the Pipeline with all necessary Functions configured
  3. Click Approve Pipeline to save and apply the pipeline
  4. Navigate to the newly created Pipeline to see the transformation ready to use

Copilot will analyze your request and generate a comprehensive transformation Pipeline. Let's take a look at what it provides:

  1. Explanation of the transformation: Copilot will explain what it's going to do and why
  2. Sample log example: It may show you what the logs look like before and after transformation
  3. Pipeline configuration: The specific Functions needed to accomplish the transformation
  4. Implementation options: Buttons to apply the changes directly or modify them

How It Works​

Behind the scenes, Copilot is:

  1. Understanding your intent from your natural language request
  2. Identifying the source format (Palo Alto Traffic logs) and target format (OCSF)
  3. Determining the required transformations to map fields correctly
  4. Generating the appropriate Pipeline Functions with proper configuration
  5. Presenting the solution in a way that you can review before implementing

This process would typically take an expert considerable time to implement manually, but Copilot completes it in seconds while maintaining your ability to review and approve the changes.

Optional Exploration

If you want to see Copilot Pipeline Editor handle other log sources, try going to routes and selecting weblogs datagen:

  • "Add pipeline to Quick Connect"
  • Select Save Connection
  • New Session
Running it back
  1. Capture live events from source
  2. Where is your data coming from? palo_traffic
  3. Click confirm
  4. Apply transformations to my input events
  5. here is the pipeline plan feel free to edit and copy
### Assumptions:
> *Validate these assumptions made by Copilot*
- The Palo Alto library parser will correctly parse the `_raw` field into structured JSON format.
- The GeoIP enrichment will be applied to the correct fields representing source and destination IPs.

### Transformations:
> *Specify the tranformation steps to be applied to the input events*
- Parse the `_raw` field using the Palo Alto library parser to convert the CSV data into JSON format.
- Apply GeoIP enrichment to the parsed source and destination IP fields to add location data, such as country, region, and city.
- Drop the original `_raw` field after parsing.

### Keep Fields:
> *List field(s) from the input / target schema that should be kept*
- None Specified

### Drop Fields:
> *List field(s) from the input / target schema that should be dropped*
- `*`
- `__id`
- `__cloneCount`
- `__ctrlFields`
- `__eventId`
- `__final`
- `__inputId`
- `__criblEventType`

After making these changes, remember to commit and deploy them as we learned earlier.

Commit & Deploy

Remember to commit and deploy your changes as explained in the setup section.

Now that we've seen how Copilot can transform our log data, let's explore another powerful capability in the next section: creating visualizations to gain insights from our data.