Skip to main content

Creating Our Sources and Destinations

We need to create two Sources (dev -> datagen, prod -> syslog) and two Destinations (dev -> Elasticsearch dev index, prod -> Elasticsearch prod index). Let's get started...

Creating the Dev Source
  1. Click the Stream Dev top tab.
  2. In Cribl's own top nav, click the Cribl logo at left to unroll the product selector.
  3. Select Stream from this menu to expand Stream's top nav.
  4. Click Manage on this top nav.
  5. Click into the default Worker Group.
  6. Click the Data submenu, then click Sources.
  7. On the resulting Manage Sources page, locate the Datagen tile near the bottom (you can type its name into the filter box), and click the tile.
  8. On the resulting Manage Sources / Datagen page, click Add Source.
  9. In the Add Source dialog, for Input ID, enter dev_apache.
  10. From the Data Generator File drop-down, select the apache_common.log file.
  11. Click the dialog's Advanced Settings left tab, and then enter dev in the Environment field. (This key setting instructs Stream to enable this Apache datagen Source only in our dev environment.)
  12. Click Save, and look for a green confirmation message.
  13. Click Commit at Stream's upper right corner. NOTE: There might be a pending deploy from our original commit. If so, go ahead and deploy. This will allow the Commit button to become active again.
  14. In the resulting Git Changes – Group default dialog, enter a commit message (e.g., Enable datagen on dev).
  15. Click Commit and Deploy, and look for green confirmation messages.
Distributed Environment Operations

In a distributed environment, you must commit changes to the local Git repo, and deploy them, for your new configuration to take effect on any Worker Group.

Best Practice - Source Naming

Note the dev_apache ID we gave this Source. In a multi-instance environment – e.g., with a development and a production branch, like we're configuring here) – it's helpful to identify the environment as you configure an environment-specific component. This makes it easier to troubleshoot and understand data flow.

After a couple of minutes, your new configuration should be deployed to the development environment.

The resulting entry on the Manage Datagen Sources page might show up as greyed out and Inactive. Some Stream versions have a bug where the UI doesn't properly honor the Environment setting (CRIBL-7532). You can still validate that your Datagen is working:

Validating the Datagen
  1. Click the dev_apache row to reopen this Source's config dialog.
  2. Click the dialog's Live Data tab.

If the resulting capture window shows data flowing, then the Source is set up properly, and you can just close the dialog. If it doesn't show data flowing, go back and check your config against the setup steps above. If the config looks correct, then it likely hasn't fully deployed yet. Wait a few minutes, and try again.

Now let's configure the production Source. Note that although we'll bind this Source to our production environment (branch), we'll configure it here in the same development Stream instance.

Why? Because GitOps. When we set our production Stream instance to its Push workflow, we made it a read-only environment. It can now get new configs only via a push operation from its remote repo.

Configuring the Production Syslog Source
  1. If you're still on the Datagen Source dialog, click its close box.
  2. On the Manage Sources page's left sidebar, click Syslog.
  3. On the Manage Sources / Syslog page, click Add Source.
  4. In the Add Source dialog, enter prod_apache in the Input ID field, and enter the number 6514 in the TCP port field. (We'll ingest data only via TCP, not UDP.)
  5. Click the dialog's Advanced Settings left tab, and then enter prod in the Environment field.
  6. Click Save, and look for a green confirmation message.
  7. Click Commit at Stream's upper right corner.
  8. In the resulting Git Changes – Group default dialog, enter a commit message (e.g., Enable Apache syslog on prod).
  9. Click Commit and Deploy, and look for green confirmation messages.

The prod_apache source should show up in the list of Syslog Sources, but greyed out. This is its intended state here in the development environment.

We now have our two Sources configured properly. Let's do the same with our Destinations. For the sake of resources, we'll use a single Elasticsearch instance, but with two different indices: dev and prod.

Configuring our Destinations

Development​

  1. From Stream's submenu, click Data > Destinations.
  2. Locate and click the Elasticsearch tile. (You can type Elastic into the filter box, or scroll down to find the tile.)
  3. On the resulting Manage Destinations / Elasticsearch page, click Add Destination.
  4. In the Add Destination dialog, for Output ID, enter dev_logs.
  5. In the Bulk API URLs column's URL field, enter: 
    http://elasticsearch-master:9200/_bulk
  6. In the Index or Data Stream* field, enter dev.
  7. Click the Advanced Settings left tab.
  8. Scroll down to the Environment field and enter dev.
  9. Click Save, and look for a green confirmation message.

Production​

  1. Still on the Manage Destinations / Elasticsearch page, again click Add Destination, to create a different instance of this Destination.
  2. In the Add Destination dialog, for the Output ID, enter prod_logs.
  3. In the Bulk API URLs column's URL field, enter: 
    http://elasticsearch-master:9200/_bulk
  4. In the Index or Data Stream* field, enter prod.
  5. Click the Advanced Settings left tab.
  6. Scroll down to the Environment field and enter prod
  7. Click Save, and look for a green confirmation message.
  8. Click Commit at Stream's upper right corner, enter a commit message (e.g., Enable 2 Elasticsearch Destinations), and then click Commit and Deploy.

Our two Sources and two Destinations are configured. Now let's create our Route and Pipeline, and get GitOps going.