Exploring Multi-Tier Routes

TL;DR

The Stream pack you installed includes three pre-configured routes that automatically send events to the appropriate destination based on their business_impact tag. Let's explore how they work, test them with live data, and understand the routing strategy.

---

The Routing Strategy

Think of routing like coffee bean distribution at Cribl Coffee Co.:

• Premium beans → Specialty cafes (expensive, immediate)
• Quality beans → Standard stores (moderate cost, good availability)
• Bulk beans → Warehouse storage (cheap, long-term)

Our data follows the same logic:

• Tier 1 (Hot): Critical events → Top-tier SIEM (expensive, real-time)
• Tier 2 (Warm): Security/UX events → Lakehouse (moderate, queryable)
• Tier 3 (Cold): Everything → Lake (cheap, archived)

The pack has already created three routes that implement this strategy. Let's explore them.

Important: Route Order Matters

Routes are evaluated top-to-bottom. The first matching route wins (unless Final is disabled).

The pack's strategy:

1. Tier 1 route catches critical events → sends to SIEM
2. Tier 2 route catches security/UX events → sends to Lakehouse
3. Tier 3 route catches everything else → sends to Lake

By setting Final: No on the first two routes, events can flow through multiple tiers (critical events go to both SIEM and Lake for archival).

Step 1: Explore Tier 1 Route (Critical → SIEM)

This route catches the most urgent events and sends them to your top-tier SIEM for immediate action.

Explore Tier 1 Route

1. In Stream, navigate to Routing > Data Routes
2. Find the tier1 route (it came from the pack)
3. Click on it to expand and observe the configuration:
   • Filter: business_impact=="critical"
   • Pipeline: passthru
   • Output: Your webhook SIEM destination
   • Final: No (allows events to continue to next routes)
4. Don't change anything yet - just understand the configuration

Understanding Tier 1

Filter: business_impact=="critical"
This route only matches events tagged as critical (status 500-599).

Pipeline: passthru
No additional processing needed - the parserlogs pipeline already did all the enrichment. We just pass the event through unchanged.

Final: No
This is important! By not marking this as final, critical events will ALSO continue down to Tier 3 (cold storage). This gives you:

• Real-time alerting in your SIEM
• Long-term retention in cheap Lake storage

Why Not Final?

Setting Final: No means "send to this destination AND keep routing." This is perfect for critical events that need:

1. Immediate attention (SIEM)
2. Historical record (Lake)

If Final was Yes, critical events would only go to the SIEM and never get archived.

Step 2: Explore Tier 2 Route (Security/UX → Lakehouse)

This route catches events that are important but not critical - perfect for the indexed, queryable Lakehouse.

Explore Tier 2 Route

1. Still in Data Routes, find the tier2 route
2. Click on it to expand:
   • Filter: business_impact=="security" || business_impact == "user_experience"
   • Pipeline: passthru
   • Output: Your Lakehouse destination
   • Final: No (these events also go to Tier 3 for archival)
3. Observe how the filter uses OR logic to catch two business_impact categories

Understanding Tier 2

Filter: business_impact=="security" || business_impact == "user_experience"
The || means OR. This route matches events tagged as either:

• security (status 401, 403)
• user_experience (status 404)

Why group these together?
Both event types are:

• Important enough to justify indexed storage (faster queries)
• Often analyzed together during investigations
• Don't need the real-time speed of the top-tier SIEM

Final: No
Like Tier 1, these events also continue to Tier 3 for long-term archival.

Alternative: Separate Routes

If you wanted to send security and UX events to DIFFERENT destinations, you'd create two routes:

• Route A: business_impact=="security" → Security Lakehouse
• Route B: business_impact=="user_experience" → UX Analytics

The pack groups them to keep the demo simple, but you have flexibility to split them in production.

Step 3: Explore Tier 3 Route (Everything → Cold Archive)

This catch-all route ensures every event lands in long-term cold storage for compliance and historical analysis.

Explore Tier 3 Route

1. Still in Data Routes, find the tier3 route (should be last)
2. Click on it to expand:
   • Filter: business_impact == "user_experience" || business_impact=="critical" || business_impact=="security" || business_impact=="routine"
   • Pipeline: passthru
   • Output: Your Lake cold storage destination
   • Final: Yes (this is the end of the routing chain)
3. Notice this is the only route with Final: Yes

Understanding Tier 3

Filter: Lists all four business_impact values
You might wonder: "Why list all four? Won't everything match anyway?"

You're right - you could simplify this to just true. But being explicit:

• Documents what you expect to match
• Catches errors if you add a new business_impact value and forget to route it
• Aids debugging by making intentions clear

Final: Yes
This is the end of the line. Once events hit Tier 3, no more routing happens.

The Catch-All Pattern

Tier 3 is your safety net. Even if an event doesn't match Tier 1 or Tier 2, it will land here. This ensures you never accidentally drop data - everything gets archived somewhere.

Step 4: Observe the Routing in Action

Since you already enabled the datagen source in Module 2, data is flowing through these routes right now. Let's verify.

Check Live Routing Stats

1. Navigate to Monitoring > Data > Routes
2. You should see statistics for your three routes:
   • tier1 - Number of critical events routed
   • tier2 - Number of security/UX events routed
   • tier3 - Number of all events routed
3. Observe the event counts and throughput
4. tier3 should have the highest count (most events are routine)
5. tier1 should have the lowest count (critical events are rare)

Let's verify each tier is receiving data.

Verify Tier 1 (SIEM)

Check Critical Events

1. Open your webhook.site tab (from Module 2 setup) or Click live data on webhook destination
2. You should see POST requests arriving with event data
3. Look for events with "business_impact":"critical" and "status":500+
4. These are your critical events reaching the "SIEM"

If you don't see events yet:

• Wait 1-2 minutes (critical events are rare - only ~2% of traffic)
• Check that the tier1 route is enabled
• Verify the output destination is set correctly

Verify Tier 2 (Lakehouse)

Check Security/UX Events

1. Switch to Cribl Search
2. Run this query:

   dataset="datatieringlogs"
   | where business_impact IN ("security", "user_experience")
   | summarize count() by status, business_impact
   | limit 100
3. You should see events with status codes: 401, 403, 404
4. But NO 500+ errors (those went to Tier 1 SIEM instead)

Lakehouse vs Lake Dataset

You might need to query the Lake dataset directly if the Lakehouse hasn't fully indexed yet. Lakehouse indexing happens in the background and can take a few minutes for the first data to appear.

If you don't see results in Lakehouse, try the Lake dataset directly which should have data immediately.

Verify Tier 3 (Cold Lake)

Check All Events

1. Still in Cribl Search, run:

   dataset="default_logs"
   | summarize count() by business_impact
2. You should see most of the categories:
   • critical
   • security
   • user_experience
   • informational
3. informational should be the highest count (~90% of events)
4. This confirms everything is landing in cold storage

Step 5: Experiment with Route Filters (Optional)

Want to see what happens when you change the routing logic? Try this experiment.

Adjust Tier 1 Threshold

1. Navigate back to Routing > Data Routes
2. Click into the tier1 route to edit
3. Change the filter to also include 404 errors:

   business_impact=="critical" OR status == 404
4. Click Save
5. Click Commit & Deploy
6. Watch your webhook.site - you should now see 404 errors arriving at the SIEM
7. When done experimenting, change the filter back to:

   business_impact=="critical"
8. Save and deploy again

This shows you how flexible routing is. You can adjust filters anytime based on changing business needs:

• Holiday season? Route all cart errors to SIEM temporarily
• Under attack? Lower threshold to catch more security events
• Budget cuts? Raise threshold to reduce SIEM volume

Advanced: Add a New Tier (Optional)

Want to try adding a new tier between critical and security? Here's an advanced exercise.

Add "High Priority" Tier

1. Edit the parserlogs pipeline
2. Modify the Eval function to add a new category:

   status >= 500 ? 'critical' :
     status >= 400 && status < 500 ? 'high_priority' :
     (status == 401 || status == 403) ? 'security' :
     'routine'
3. Save the pipeline and commit/deploy
4. Create a new destination for high-priority events (or reuse an existing one)
5. Add a new route between tier1 and tier2:
   • Name: tier1_5
   • Filter: business_impact=="high_priority"
   • Output: Your choice of destination
   • Final: No
6. Save and deploy
7. Watch the new category flow through your system

This demonstrates how extensible the tiering system is. You're not locked into three tiers - you can create as many as needed for your business.

Understanding Multi-Path Routing

Here's what happens to each event type:

Critical Event (status: 500)

1. Enters Stream via sbx_weblogs source
2. Goes through parserlogs pipeline → tagged business_impact: critical
3. Hits Tier 1 route → matches → sent to SIEM
4. Tier 1 is Final: No → continues to next route
5. Hits Tier 2 route → doesn't match → skips
6. Hits Tier 3 route → matches → sent to Lake
7. Result: Event in SIEM (for immediate action) AND Lake (for historical record)

Security Event (status: 401)

1. Enters Stream via sbx_weblogs source
2. Goes through parserlogs pipeline → tagged business_impact: security
3. Hits Tier 1 route → doesn't match → skips
4. Hits Tier 2 route → matches → sent to Lakehouse
5. Tier 2 is Final: No → continues to next route
6. Hits Tier 3 route → matches → sent to Lake
7. Result: Event in Lakehouse (for investigation) AND Lake (for retention)

Routine Event (status: 200)

1. Enters Stream via sbx_weblogs source
2. Goes through parserlogs pipeline → tagged business_impact: routine
3. Hits Tier 1 route → doesn't match → skips
4. Hits Tier 2 route → doesn't match → skips
5. Hits Tier 3 route → matches → sent to Lake
6. Result: Event only in Lake (cheap cold storage)

This multi-path strategy ensures:

• Critical events get real-time attention (SIEM) AND long-term retention (Lake)
• Important events get queryable storage (Lakehouse) AND archival (Lake)
• Routine events don't waste expensive storage (Lake only)

Cost Impact Analysis

Let's do some napkin math on what this saves you.

Assumptions:

• 1,000 events/second = 86.4M events/day
• Event breakdown: 2% critical, 8% security/UX, 90% routine
• SIEM cost: $150/GB ingested
• Lakehouse cost: $50/GB stored
• Lake cost: $2/GB stored

Traditional Approach (send everything to SIEM):

• Daily volume: ~100GB (compressed)
• Monthly SIEM cost: $$$$$

Tiered Approach:

• Tier 1 (2%): 2GB/day
• Tier 2 (8%): 8GB/day
• Tier 3 (90%): 90GB/day

Savings: $$$$$

Your actual numbers will vary, but the principle holds: route based on value, not volume.

---

Next: Interactive Forensic Search →

---