Splunk Enterprise

Why does filtering logs before indexing using transforms.conf and props.conf creates ingestion latency problem?

Username_splunk
Engager
Filtering logs before indexing using transforms.conf and props.conf creates ingestion latency problem.
 
Tags (2)
0 Karma

Username_splunk
Engager

To reply to your question about latency:

  • Events from tracker.log have not been seen for the last 546 seconds, which is more than the red threshold (210 seconds). 
  • Events from tracker.log are delayed for 32126 seconds, which is more than the red threshold (180 seconds).

The regex is efficient, i tried it on regex101.

On indexing time, there is only one regex that i wrote for firewall incoming data to only accept blocked traffic logs. And because there are a lot of logs sent by the firewall, the indexer should filter all of those logs on indexing time to filter them and only take the blocked traffic to index it. 

 

Question: How can adding an indexer help me in this case, will the two indexers work on filtering logs in indexing time together?

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Maybe better option in your case is use something else (rsyslog, syslog-ng or cribl) before take those logs into splunk? I suppose that those are more powerful to do that kind of event dropping than splunk is? Also you could check if you can configure FW to send only wanted events not all.

r. Ismo

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Additional indexers help by sharing the workload.  If data is distributed across all indexers then they each will work on filtering logs.  They'll also share the search load to help make searches run faster.

---
If this reply helps you, Karma would be appreciated.
0 Karma

richgalloway
SplunkTrust
SplunkTrust

This is true.  The more work the indexer has to do before it indexes data the longer ingestion takes.

There are some things you can do about it.

  • Only do at index time what absolutely has to be done at index time.  Field extractions, for example, are best done at search time.
  • If you're using regular expressions in your filters then test them in regex101.com to make sure they're as efficient as you can make them.
  • Consider adding more indexers to your environment.

How much latency are we talking about?

---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...