Why does filtering logs before indexing using transforms.conf and props.conf creates ingestion latency problem?

[Username_splunk]

Filtering logs before indexing using transforms.conf and props.conf creates ingestion latency problem.

[Username_splunk]

To reply to your question about latency:

• Events from tracker.log have not been seen for the last 546 seconds, which is more than the red threshold (210 seconds). 
• Events from tracker.log are delayed for 32126 seconds, which is more than the red threshold (180 seconds).

The regex is efficient, i tried it on regex101.

On indexing time, there is only one regex that i wrote for firewall incoming data to only accept blocked traffic logs. And because there are a lot of logs sent by the firewall, the indexer should filter all of those logs on indexing time to filter them and only take the blocked traffic to index it. 

Question: How can adding an indexer help me in this case, will the two indexers work on filtering logs in indexing time together?

[isoutamo]

Maybe better option in your case is use something else (rsyslog, syslog-ng or cribl) before take those logs into splunk? I suppose that those are more powerful to do that kind of event dropping than splunk is? Also you could check if you can configure FW to send only wanted events not all.

r. Ismo

[richgalloway]

Additional indexers help by sharing the workload.  If data is distributed across all indexers then they each will work on filtering logs.  They'll also share the search load to help make searches run faster.

[richgalloway]

This is true.  The more work the indexer has to do before it indexes data the longer ingestion takes.

There are some things you can do about it.

• Only do at index time what absolutely has to be done at index time.  Field extractions, for example, are best done at search time.
• If you're using regular expressions in your filters then test them in regex101.com to make sure they're as efficient as you can make them.
• Consider adding more indexers to your environment.

How much latency are we talking about?