Splunk Enterprise

Using regex to filter indexed data caused problems in splunk health ( Ingestion Latency + TailReader-0)

Username_splunk
Engager

Hello, 

I have that limit of license indexation per day. So i wanted to limit data to be indexed from a specific Equipment.

I received a great amount of logs from a source equipement using syslog (i can't change which types of logs to be sent to splunk). So, to limit the amount of data being indexed. i filtered data in the indexation phase using splunk. I added a regex in splunk so that splunk only indexes the wanted types of logs and ignore other received sylog logs from that specific equipment. I did this using TRANSFORMS-set in props.conf and using the regex expression in transforms.conf file. 

As a result, i had the following errors in splunk health that i couldn't fix:

 Ingestion Latency

  • Events from tracker.log have not been seen for the last 2940 seconds, which is more than the red threshold (210 seconds). This typically occurs when indexing or forwarding are falling behind or are blocked. 

TailReader-0   

  •    The monitor input cannot produce data because splunkd's processing queues are full. This will be caused by inadequate indexing or forwarding rate, or a sudden burst of incoming data.

Whenever i remove the regex expression the problem is solved => meaning that the regex is the only source of this problem/error.

 

Thank you in advance for help.

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

How much can you really learn in 3 minutes?

Observability can certainly be hard to understand – there's a lot of jargon and buzzwords and it seems to ...

Event Series: The Agentic SOC: Trust Before Autonomy

AI is fundamentally changing security operations, but true progress requires more than just automation—it ...

Free Professional Services for .conf26 Attendees

This year at .conf26, we are doing something a little different. We are bringing the best minds from ...