Splunk Enterprise

Using regex to filter indexed data caused problems in splunk health ( Ingestion Latency + TailReader-0)

Username_splunk
Engager

Hello, 

I have that limit of license indexation per day. So i wanted to limit data to be indexed from a specific Equipment.

I received a great amount of logs from a source equipement using syslog (i can't change which types of logs to be sent to splunk). So, to limit the amount of data being indexed. i filtered data in the indexation phase using splunk. I added a regex in splunk so that splunk only indexes the wanted types of logs and ignore other received sylog logs from that specific equipment. I did this using TRANSFORMS-set in props.conf and using the regex expression in transforms.conf file. 

As a result, i had the following errors in splunk health that i couldn't fix:

 Ingestion Latency

  • Events from tracker.log have not been seen for the last 2940 seconds, which is more than the red threshold (210 seconds). This typically occurs when indexing or forwarding are falling behind or are blocked. 

TailReader-0   

  •    The monitor input cannot produce data because splunkd's processing queues are full. This will be caused by inadequate indexing or forwarding rate, or a sudden burst of incoming data.

Whenever i remove the regex expression the problem is solved => meaning that the regex is the only source of this problem/error.

 

Thank you in advance for help.

0 Karma
Get Updates on the Splunk Community!

Enhance Your Splunk App Development: New Tools & Support

UCC FrameworkAdd-on Builder has been around for quite some time. It helps build Splunk apps faster, but it ...

Prove Your Splunk Prowess at .conf25—No Prereqs Required!

Your Next Big Security Credential: No Prerequisites Needed We know you’ve got the skills, and now, earning the ...

Splunk Observability Cloud's AI Assistant in Action Series: Observability as Code

This is the sixth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how to ...