Splunk Enterprise

Using regex to filter indexed data caused problems in splunk health ( Ingestion Latency + TailReader-0)

Username_splunk
Engager

Hello, 

I have that limit of license indexation per day. So i wanted to limit data to be indexed from a specific Equipment.

I received a great amount of logs from a source equipement using syslog (i can't change which types of logs to be sent to splunk). So, to limit the amount of data being indexed. i filtered data in the indexation phase using splunk. I added a regex in splunk so that splunk only indexes the wanted types of logs and ignore other received sylog logs from that specific equipment. I did this using TRANSFORMS-set in props.conf and using the regex expression in transforms.conf file. 

As a result, i had the following errors in splunk health that i couldn't fix:

 Ingestion Latency

  • Events from tracker.log have not been seen for the last 2940 seconds, which is more than the red threshold (210 seconds). This typically occurs when indexing or forwarding are falling behind or are blocked. 

TailReader-0   

  •    The monitor input cannot produce data because splunkd's processing queues are full. This will be caused by inadequate indexing or forwarding rate, or a sudden burst of incoming data.

Whenever i remove the regex expression the problem is solved => meaning that the regex is the only source of this problem/error.

 

Thank you in advance for help.

0 Karma
Get Updates on the Splunk Community!

Technical Workshop Series: Splunk Data Management and SPL2 | Register here!

Hey, Splunk Community! Ready to take your data management skills to the next level? Join us for a 3-part ...

Spotting Financial Fraud in the Haystack: A Guide to Behavioral Analytics with Splunk

In today's digital financial ecosystem, security teams face an unprecedented challenge. The sheer volume of ...

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability As businesses scale ...