Getting Data In

Why does monitoring for a log file keep stopping and getting error "BatchReader - Removed from queue..."?

tkwaller
Builder

Hello

I have a monitor on a log file that is continuously written to. It seems that the monitor keeps stopping and therefore I get no logs in the index. I can restart Splunk and logs come in for a bit, but then stop again. I can see this in the splunkd.log:

04-13-2016 14:19:37.331 +0000 INFO  BatchReader - Removed from queue file='/var/log/akcloudmon/node1_cloudmon1.log'.
04-13-2016 14:19:47.239 +0000 INFO  BatchReader - Removed from queue file='/var/log/akcloudmon/node1_cloudmon.log'.

Here is the input stanza:

[monitor:///var/log/akcloudmon/node*_cloudmon.log
sourcetype=akamai
source=akamai
index=akamai
disabled=0
crcSalt=akamaisalt2

How can I fix this?
Thanks!

0 Karma
1 Solution

tkwaller
Builder

Heres what we did to fix this issue. We upgraded the UF to an HF. We have decided to do this for all UFs that are LARGE data ingestion hosts. This has resolved any issues we were having. We also added:

min_batch_size_bytes = 10737418240

to limits.conf on these hosts as well.

View solution in original post

0 Karma

tkwaller
Builder

Heres what we did to fix this issue. We upgraded the UF to an HF. We have decided to do this for all UFs that are LARGE data ingestion hosts. This has resolved any issues we were having. We also added:

min_batch_size_bytes = 10737418240

to limits.conf on these hosts as well.

0 Karma

woodcock
Esteemed Legend

How many files are there in the directory with the file that you are monitoring?

0 Karma

jkat54
SplunkTrust
SplunkTrust

Just curious if your input stanza has a "trailing/ending" square bracket ']'? I"m sure it does, but figured I'd ask just in case.

As to your solution...

Try changing your crcSalt to <SOURCE> instead, the literal strings are intended for singleshots (i believe).

  [monitor:///var/log/akcloudmon/node*_cloudmon.log]
  sourcetype=akamai
  source=akamai
  index=akamai
  disabled=0
  crcSalt=<SOURCE>
0 Karma

tkwaller
Builder

Actually we did that, as a first step. No dice.
We also tried a few other things but to no avail.

0 Karma

jkat54
SplunkTrust
SplunkTrust

are they rolling log files with the same name? If so how often do they roll?

0 Karma
Get Updates on the Splunk Community!

Technical Workshop Series: Splunk Data Management and SPL2 | Register here!

Hey, Splunk Community! Ready to take your data management skills to the next level? Join us for a 3-part ...

Spotting Financial Fraud in the Haystack: A Guide to Behavioral Analytics with Splunk

In today's digital financial ecosystem, security teams face an unprecedented challenge. The sheer volume of ...

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability As businesses scale ...