Getting Data In

Why does monitoring for a log file keep stopping and getting error "BatchReader - Removed from queue..."?

tkwaller
Builder

Hello

I have a monitor on a log file that is continuously written to. It seems that the monitor keeps stopping and therefore I get no logs in the index. I can restart Splunk and logs come in for a bit, but then stop again. I can see this in the splunkd.log:

04-13-2016 14:19:37.331 +0000 INFO  BatchReader - Removed from queue file='/var/log/akcloudmon/node1_cloudmon1.log'.
04-13-2016 14:19:47.239 +0000 INFO  BatchReader - Removed from queue file='/var/log/akcloudmon/node1_cloudmon.log'.

Here is the input stanza:

[monitor:///var/log/akcloudmon/node*_cloudmon.log
sourcetype=akamai
source=akamai
index=akamai
disabled=0
crcSalt=akamaisalt2

How can I fix this?
Thanks!

0 Karma
1 Solution

tkwaller
Builder

Heres what we did to fix this issue. We upgraded the UF to an HF. We have decided to do this for all UFs that are LARGE data ingestion hosts. This has resolved any issues we were having. We also added:

min_batch_size_bytes = 10737418240

to limits.conf on these hosts as well.

View solution in original post

0 Karma

tkwaller
Builder

Heres what we did to fix this issue. We upgraded the UF to an HF. We have decided to do this for all UFs that are LARGE data ingestion hosts. This has resolved any issues we were having. We also added:

min_batch_size_bytes = 10737418240

to limits.conf on these hosts as well.

0 Karma

woodcock
Esteemed Legend

How many files are there in the directory with the file that you are monitoring?

0 Karma

jkat54
SplunkTrust
SplunkTrust

Just curious if your input stanza has a "trailing/ending" square bracket ']'? I"m sure it does, but figured I'd ask just in case.

As to your solution...

Try changing your crcSalt to <SOURCE> instead, the literal strings are intended for singleshots (i believe).

  [monitor:///var/log/akcloudmon/node*_cloudmon.log]
  sourcetype=akamai
  source=akamai
  index=akamai
  disabled=0
  crcSalt=<SOURCE>
0 Karma

tkwaller
Builder

Actually we did that, as a first step. No dice.
We also tried a few other things but to no avail.

0 Karma

jkat54
SplunkTrust
SplunkTrust

are they rolling log files with the same name? If so how often do they roll?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...