topic Re: File not being read by Splunk in a directory while others are in Getting Data In

File not being read by Splunk in a directory while others are

SirHill17 — Wed, 04 Oct 2017 14:01:52 GMT

Hi,
I have a directory which is defined in inputs.conf on a host (which has UF running), directory is:

/var/middleware/inventory/var

As per the logs (splunkd.log), the directory is now monitored:

10-04-2017 11:50:50.105 +0200 INFO TailingProcessor - Adding watch on path: /var/middleware/inventory/var.

In this directory there are nine different files. But only eight of them are read. They all have the same permissions and the content format is also the same.

Does anyone know why the last file is not being read by Splunk? There is no log about it.

Thanks for your help.

Re: File not being read by Splunk in a directory while others are

mattymo — Wed, 04 Oct 2017 14:06:35 GMT

Try running:

./splunk list inputstatus on the UF and looking for the file in question. It should show you why the tailReader may not have actioned it. If you have many files, it can be easier to output the command to a file.

Or try searching:

index=_internal source=*splunkd.log tailreader ERROR

might turn up something like:

10-03-2017 21:27:33.978 -0400 ERROR TailReader - File will not be read, is too small to match seekptr checksum (file=/home/splunker/splunk/var/log/splunk/splunk_app_stream.log.8). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.

Re: File not being read by Splunk in a directory while others are

SirHill17 — Wed, 04 Oct 2017 14:18:47 GMT

Are you sure this command can be run on the UF ? I got the below error message when running it (but I am able to do it on the Splunk Servers):

./splunk list inputstatus Command
error: The subcommand 'inputstatus' is
not valid for command 'list'. Data
forwarding configuration management
tools. Commands:
enable local-index [-parameter ] ...
disable local-index [-parameter ] ...
display local-index
add [forward-server|search-server] server
remove [forward-server|search-server] server
list [forward-server|search-server]

Objects:
forward-server a Splunk forwarder to forward data to be
indexed
search-server a Splunk server to forward searches
local-index a local search index on the Splunk server

Searching for index=_internal source=*splunkd.log tailreader ERROR there is no log for the host where I am trying to get the file to be read.

Thanks

Re: File not being read by Splunk in a directory while others are

mattymo — Wed, 04 Oct 2017 15:12:19 GMT

What version is the UF? If it is pre 6.3-ish then you may not have the option to run it like that. You would need to try this:

https://www.splunk.com/blog/2011/01/02/did-i-miss-christmas-2.html

Also, try grep ERROR splunkd.log on the on the UF located at $SPLUNK_HOME/var/log/splunk/splunkd.log

Do you see any logs from this UF if you run index=_internal source=*splunkd.log ?

Re: File not being read by Splunk in a directory while others are

SirHill17 — Wed, 04 Oct 2017 15:17:01 GMT

Got it, UF version is 6.2.7. The servers are on 6.4.3.
It explains, if I could have it upgraded soon I will have a try but anyway I also already tried with crcSalt = .

Re: File not being read by Splunk in a directory while others are

mattymo — Wed, 04 Oct 2017 15:27:27 GMT

ok so try the rest call on the UF. Just need to make sure the UF is serving 8089, you can get the same output as the ./splunk list inputstatus command. Also check the splunkd.log locally for the TailReader logs.

Re: File not being read by Splunk in a directory while others are

SirHill17 — Wed, 11 Oct 2017 08:06:55 GMT

Ok so I upgraded the forwarder to version 6.4.3 and run the command ./splunk list inputstatus

The output contains the file which is not read and there is still no error in the logs.

I did test to renamed the file to tomcat_jvm2.out and Splunk is picking it up...

Any other suggestion?

Thanks!

Re: File not being read by Splunk in a directory while others are

harsmarvania57 — Wed, 11 Oct 2017 08:12:03 GMT

Try to run this command, it will display which files has been read by splunk and which one is not and also gives reason why it didn't read.

/opt/splunkforwarder/bin/splunk _internal call /services/admin/inputstatus/TailingProcessor:FileStatus

I hope this helps.

Thanks,
Harshil

Re: File not being read by Splunk in a directory while others are

SirHill17 — Wed, 11 Oct 2017 08:33:23 GMT

It helps thanks!
I will let the forwarder running for few days and monitor if the file will be read or not and use the command to get info.
I will come back with comments.

Re: File not being read by Splunk in a directory while others are

mattymo — Wed, 11 Oct 2017 12:03:34 GMT

This is the exact same command I provided, fyi. Same output.

Re: File not being read by Splunk in a directory while others are

mattymo — Wed, 11 Oct 2017 12:19:59 GMT

Please post that status of the file you are looking for. We are not just looking that the ouptut contains the file...we want the status. If you change the name and it gets picked up, it must be failing CRC Check, or have some other issue with rotation, etc:

/var/log/secure
    file position = 677
    file size = 677
    parent = /var/log
    percent = 100.00
    type = finished reading

Re: File not being read by Splunk in a directory while others are

SirHill17 — Wed, 11 Oct 2017 12:24:39 GMT

That's the output:

/var/middleware/inventory/var/tomcat_jvm.out
                file position = 764
                file size = 764
                parent = /var/middleware/inventory/var/
                percent = 100.00
                type = open file

Re: File not being read by Splunk in a directory while others are

mattymo — Wed, 11 Oct 2017 12:31:01 GMT

Thats better. The file is being read.

Perhaps your timestamps are messed up? Have you tried:

index=* host=<yourForwarder> source=/var/middleware/inventory/var/tomcat_jvm.out over all time??

If you do:

tail -10 tomcat_jvm.out

What do the last few events look like?

Re: File not being read by Splunk in a directory while others are

hardikJsheth — Wed, 11 Oct 2017 12:40:00 GMT

This output suggest that Splunk has already read the file so it won't re-read unless you clean fish bucket.

The issue has with the rate at which logs are written and then rotated. Splunk UF is unable to match this load.

In my environment we faced similar issue with reading firewall logs which were generated/rotated at very fast rate (5GB/30minutes). We were able to solve our problem after switching to Heavy Forwarder.

Re: File not being read by Splunk in a directory while others are

mattymo — Wed, 11 Oct 2017 12:59:40 GMT

Hey hardikJsheth, can you please post your own answer and thread? I think you are getting way ahead of yourself, but feel free to work your own answer post above, rather than hijacking this one.

switching to a hf is a bad and downvote worthy idea and could have been avoided completely by simply editing limits.conf of the UF (defaults to maxKBps=256) or using parallel pipes, etc. a uf can easily keep up if tuned properly. not to mention the HF impacts the host and network more

I havent gotten the feeling they are trying to re-read anything...so fishbucket might help if OP is trying to reread but as far as i can tell they cant even find the original events. Furthermore the file is open, which means tailreader is watching for new events! lets find those before we go too far.

Re: File not being read by Splunk in a directory while others are

SirHill17 — Thu, 12 Oct 2017 12:30:30 GMT

/var/middleware/inventory/var/tomcat.out
                file position = 74
                file size = 74
                parent = /var/middleware/inventory/var/
                percent = 100.00
                type = finished reading

/var/middleware/inventory/var/tomcat_jvm.out
                file position = 764
                file size = 764
                parent = /var/middleware/inventory/var/
                percent = 100.00
                type = open file

What's the difference between "open file" and "finished reading" ? Really strange behaviour today, only the file that was not read before has been read today.

Re: File not being read by Splunk in a directory while others are

mattymo — Thu, 12 Oct 2017 13:21:01 GMT

Open file means it being tailed, finished reading probably means we hit an end of file.

Ok, so since we have ruled out Splunk not being able to read the files, how many files are you monitoring on this server? Just this directory? just these 9? or many files???

In my dealings with middleware teams, especially when I see jvms involved, the UF may require tuning to ensure we can move data fast enough to remain "realtime".

Take a look at $SPLUNK_HOME/var/log/splunk/metrics.log and grep for "blocked=true" are you seeing any blocking?

Also check to see if you are perhaps hitting large files that are taking up all the bandwidth in splunk.log. (search for "large file" in splunkd.log and you should see the batch processor being invoked)

THEN we can start to talk about tuning, like @hardikJsheth alluded to

Re: File not being read by Splunk in a directory while others are

SirHill17 — Wed, 25 Oct 2017 07:57:58 GMT

Since the UF version was upgraded it resolved the problem, not sure what was the issue but thanks for the useful command you provided which helps for other troubleshooting.