Splunk occasionally not capturing all logs

[rileykohl21]

Hey all, I'm running into some odd behavior. I currently have splunkforwarder set up on a container and it should be pulling in logs from three log files. Here's a quick rundown of the issue:

• "daily.one.log", "daily.two.log", and "daily.three.log" are generated when I run three different backups for another application
• Backups run around 1am every morning and are usually pretty quick (a matter of minutes)
• a Splunk search for those files usually returns the complete contents of all three logs for a given day
• every few days or so, Splunk will not have all of the lines from the log files or even all the logs
• A recent example would be that Splunk had the complete logs for "daily.one.log", nothing from "daily.two.log", and some of the lines from "daily.three.log"

Any ideas as to what could be causing this to happen sporadically?

[whrg]

Perhaps you can find the source of the problem via Add Data / Upload in Splunk web interface: Upload a faulty file, select the according sourcetype and check if everything is alright.

One idea: Perhaps timestamp recognition is not working correctly. Again, use Add Data / Upload to verify that all timestamps are parsed correctly. If not, have a look at Configure timestamp recognition, specifically TIME_FORMAT and TIME_PREFIX in props.conf.

Another idea: Splunk will not index files which are completely identical so that Splunk does not index log rotated files. (I'm not sure though if this applies in your case, since you said that sometimes only partial files are indexed.)
You can prevent this behaviour in your inputs.conf with:

crcSalt = <SOURCE>