Getting Data In

Splunk occasionally not capturing all logs

New Member

Hey all, I'm running into some odd behavior. I currently have splunkforwarder set up on a container and it should be pulling in logs from three log files. Here's a quick rundown of the issue:

  • "", "daily.two.log", and "daily.three.log" are generated when I run three different backups for another application
  • Backups run around 1am every morning and are usually pretty quick (a matter of minutes)
  • a Splunk search for those files usually returns the complete contents of all three logs for a given day
  • every few days or so, Splunk will not have all of the lines from the log files or even all the logs
  • A recent example would be that Splunk had the complete logs for "", nothing from "daily.two.log", and some of the lines from "daily.three.log"

Any ideas as to what could be causing this to happen sporadically?

Tags (2)
0 Karma


Perhaps you can find the source of the problem via Add Data / Upload in Splunk web interface: Upload a faulty file, select the according sourcetype and check if everything is alright.

One idea: Perhaps timestamp recognition is not working correctly. Again, use Add Data / Upload to verify that all timestamps are parsed correctly. If not, have a look at Configure timestamp recognition, specifically TIME_FORMAT and TIME_PREFIX in props.conf.

Another idea: Splunk will not index files which are completely identical so that Splunk does not index log rotated files. (I'm not sure though if this applies in your case, since you said that sometimes only partial files are indexed.)
You can prevent this behaviour in your inputs.conf with:

crcSalt = <SOURCE>
0 Karma
Get Updates on the Splunk Community!

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...