Hey all, I'm running into some odd behavior. I currently have splunkforwarder set up on a container and it should be pulling in logs from three log files. Here's a quick rundown of the issue:
Any ideas as to what could be causing this to happen sporadically?
Perhaps you can find the source of the problem via Add Data / Upload in Splunk web interface: Upload a faulty file, select the according sourcetype and check if everything is alright.
One idea: Perhaps timestamp recognition is not working correctly. Again, use Add Data / Upload to verify that all timestamps are parsed correctly. If not, have a look at Configure timestamp recognition, specifically TIME_FORMAT and TIME_PREFIX in props.conf.
Another idea: Splunk will not index files which are completely identical so that Splunk does not index log rotated files. (I'm not sure though if this applies in your case, since you said that sometimes only partial files are indexed.)
You can prevent this behaviour in your inputs.conf with:
crcSalt = <SOURCE>