I have a script that creates a file, thoguh the command that is run, has a very long output, and it takes about 20 seconds to build the input. This script is ran every 60 seconds, and I use crcSALT = <SOURCE> to capture the input, due to this just being an output of a command with no timestamp.
I am getting discrepancies in data between a simply 'grep | wc -l' and splunk, and I'm pretty sure it's due to timing. I think the forwarder is ingesting the file before it is fully populated, so it only gets a portion of the data, but it still misses some things.
My question is, how do I get the forwarder to either wait the 10 seconds it takes to populate that file every minute, or to know that the who file hasn't loaded yet, and wait until the file itself has fully populated before ingestion?
so far in inputs.conf I have attempted:
followTail = 0
time_before_close = 0
Could you just increase the size of the buffer you are using to write the file? Here's a post about that if you are using Python. Other docs are available, and if you are using some other scripting language you'll have to look it up. But if you made your buffer larger than your typical file size, that would make the file write be one operation and should get around this problem.