Troubleshooting why only part of a log file was imported

[jwinderDDS]

I have an application that creates a separate log file every time a major process runs. I discovered a single log file where the top several lines were not imported. A sample of the log files look like:

06/22/2015 20:30:06   > Header line 1
06/22/2015 20:30:06   > Header line 2
06/22/2015 20:30:06   > Header line 2
06/22/2015 20:30:06   > starting [SOMEPROC1]....
06/22/2015 20:30:06   > proc1 step 1
06/22/2015 20:30:07   > proc1 step 2
...
06/22/2015 20:30:07   > starting [SOMEPROC2]...
06/22/2015 20:30:07   > proc2 step 1
06/22/2015 20:30:30   > proc2 step 2
...

Splunk does not have any of the lines before the 'proc2 step2', starting at 20:30:30. Also, as far as I've been able to tell, this is the only time a log file hasn't been imported in its entirety. I did find the following in the UniversalForwarder's splunkd.log file:

06-22-2015 20:30:14.025 -0400 ERROR TcpOutputFd - Read error. An existing connection was forcibly closed by the remote host.
06-22-2015 20:30:14.025 -0400 INFO  TcpOutputProc - Connection to XXX.XXX.XXX.XXX:9997 closed. Read error. An existing connection was forcibly closed by the remote host.
06-22-2015 20:30:14.199 -0400 INFO  TcpOutputProc - Connected to idx=XXX.XXX.XXX.XXX:9997

This error occurs exactly between the lines that are missing and the lines Splunk has. My current understanding is the Universal Forwarder will queue events if it can't talk to the indexer and forward those events when the connection is re-established. Is this incorrect or do I have some other issue going on?

How do I go about troubleshooting why this happened?

Thank you in advance,

Jeremy

[jnicholsenernoc · Accepted solution]

If you aren't using useACK=true in your outputs.conf, that may be why. They were in flight before the forwarder detected the receiver was down.

[jwinderDDS]

This does seem to be the case. Thank you!

I have been able to verify that this has happened to roughly 40 files in the last 5 days. What is the best method of reindexing these files to pick up the missing events?

Thank you again,

Jeremy