How to troubleshoot why a universal forwarder is not forwarding contents of a file?

[cdupuis123]

Beating my head off this one guys. I'm simply trying to forward several logs from my SEPM (SYmantec EndPoint Manager). All except the risk log is staying up to date. I've restarted the manager service to have it create it's local type backup of .txt files and changed my input to have it read the agt_risk.txt and it still doesn't make it. The forwarder read the file yesterday! What am I doing wrong?

Excerpt from the splunkd.log:

05-28-2015 10:31:05.516 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor://D:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\data\dump\agt_risk.txt.
05-28-2015 10:31:05.516 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor://D:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\data\dump\agt_scan.tmp.
05-28-2015 10:31:05.517 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor://D:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\data\dump\agt_security.tmp.
05-28-2015 10:31:05.517 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor://D:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\data\dump\agt_system.tmp.
05-28-2015 10:31:05.517 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor://D:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\data\dump\agt_traffic.tmp.
05-28-2015 10:31:05.518 -0400 INFO  TailingProcessor - Adding watch on path: D:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\data\dump\agt_behavior.tmp.
05-28-2015 10:31:05.518 -0400 INFO  TailingProcessor - Adding watch on path: D:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\data\dump\agt_proactive.tmp.
05-28-2015 10:31:05.518 -0400 INFO  TailingProcessor - Adding watch on path: D:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\data\dump\agt_risk.txt.

My inputs.conf

[monitor://D:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\data\dump\agt_behavior.tmp]
disabled = false
crcSalt = <SOURCE>
sourcetype = sep_behavior

[monitor://D:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\data\dump\agt_risk.txt]
disabled = false
crcSalt = <SOURCE>
sourcetype = sep_risk

[monitor://D:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\data\dump\agt_scan.tmp]
disabled = false
crcSalt = <SOURCE>
sourcetype = sep_scan

[monitor://D:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\data\dump\agt_security.tmp]
disabled = false
crcSalt = <SOURCE>
sourcetype = sep_security

[monitor://D:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\data\dump\agt_system.tmp]
disabled = false
crcSalt = <SOURCE>
sourcetype = sep_system

[monitor://D:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\data\dump\agt_traffic.tmp]
disabled = false
crcSalt = <SOURCE>
sourcetype = sep_traffic

[monitor://D:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\data\dump\agt_proactive.tmp]
disabled = false
crcSalt = <SOURCE>
sourcetype = sep_proactive

Yes I know I'm trying to read the agt_risk.txt and the rest of the inputs are .tmp files. This is me trouble-shooting.

I'm stumped. The version of the forwarder I'm running is: 6.2.2

The size of the file I'm struggling with is: 97kb

[woodcock]

Perhaps you have a TZ issue which is sending events "into the future" so when you know "the file was just written" (and it was) and then you look with "last 5 minutes" or something, you don't see any entries. Try searching for "All time"; do you see the recent events (in the future)? Try this search:

index=* | eval lagSecs=_indextime - _time | stats avg(lagSecs) by index,sourcetype,host

The avg should NEVER be neagive and it should be very small.

[cdupuis123]

Thanks woodcock, so I assume this is bad:
main sep_risk SERVERNAME -282.000000

Any ideas or threads on how to fix this? /back to splunk answers......

[woodcock]

Yes, that sourcetype is mis-timestamped for sure: it is impossible for events to "occur" after they have been indexed; furthermore, Splunk's default configuration is to ignore events that "will occur" too far in the future (I believe there is an error log in _index for this) so this may be why some of your events are completely gone. You either have a NTP (clock-sync, wrong time) issue or an incorrect TZ. Only you can debug from here because it is "all in the data".

[ekost]

I rather like the troubleshooting steps taken in the Answers post on debugging a UF that's not reading a log file.

[woodcock]

Are you sure the file is being written? Are you expecting the entire file to be re-forwarded for some reason?

[cdupuis123]

No sir, and yes there's data in the file, it's not a chatty one, but last update was at 1:31 EST today, small file though only 10kb, could that be something???