I've installed UF on a Windows 2012 R2 server and created a directory monitor via the inputs.conf file at C:\Program Files\SplunkUniversalForwarder\etc\system\local. A scheduled script drops a CSV into the monitored directory on a routine basis and Splunk indexes the data. The problem I'm having is that Splunk is not indexing everything in the CSV and the data that is skipped appears to be random.
I've tested cscSalt = and I've checked the data for any weird timestamps that may be parsed, but it's an unmodified CSV and Splunk seems to be splitting events appropriately. I've been browsing Splunk Answers for a few hours and haven't had any luck.
Does anyone have any ideas here? I've copied the monitor stanza below.
[monitor://C:\Security\Logs\MessageTrace] disabled=false index=o365 host=o365_custom source=o365_MessageTrace crcSalt = <SOURCE>
The file name pattern is "MessageTrace_MM-dd-yyyy-HH-mm-ss.csv". The script runs every 5 minutes and generates an entirely new file. I am not appending to an existing file so there isn't really a rolling strategy. The directory is populated with a new file each time the script runs that has an updated name for that specific time and date with logs that were generated within that 5 minute window.
The inputs.conf stanza was not properly formatted and may have been missing few characters. I did the formatting but if you could verify if it's all there. The monitoring stanza is
monitor://C:\Security\Logs\MessageTrace*? Did you use the
crcSalt = <SOURCE> (literal string
Looks like that has identified the issue.
04-27-2018 10:46:22.406 -0400 WARN DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (128) characters of event. Defaulting to timestamp of previous event (Fri Apr 27 14:34:24 2018). Context: source::o365_MessageTrace|host::o365_custom|csv|1754\r\n 6851 similar messages suppressed. First occurred at: Fri Apr 27 10:35:32 2018
The timestamp of the event is the first column of the CSV, so I'm not sure why it's not picking that up?
The monitoring stanza does not contain an asterisk. My understanding is that if I want to include all files in the folder named "MessageTrace" at the specified path the monitor should be "monitor://C:\Security\Logs\MessageTrace".
For crcSalt I used the literal string per the inputs.conf Splunk documentation.