Getting Data In

File monitor not indexing all data in a file

ericlavalley
Explorer

I've installed UF on a Windows 2012 R2 server and created a directory monitor via the inputs.conf file at C:\Program Files\SplunkUniversalForwarder\etc\system\local. A scheduled script drops a CSV into the monitored directory on a routine basis and Splunk indexes the data. The problem I'm having is that Splunk is not indexing everything in the CSV and the data that is skipped appears to be random.

I've tested cscSalt = and I've checked the data for any weird timestamps that may be parsed, but it's an unmodified CSV and Splunk seems to be splitting events appropriately. I've been browsing Splunk Answers for a few hours and haven't had any luck.

Does anyone have any ideas here? I've copied the monitor stanza below.

[monitor://C:\Security\Logs\MessageTrace]
disabled=false
index=o365
host=o365_custom
source=o365_MessageTrace
crcSalt = <SOURCE>
Tags (1)
0 Karma

woodcock
Esteemed Legend

How many files are in that directory? If there are thousands, you will have this kind of problem.

0 Karma

ssadanala1
Contributor

It would be ideal if you post your props tooo .

0 Karma

ericlavalley
Explorer

As the forwarding is being performed via a Universal Forwarder I don't have a props.conf on this server.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Whats the name pattern of the CSV file? Whats the file rolling strategy?

0 Karma

ericlavalley
Explorer

The file name pattern is "MessageTrace_MM-dd-yyyy-HH-mm-ss.csv". The script runs every 5 minutes and generates an entirely new file. I am not appending to an existing file so there isn't really a rolling strategy. The directory is populated with a new file each time the script runs that has an updated name for that specific time and date with logs that were generated within that 5 minute window.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

The inputs.conf stanza was not properly formatted and may have been missing few characters. I did the formatting but if you could verify if it's all there. The monitoring stanza is monitor://C:\Security\Logs\MessageTrace OR monitor://C:\Security\Logs\MessageTrace*? Did you use the crcSalt = <SOURCE> (literal string <SOURCE>)?

0 Karma

ericlavalley
Explorer

Looks like that has identified the issue.

04-27-2018 10:46:22.406 -0400 WARN DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (128) characters of event. Defaulting to timestamp of previous event (Fri Apr 27 14:34:24 2018). Context: source::o365_MessageTrace|host::o365_custom|csv|1754\r\n 6851 similar messages suppressed. First occurred at: Fri Apr 27 10:35:32 2018

The timestamp of the event is the first column of the CSV, so I'm not sure why it's not picking that up?

0 Karma

somesoni2
SplunkTrust
SplunkTrust

This is where we'd dig into your o365_MessageTrace sourcetype definition in props.conf on your Indexers/Heavy Forwarders.

0 Karma

ericlavalley
Explorer

The monitoring stanza does not contain an asterisk. My understanding is that if I want to include all files in the folder named "MessageTrace" at the specified path the monitor should be "monitor://C:\Security\Logs\MessageTrace".

For crcSalt I used the literal string per the inputs.conf Splunk documentation.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Can you check if you see any error in splunkd.log of the UF for any of the missing file(s)?

index=_internal  sourcetype=splunkd host=yourUFHostName *<<Missed File Name here>>
0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...