Why are no events being indexed for files being monitored on a universal forwarder?

[att35]

Hi,

I am trying to enable file monitoring using a Splunk universal forwarder, but not able to see any events generated. I've followed other articles for this issue, but in vain.

As a test, I created a text file on the desktop and added to monitoring under inputs.conf

[monitor://C:\Users\UserName\Desktop\splunk_monitor\testing.txt]

Verified using splunk list monitor. ( Please see attachment).

What could be going wrong here? The agent is sending regular event viewer data back to the indexer.. just not for the file modifications.

Please advise.

Thanks,
~ Abhi

[lguinn2 · Accepted solution]

You said "Being a config file, the changes could be anywhere and not necessarily towards the end."
These type of changes are hard to track with the "monitor" capability of Splunk. Splunk maintains a file pointer for each file that is monitored; when new data is added past the current value of the file pointer, Splunk will index the new data. Splunk also looks at the beginning of the file each time the modification time changes: if the beginning of the file is different, Splunk assumes that the file has been completely rewritten and indexes it again from the beginning.

However, you may be able to "trick" Splunk into re-indexing the entire file whenever the contents change. To do this, add the following to the monitor stanza in inputs.conf

initCrcLength = 1048575

This will cause Splunk to look at the first megabyte of the file; if anything has changed in the first megabyte, the entire file will be considered new and will be indexed from the beginning. So this will work only if your configuration file is less than a megabyte in length.

If you only want to index the differences, or if you only want to be alerted if the file changes: use a tool (or some feature of the OS) to identify the changes; have the tool write a message to a log file. Then have Splunk track the log file, not the configuration file.

[lguinn2]

First, take a look at the splunkd.log file; it may give you some hints.

Second, is it possible that the forwarder read the input file, but the indexer was unable to parse/index it? If there was a configuration error, you could use btprobe to reset the file pointer so that the forwarder would re-read and re-send the file.

Third, is new data added at the end of the file? Is the file completely replaced when it is updated? If this is not a "typical" sort of log, where new data is appended at the end, Splunk may not be able to tell what has been added. This is especially likely if a log file is replaced, but the first part of the log file is static - always the same for each log.

[att35]

Thanks Lisa.

File in question is an xml configuration file. We would like to use splunk monitoring to alert each time the file is modified. Being a config file, the changes could be anywhere and not necessarily towards the end.

I see the following under splunkd.log file. File in question is an xml file. I notice splunk successfully parsed and added watch but it does not list "WatchedFile" message for this. Does this mean that splunk is unable to read the file?

**02-03-2016 14:23:26.405 -0500 INFO  TailingProcessor - Parsing configuration stanza: monitor://C:\Program Files (x86)\....\configuration.xml.
02-03-2016 14:23:26.405 -0500 INFO  TailingProcessor - Adding watch on path: C:\Program Files (x86)\....\configuration.xml.**
02-03-2016 14:23:26.405 -0500 INFO  TailingProcessor - Adding watch on path: C:\Program Files\SplunkUniversalForwarder\etc\splunk.version.
02-03-2016 14:23:26.405 -0500 INFO  TailingProcessor - Adding watch on path: C:\Program Files\SplunkUniversalForwarder\var\log\splunk.
02-03-2016 14:23:26.405 -0500 INFO  TailingProcessor - Adding watch on path: C:\Program Files\SplunkUniversalForwarder\var\spool\splunk.
02-03-2016 14:23:26.405 -0500 INFO  BatchReader - State transitioning from 2 to 0 (initOrResume).
02-03-2016 14:23:26.457 -0500 INFO  WatchedFile - Will begin reading at offset=115766 for file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\audit.log'.
02-03-2016 14:23:26.513 -0500 INFO  WatchedFile - Will begin reading at offset=21224502 for file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log'.
02-03-2016 14:23:26.593 -0500 INFO  WatchedFile - Will begin reading at offset=21707 for file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd-utility.log'.
02-03-2016 14:23:26.628 -0500 INFO  TcpOutputProc - Connected to idx=X.X.X.X:9997

[att35]

Tried resetting the file pointer using btprobe. No change in result.

C:\Program Files\SplunkUniversalForwarder\bin>splunk.exe cmd btprobe.exe -d "C:\Program Files\SplunkUniversalForwarder\var\lib\splunk\fishbucket\splunk_private_db" --file "C:\Program Files (x86)\...\configuration.xml" --reset`
Using logging configuration at C:\Program Files\SplunkUniversalForwarder\etc\log-cmdline.cfg.
key=0xc1275e864d63a6a8 scrc=0x998140935b54f7b sptr=5428 fcrc=0x1e06908327507ad5flen=0 mdtm=1454525773 wrtm=1454525774
Record (key 0xc1275e864d63a6a8) reset.