Getting Data In
Highlighted

What is/isn't getting monitored and why?

Explorer

Is there a way to see what files are being read by the various monitor/fschange stanzas in input.conf?

Highlighted

Re: What is/isn't getting monitored and why?

Explorer

Sure! Point you browser to the splunkd host in question

https://localhost:8089/services/admin/inputstatus/TailingProcessor:FileStatus

to get a full break down of all the files that splunk is aware of, their status, and the reason for their status. I.E.

  • /var/log/apache/access.log
    parent /var/log
    type could not read

  • /var/log/acpid.1.gz
    parent /var/log
    type Did not match whitelist '(.log|log$|messages$|mesg$|cron$|acpid$|.out)'.

  • /opt/splunk/var/log/splunk/splunkd.log
    file position 642076
    file size 642076
    parent $SPLUNK_HOME/var/log/splunk
    percent 100.00
    type open file

Highlighted

Re: What is/isn't getting monitored and why?

Splunk Employee
Splunk Employee

There are a few ways.

The command line has an invocation 'splunk list monitor' which will show you files that splunk found that it thinks it's supposed to read. Generally it will list files that it was configured to read which have no new data, so this is more of a way to validate that the configuration agrees with your file layout than to see what's live.

You can enable more verbose logging to see what's going on. http://www.splunk.com/wiki/Community:Troubleshooting_Monitor_Inputs If you enable category.TailingProcessor to higher output levels via $SPLUNK_HOME/etc/log-local.cfg or via the manager screens, then you'll see greater detail in splunkd.log about what files are being looked at, included, excluded etc. You can search this with splunk, looking at index=_internal.

There's also a protoype endpoint (4.1+) available at https://your.instance:yourport/services/admin/inputstatus/TailingProcessor:FileStatus You can see things like eliminated for crc-collision reasons, eliminated for binary status, didn't match whitelist, matched blacklist, and so on. This can be used remotely on forwarders, so long as the default admin password has been changed (or you've allowed remote login anyway).

Note:

  • This is the splunkd management port, not the splunk web interface.
  • http*s*
  • It will complain about security in most browsers, because it's a self-signed certificate
  • This interface is likely to move or change in future releases, so automating against it is probably not a good idea

View solution in original post