Getting Data In

What is/isn't getting monitored and why?

drawks
Explorer

Is there a way to see what files are being read by the various monitor/fschange stanzas in input.conf?

1 Solution

jrodman
Splunk Employee
Splunk Employee

There are a few ways.

The command line has an invocation 'splunk list monitor' which will show you files that splunk found that it thinks it's supposed to read. Generally it will list files that it was configured to read which have no new data, so this is more of a way to validate that the configuration agrees with your file layout than to see what's live.

You can enable more verbose logging to see what's going on. http://www.splunk.com/wiki/Community:Troubleshooting_Monitor_Inputs If you enable category.TailingProcessor to higher output levels via $SPLUNK_HOME/etc/log-local.cfg or via the manager screens, then you'll see greater detail in splunkd.log about what files are being looked at, included, excluded etc. You can search this with splunk, looking at index=_internal.

There's also a protoype endpoint (4.1+) available at https://your.instance:yourport/services/admin/inputstatus/TailingProcessor:FileStatus You can see things like eliminated for crc-collision reasons, eliminated for binary status, didn't match whitelist, matched blacklist, and so on. This can be used remotely on forwarders, so long as the default admin password has been changed (or you've allowed remote login anyway).

Note:

  • This is the splunkd management port, not the splunk web interface.
  • http*s*
  • It will complain about security in most browsers, because it's a self-signed certificate
  • This interface is likely to move or change in future releases, so automating against it is probably not a good idea

View solution in original post

jrodman
Splunk Employee
Splunk Employee

There are a few ways.

The command line has an invocation 'splunk list monitor' which will show you files that splunk found that it thinks it's supposed to read. Generally it will list files that it was configured to read which have no new data, so this is more of a way to validate that the configuration agrees with your file layout than to see what's live.

You can enable more verbose logging to see what's going on. http://www.splunk.com/wiki/Community:Troubleshooting_Monitor_Inputs If you enable category.TailingProcessor to higher output levels via $SPLUNK_HOME/etc/log-local.cfg or via the manager screens, then you'll see greater detail in splunkd.log about what files are being looked at, included, excluded etc. You can search this with splunk, looking at index=_internal.

There's also a protoype endpoint (4.1+) available at https://your.instance:yourport/services/admin/inputstatus/TailingProcessor:FileStatus You can see things like eliminated for crc-collision reasons, eliminated for binary status, didn't match whitelist, matched blacklist, and so on. This can be used remotely on forwarders, so long as the default admin password has been changed (or you've allowed remote login anyway).

Note:

  • This is the splunkd management port, not the splunk web interface.
  • http*s*
  • It will complain about security in most browsers, because it's a self-signed certificate
  • This interface is likely to move or change in future releases, so automating against it is probably not a good idea

drawks
Explorer

Sure! Point you browser to the splunkd host in question

https://localhost:8089/services/admin/inputstatus/TailingProcessor:FileStatus

to get a full break down of all the files that splunk is aware of, their status, and the reason for their status. I.E.

  • /var/log/apache/access.log
    parent /var/log
    type could not read

  • /var/log/acpid.1.gz
    parent /var/log
    type Did not match whitelist '(.log|log$|messages$|mesg$|cron$|acpid$|.out)'.

  • /opt/splunk/var/log/splunk/splunkd.log
    file position 642076
    file size 642076
    parent $SPLUNK_HOME/var/log/splunk
    percent 100.00
    type open file

Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Think Like an Architect: Introducing the Splunk Certified Cybersecurity Defense ...

In cybersecurity, defenders respond to threats. Architects design the systems that stop them.    As ...

Best Practices: Splunk auto adjust pipeline queue

When you enable autoAdjustQueue in Splunk, maxSize should be understood as the queue size Splunk starts with ...

Announcing Modern Navigation: A New Era of Splunk User Experience

We are excited to introduce the Modern Navigation feature in the Splunk Platform, available to both cloud and ...