How to monitor and alert on a folder for an unindexed file due to no matching sourcetype

[dmcgeearke]

I am looking for a way to monitor a folder for files that are [not yet defined] into a sourcetype so that i can have a report of un-indexed log files in splunk to review on occasion.

We have a custom application that comes with default logs, but it also allows the architects to create custom logs based on modules that are added on top of the default build. So each implementation of the application will have the 4 base logs + any number of other logtypes in the same folder. Each time we run across an implementation with a new log type, we build the new source type, if we deem it necessary, but we'd like to know when and if that new log type appears in the folder, even if it's not yet being indexed.

basically, i want this:

Splunk says:

I found the following files, and have parsed them accordingly:

log.foo - sourcetype:foo
log.bar - sourcetype:bar
log.zoo - sourcetype:zoo
log.foo2 - sourcetype:foo

I also found this, but i didn't know what to do with it, so here's an alert:

log.zam

At which point we can decide if log.zam is worth monitoring, whether or not we need a new sourcetype for it, and then work on getting it indexed into splunk

[woodcock]

You do it with a script. Have it tail off the last event and then run a remote search with ssh -c to a Search Head to see if splunk has that event.

[DalJeanis]

Follow the instructions on this page, which tells you where the REST endpoint would be.

https://www.splunk.com/blog/2011/01/02/did-i-miss-christmas-2.html

This basically amounts to either

./splunk list inputstatus

or

https://(the forwarder in question):8089/services/admin/inputstatus/TailingProcessor:FileStatus

According to my tests, both of those give roughly the same output.

Parse the output to identify the files with type = "ignored".

Please let us know how it goes.