Hello, I am struggling with a directory monitoring problem. I have a directory with a ton of different incremental log names. It's pretty easy to get all of these with the following in inputs.conf:
[monitor://path_to_directory\*.*] sourcetype = source:one index = index01
Now the problem becomes when there is one log that is very similar to all of the other logs that is not indexed correctly without a custom props.conf stanza to correctly format the log. For instance if my logs looks like:
It is the sdk log that needs to formatted differently. Imagine there are 100 logs all with different variations, however only a very specific few that need the custom formatting, each which contains sdk. I was thinking this should be simple by adding another monitor stanza in inputs.conf:
[monitor://path_to_directory\*.*] sourcetype = source:one index = index01 [monitor://path_to_directory\*sdk*.*] sourcetype = source:two index = index01
And then creating the proper [source:two] stanza in props.conf to achieve the formatting. However it appears that this does not work as I was expecting. It does not incorrectly format the sdk logs, it just simply makes splunk not monitor the sdk log at all. It doesn't matter the order of the monitor stanzas in inputs.conf....either way it only monitors all files NOT containing sdk. This doesn't make sense to me. Is there a simple way of monitoring this file that I am not thinking of? There are simply too many variations and different logs that creating 30-40 different monitoring stanzas to specifically target each type individually (imagine 100 more logs with variations on the names).
Finally, if I specify only to monitor the sdk logs...it does it correctly. So the input and props stanzas are correct and would work just fine without the star(dot)star monitor stanza. Any thoughts? Ideally there would be a way to create a stanza that said monitor where files != sdk and a stanza that says monitor files = sdk, but I don't think that syntax is available in a [monitor://] format.
We have our httpd logs in /var/log/httpd, but some are of a different format. However, they do have consistent naming options, so we have:
[monitor:///var/log/httpd/*_log] disabled = false
so all files named "*_log" in the folder are monitored, but they might not get the right sourcetype. Within props.conf, we have:
[source::/var/log/httpd/http...error_log] sourcetype=apache_error [source::/var/log/httpd/http...access_log] sourcetype=access_common [source::/var/log/httpd/...track_log] sourcetype=track_log
That sets the appropriate sourcetype on each. Note that props.conf uses "..." as a wildcard, NOT "*" which might be your problem.