Why are inputs.conf stanzas not working properly from rsyslog host?

[pickerin]

I have the following directories on my rsyslog forwarder (sysloghost):
/var/log/remote/servacsv/2015-09-27.log
/var/log/remote/clientacsv/2015-09-27.log
/var/log/remote/headdlpmgrv/2015-09-27.log

I'd like to specify the sourcetype and index for the ACS servers separate from the DLP. I have the following stanzas in my $SPLUNKHOME/etc/system/local/inputs.conf:

[monitor://var/log/remote/*acsv/*.log]
disabled = 0
host = 
host_segment = 4
index = network_auth
sourcetype = cisco:acs
blacklist = .gz$

[monitor://var/log/remote/*dlpmgrv/*.log]
disabled = 0
host = 
host_segment = 4
index = dlp
sourcetype = dlp
blacklist = .gz$

The ACS logs are being forwarded to the indexer, but showing up with host "sysloghost" (ignoring the host_segment), with an index of "os", and a sourcetype of ":0.log"

I cannot figure out why.

[pickerin · Accepted solution]

I'm pretty sure the issue was that the files were being indexed by the Splunk_TA_nix, which monitors everything under the /var/log directory. After moving the logs on the syslog aggregator to /var/logs and updating the inputs.conf, things started working properly.

I suppose I could blacklist that path in the Splunk_TA_nix add-on for that host, and may revisit and do just that in the future.

Thanks for reading and responding.

[woodcock]

I converted your comment to an answer; click "Accept" to close it out.

[maciep]

one thing to note, if you're trying to monitor the path /var/log/.... then you should have three slashes after monitor: in your stanza. For example:

[monitor:///var/log/remote/*dlpmgrv/*.log]

I'm wondering if that's somehow causing those symptoms of seemingly indexing the data but in the wrong index/sourcetype.

Also, I think the nix app uses an os index. If that's installed on that box, maybe there are some conflicts there?

[pickerin]

@maciep,

Thanks for the response. I keep seeing different formats of monitor paths, some with two /'s and some with three /'s, so I'm unclear on when to use each. That said, I added a slash to no avail.

However, you then mentioned that the TA for Unix could be causing conflicts, so I checked it. It's monitoring /var/log, though I don't see "recursive" on it, but I'm guessing it was indexing remote (and everything in it).
So, I'm guessing this is how my logs were making it into the index.

I moved my directory out from under /var/log, and now it's not indexing at all. So, I'm guessing there's something wrong with the forwarder configuration.

I put the above stanzas, modified for the new location, in my $SPLUNK_HOME/etc/system/local/inputs.conf

However, now it appears that local/inputs.conf is being ignored, as the files aren't indexing at all now.

[maciep]

Just a quick comment on the monitor format. This is from the inputs.conf spec file on splunk docs:

[monitor://<path>]
* This directs Splunk to watch all files in <path>. 
* <path> can be an entire directory or just a single file.
* You must specify the input type and then the path, so put three slashes in your path if you are starting 
at the root (to include the slash that goes before the root directory).

So the first 2 slashes are actually part of monitor definition. And then typically 3rd slash (on nix) is the start of the path to the file/directory to monitor. On windows, instead of having a 3rd slash, you'd start your path there, like this

[monitor://C:\program files\blah\blah.txt]

Not sure if that's where you've seen the difference between 2 and 3 slashes, but hopefully it helps a little