I have an environment where I want to use apps like Splunk for Nix, but have the logs go to different indexes.
[monitor:///var/log] whitelist=(\.log|log$|messages|secure|auth|mesg$|cron$|acpid$|\.out) blacklist=(lastlog|anaconda\.syslog) index=os disabled = 1
I don't want the default inputs.conf to have index=os. I want to set the index in another app and be able to upgrade the app without messing with the default inputs.conf of Splunk for Nix each time. For example...
[serverClass:TEST1] whitelist.0 = 184.108.40.206 [serverClass:TEST1:app:TEST1-IndexConfig] [serverClass:TEST2] whitelist.0 = 220.127.116.11 [serverClass:TEST2:app:TEST2-IndexConfig]
TEST1-IndexConfig default inputs.conf
TEST2-IndexConfig default inputs.conf
Am I going to be stuck commenting out all the "index=" in the defaults each time I want to upgrade the app? Or can I specify in the local confs to ignore the default conf attribute and respect the [default] in my other app?
I see your problem. Your best bet is to probably create a separate app for each division. Whenever there's an update you will have to unpack the updated tar to all your apps, but you can have a separate local config for each one then.
Alternatively on the indexer you could rewrite the index with a transforms.conf/props.conf combo based on the host name. Theres a few examples in Splunk answers already. (eg https://answers.splunk.com/answers/135315/different-index-based-on-hostname.html)
Also, you could set search filters for your users to prevent them from accessing data their not supposed to. Have a look at http://docs.splunk.com/Documentation/Splunk/6.3.1511/Security/Addandeditroles#Search_filter_format
I considered this route, however, I want to maintain the flexibility of sending data to specific indexes. For example, if I have a nix box that has the Splunk_TA_nix app sending to index=div02 for the sysadmins; I might also want to send other data to index=finance for the finance people.
I'm getting the feeling that this might be a case of wanting my cake and eating it too lol
It's not the same data. I want the default index to be set depending on the subnet that is used to check into the deployment server. That index holds the generic logs, like /var/log/secure and syslog, etc. And in one-off cases, like /var/log/finance.log would go to index=finance.
I'm a bit confused. This is what I think you're saying.
You have two servers in different subnets.
You want the /var/log/secure to go to a different index based on the subnet the host is in
You do not have any way to distinguish their servers except their subnet (ie hostnames do not relate to status)
You may have other log files in /var/log on the servers that you wan to send to a different index
Is that correct?
Yes, that is an accurate summary.
Also, I have looked at Splunk's routing and filtering of data that you linked. It doesn't solve my problem because very often in my environment, sysadmins will install a Splunk Universal Forwarder on their systems but won't allow me access to them for purposes of configuration. So, I can't control hostnames. All I can do is point them to my deployment server. Filtering on sourcetype doesn't work either, because the sourcetypes will be the same.
Any changes you make to an app's configuration should be done in local for that very reason. Your settings in local override those shipped with the app in default.
I doubt one app can change another app's settings. Instead of removing the index attribute can you just set it to the same value as in the other app?
I wish it were that easy. In my case, when a Universal Forwarder checks in to my deployment server, it gets an inputs.conf with it's [default] index=. All data from that client, unless otherwise specified, goes there.
That way, I can give login's to sysadmins who represent each "div" and they won't have unnecessary access to another division's data.