Recently installed the Splunk App and Add-on for Unix and Linux in our Splunk environment. We have a distributed search environment that includes clustered indexers, multiple search heads, and heavy forwarders. Some of our servers are running syslog-ng to collect logs from other systems, and all of those syslog-ng logs are stored on our indexers under /var/log/syslog-ng. The other servers are running rsyslogd. My question is this - I added syslog-ng to the blacklist on the /var/logs section of inputs.conf - [monitor:///var/log] whitelist = (\.log|log$|messages|secure|auth|mesg$|cron$|acpid$|\.out) blacklist = (syslog-ng|lastlog|anaconda\.syslog) index = os disabled = false I did this to prevent the Unix add-on from scraping syslogs that were already being indexed by another app. (the indexers are writing their internal logs to /var/log/messages). Problem is that none of my servers are getting anything from /var/log/messages indexed except for my single Linux server that is not actually part of the Splunk environment. That server is the same OS/in the same environment, but is being used as a monitoring and puppet server. All servers have the same version of inputs.conf. The monitoring/puppet server is running rsyslogd and writing output to /var/log/messages, but as mentioned, other servers in the environment are also running rsyslogd and all are writing their local logs to /var/log/messages. What am I doing wrong that is preventing these other servers from indexing /var/log/messages? It looks like the only thing that is actually being indexed on these other servers at this point is /var/log/cron. UPDATE: Ok, I just realized why it isn't scraping messages - it's because the splunk user doesn't have permission to read /var/log/messages! Which is how it is supposed to be! Now, do I make my server insecure by making messages readable by splunk? ... View more

We are in the process of deploying the Splunk App for Unix and Linux on our Linux servers in a distributed Splunk environment. I was able to successfully change the indexer from the default (os) to the one that we want to use in a standalone instance by modifying the instance name in the untarred source files for Unix app, then installing from those modified files. However, in the distributed environment, we want to be able to install from the source files and then be able to change the index after the install. We already have the index name that we want to use defined on our indexers, but I don't really understand how we can change the indexes after the app is installed. Can anyone give me a hand with this? ... View more

We have syslog-ng configured to collect and forward logs from various network elements, but the linux servers that are running syslog-ng are not configured to collect any logs on themselves. If we deploy the splunk unix/linux app to collect data from these servers, is there any point in having syslog-ng collect logs locally for these servers? It looks like the app collects almost everything that would end up in /var/log/messages (i.e. collecting data from /var/log/audit and lsof). Is there any benefit in having syslog-ng writing the system info to /var/log/messages on these servers once we have the unix/linux app running? ... View more