I have a Splunk indexer cluster that is using a service account (non-root) to start Splunk. How do I get the OS logs, like /var/log/messages, /var/log/secure etc... into the cluster indexes? I know that I could stream this to a syslog server and grab it there, but is there an easier way?
Any thoughts are welcome!
The OS logs that you want to collect is from splunk cluster server only OR all other linux servers in your company?
three ways that I know of.
1) chmod -r 777 the log directory
2) add the splunk user to the wheel or root group
3) chown -R root:SplunkGroup /var/log/
Hope this helps? Don't know there may be a more restrictive way to do this?
We faced the same issue. Assuming "splunk" have read access to the OS logs, what we have done is using SplunkTAnix. Put into the "local" of this app, with what files you want to collect by adding the paragraph and putting disable = false (Most of things are already part of TA_nix)
For different layers, enable SplunkTAnix in below fashion.
- For Splunk Forwarders push using deployment-server. It goes into $SPLUNKHOME/etc/apps of forwarders.
- Copy and restart SplunkTAnix into $SPLUNKHOME/etc/apps for deployment-server,
- Copy and restart SplunkTAnix into $SPLUNKHOME/etc/apps for cluster master,
- For clustered Search Heads, package into $SPLUNKHOME/shcluster/etc/apps and push to Search Members. In search members, it will be merged into "default", but works.
- For clustered Indexers, copy SplunkTAnix using cluster master via master-apps. This goes into "slave-apps" of Indexer slaves and works perfectly.
If you enable SplunkTAnix, then you can start colllecting every information about your Splunk Infrastructure/OS
I want to collect OS logs from only the spunk servers themselves, not the forwarders. The forwarders is easy as the univfwd runs as admin on all platforms, its the spunk servers I am concerned about.
Changing the log dir permissions won't work (I do not believe) because when logrotate runs it will create the files with orig permissions.
I think my best bet is going to be to stream the logs to a remote syslog server!?!?
@starcher thanks for the link to github. I went to this talk and I agree with Matt. In slide #13 he basicly put in what I had said above. Matt said
Create a “log reading” group and add the spunk user to it, or simply change group ownership to splunk
chown -R :syslog /var/log
chmod -R g+s /var/log
usermod -a -G syslog splunk
@koshyk It would be great is you could find an answer here and try it out. If you do find an answer please select the answer you like.