I have a Splunk indexer cluster that is using a service account (non-root) to start Splunk. How do I get the OS logs, like /var/log/messages, /var/log/secure etc... into the cluster indexes? I know that I could stream this to a syslog server and grab it there, but is there an easier way?
Any thoughts are welcome!
@koshyk It would be great is you could find an answer here and try it out. If you do find an answer please select the answer you like.
We faced the same issue. Assuming "splunk" have read access to the OS logs, what we have done is using Splunk_TA_nix. Put into the "local" of this app, with what files you want to collect by adding the paragraph and putting disable = false (Most of things are already part of TA_nix)
For different layers, enable Splunk_TA_nix in below fashion.
- For Splunk Forwarders push using deployment-server. It goes into $SPLUNK_HOME/etc/apps of forwarders.
- Copy and restart Splunk_TA_nix into $SPLUNK_HOME/etc/apps for deployment-server,
- Copy and restart Splunk_TA_nix into $SPLUNK_HOME/etc/apps for cluster master,
- For clustered Search Heads, package into $SPLUNK_HOME/shcluster/etc/apps and push to Search Members. In search members, it will be merged into "default", but works.
- For clustered Indexers, copy Splunk_TA_nix using cluster master via master-apps. This goes into "slave-apps" of Indexer slaves and works perfectly.
If you enable Splunk_TA_nix, then you can start colllecting every information about your Splunk Infrastructure/OS
@starcher thanks for the link to github. I went to this talk and I agree with Matt. In slide #13 he basicly put in what I had said above. Matt said
Create a “log reading” group and add the spunk user to it, or simply change group ownership to splunk
groupadd syslog
chown -R :syslog /var/log
chmod -R g+s /var/log
usermod -a -G syslog splunk
I want to collect OS logs from only the spunk servers themselves, not the forwarders. The forwarders is easy as the univfwd runs as admin on all platforms, its the spunk servers I am concerned about.
Changing the log dir permissions won't work (I do not believe) because when logrotate runs it will create the files with orig permissions.
I think my best bet is going to be to stream the logs to a remote syslog server!?!?
My colleague Matt Uebel gave a talk at .conf that covers this topic. His materials are in his git repo at https://github.com/MattUebel/splunk_UF_hardening
three ways that I know of.
1) chmod -r 777 the log directory
2) add the splunk user to the wheel or root group
3) chown -R root:SplunkGroup /var/log/
Hope this helps? Don't know there may be a more restrictive way to do this?
The OS logs that you want to collect is from splunk cluster server only OR all other linux servers in your company?