Getting Data In
Highlighted

How to get Linux OS logs off a Splunk server, where Splunk is started as a non root account, to index in an indexer cluster?

Builder

I have a Splunk indexer cluster that is using a service account (non-root) to start Splunk. How do I get the OS logs, like /var/log/messages, /var/log/secure etc... into the cluster indexes? I know that I could stream this to a syslog server and grab it there, but is there an easier way?

Any thoughts are welcome!

0 Karma
Highlighted

Re: How to get Linux OS logs off a Splunk server, where Splunk is started as a non root account, to index in an indexer cluster?

SplunkTrust
SplunkTrust

The OS logs that you want to collect is from splunk cluster server only OR all other linux servers in your company?

0 Karma
Highlighted

Re: How to get Linux OS logs off a Splunk server, where Splunk is started as a non root account, to index in an indexer cluster?

Motivator

three ways that I know of.

1) chmod -r 777 the log directory
2) add the splunk user to the wheel or root group
3) chown -R root:SplunkGroup /var/log/

Hope this helps? Don't know there may be a more restrictive way to do this?

Highlighted

Re: How to get Linux OS logs off a Splunk server, where Splunk is started as a non root account, to index in an indexer cluster?

Super Champion

We faced the same issue. Assuming "splunk" have read access to the OS logs, what we have done is using SplunkTAnix. Put into the "local" of this app, with what files you want to collect by adding the paragraph and putting disable = false (Most of things are already part of TA_nix)

For different layers, enable SplunkTAnix in below fashion.
- For Splunk Forwarders push using deployment-server. It goes into $SPLUNKHOME/etc/apps of forwarders.
- Copy and restart Splunk
TAnix into $SPLUNKHOME/etc/apps for deployment-server,
- Copy and restart SplunkTAnix into $SPLUNKHOME/etc/apps for cluster master,
- For clustered Search Heads, package into $SPLUNK
HOME/shcluster/etc/apps and push to Search Members. In search members, it will be merged into "default", but works.
- For clustered Indexers, copy SplunkTAnix using cluster master via master-apps. This goes into "slave-apps" of Indexer slaves and works perfectly.

If you enable SplunkTAnix, then you can start colllecting every information about your Splunk Infrastructure/OS

Highlighted

Re: How to get Linux OS logs off a Splunk server, where Splunk is started as a non root account, to index in an indexer cluster?

Builder

I want to collect OS logs from only the spunk servers themselves, not the forwarders. The forwarders is easy as the univfwd runs as admin on all platforms, its the spunk servers I am concerned about.
Changing the log dir permissions won't work (I do not believe) because when logrotate runs it will create the files with orig permissions.

I think my best bet is going to be to stream the logs to a remote syslog server!?!?

0 Karma
Highlighted

Re: How to get Linux OS logs off a Splunk server, where Splunk is started as a non root account, to index in an indexer cluster?

SplunkTrust
SplunkTrust

My colleague Matt Uebel gave a talk at .conf that covers this topic. His materials are in his git repo at https://github.com/MattUebel/splunk_UF_hardening

0 Karma
Highlighted

Re: How to get Linux OS logs off a Splunk server, where Splunk is started as a non root account, to index in an indexer cluster?

Motivator

@starcher thanks for the link to github. I went to this talk and I agree with Matt. In slide #13 he basicly put in what I had said above. Matt said

Create a “log reading” group and add the spunk user to it, or simply change group ownership to splunk

groupadd syslog
chown -R :syslog /var/log
chmod -R g+s /var/log
usermod -a -G syslog splunk

0 Karma
Highlighted

Re: How to get Linux OS logs off a Splunk server, where Splunk is started as a non root account, to index in an indexer cluster?

Motivator

@koshyk It would be great is you could find an answer here and try it out. If you do find an answer please select the answer you like.

0 Karma