Use of SPLUNK ID

[dnmon]

Hi guys, I have a scenario where I have three accounts: splunk, perf, and a user ID (belonging to person using Splunk).
On the Splunk server (say 1), the user ID and the other two will be there. Is it possible for the user to access the splunk ID from the Splunk server under the user ID, e.g. su splunk, and run/submit any commands or scripts under splunk ID on any server or whatever?
What I am getting at here is that I want to remove the user ID from all the servers that have Splunk installed and only have the user ID on the Splunk server itself. Is this doable? Is there a reason why a user will need a user ID on the servers (where it does not belong) where Splunk installed to run commands and create scripts? Is there a reason why scripts have to be created for Splunk (scripts that will run under perf only? And if so, for what reasons? I would think Splunk can just have the nmon files copied over, and whatever commands the user uses can be run under a script as well and the output also sent to Splunk. There should not have to be a user presence in all the non-Splunk servers. Any info here will be greatly appreciated. Thanks in advance!
If I can remove both perf and the user ID, that will be great. I want to know, if Splunk ID is not enough on the non-Splunk servers, what are the real benefits behind Splunk if it cannot function without user intervention to improve the process or send more data to Splunk?

[ddrillic]

We use the KISS principle and have a splunk id and a splunkfwd accounts which the entire Splunk team uses across many servers.

If you want to restrict access to the splunk id you can do it via the sudo command. In such a case, you'll need individual ids on the servers and the users will sudo to the splunk id. Sudo is very specific, if you want it to be, and you can grant the users access to certain scripts, areas, etc.

[dnmon]

Hi, thanks for your response. Yes, KISS principle is good. Unfortunately, our scenario is complicated. I seriously want to avoid having an individual user ID on any AIX/Linux server. The Splunk servers will be the only ones with the individual user ID to access splunk ID via su or sudo, but will it be enough to access the AIX/Linux servers? Should there be any need for an individual user ID to exist on all the AIX/Linux servers (non-Splunk)? If so, for what reasons? We can restrict access using sudo for sure, and we do want to remove the individual user ID, except on the Splunk servers, or can the individual user ID be removed there as well?

Please bear with me, but you say the entire Splunk team uses splunk ID and splunkfwd IDs across many servers. Can you please provide further info on your methods of accessing the many servers through those IDs? Please bear with me on this, but at this time, no one is allowed to directly login to the Splunk ID and the perf ID on any server, Splunk server inclusive. Is it necessary to need the perf to run filemon command? I am trying to get a list of commands the Splunk user wants to run to set up a script instead of manually going into every server and such. Thanks so much!

[ddrillic]

dnmon, I don't know in which environment you are but it seems a bit restrictive. For any software we use, including Splunk, we generate generic Unix accounts which the team uses. Let's take the example of the forwarders on 1,500 servers, which we have. The best solution, I'm aware of, is to have a generic Unix account. For the servers themselves, it's obviously more complex but the generic accounts worked well for us.

Let me know please if it doesn't make sense ; - )

[dnmon]

Hi ddrillic, thanks for your response. Yes, we are restrictive where is it not recommended for a user to log into a generic Unix account, sorry. With the generic user account, that can only be used to run process/job/batch and nothing else, So it appears that an individual user ID is needed for all the servers. When you need to use your generic Unix account, do you just run whatever commands you use for Splunk, e.g. send data to Splunk? Why would you guys log into a server with the generic Unix account, if you do not mind my asking?

[ddrillic]

Why would you guys log into a server with the generic Unix account, if you do not mind my asking?

We do so to bring up the processes if they are down, to check the log files. Practically any Splunk admin task..

[dnmon]

I forgot to mention, the non-Splunk servers I am referencing are Linux and Unix servers. Why would a user need an ID on such servers that have Splunk installed?