Configuration:
We were thinking of creating one specific user for accessing the Splunk server while searching data from the app server.
The question is how do we configure the trust relationship between the application server and the splunk server such that the application server does not need to provide authentication information (user id and password) in order to invoke Restful APIs on the Splunk server?
The sdk talks about creating a .splunkrc file in the user's home directory. However this would not be safe specially if the passwords are kept in clear text.
The setup that would correspond best to what I think you're trying to achieve is really to setup distributed search. Make your application server a search head, add the indexer as its search peer (this sets up a trust relationship that works very much like what you're describing) and then issue searches locally on the search head. You'll still have to decide how to deal with authentication anyway, but at least the trust relationship between the application server and the indexer is established.
Well that's what I mean - what difference would there be between having this "trusted server" authenticate with some credentials lying on that server, and having it access Splunk without having to authenticate at all? End result is the same - if you're worried about users with access to the server being able to access Splunk through it, that will be the result in both scenarios.
@Ayn
For the splunk web, only the authenticated users can logon to Splunk server. Essentially our requirement is that if the Splunk server is configured with a list of trusted hosts, then server to server requests being initiated from those hosts (application servers - no individual users) would be allowed to retrieve information from Splunk.
@Ayn
If we put splunk userid/password in a file, then any one who can read that file can logon to splunk (using the same user id and password) to retrieve the information stored in splunk. However our request is that user id and password would not be needed if a configured process (not a user) within the application server needs to access information on the Splunk server.
@Kristian
No this is not for distributed search. We are creating a front-end application to query splunk. However access control is being implemented on the application tier. The application tier is the only one entity that talks to our front end.
Also I'm not sure how you would consider passwords in a file on the server less secure than a connection with no need for credentials whatsoever?...
Er.. this sounds as if you want to set up distributed search, no? One search head (GUI, users etc), and one indexer, (holds the data).
That is core splunk functionality. See the docs. Or perhaps you have a reason to build it yourselves.