Is there a way to retrieve Universal Forwarder configuration remotely for security configuration compliance auditing?

[jtsplunk1]

Hi,
I am developing a plugin for my organisation's security configuration compliance auditing system, and some Windows Server-based devices have come into scope which are using the Splunk Universal Forwarder to monitor privileged access events. As part of the auditing process for these devices, I will need to verify that Splunk is collecting the correct events and sending them to the correct destination. So far I have come to the conclusion that the audit criteria should be:
1) that inputs.conf includes all necessary logfiles and that disable = 0 for each,
2) that outputs.conf is sending the log digests to the right destination, and
3) the SplunkForwarder service is running and configured to start automatically.
Checking the service is easily done using the svSvc table in the lmmib2 (LanMgr MIB). But I'm struggling to find a way to retrieve the contents of inputs.conf and outputs.conf without literally retrieving the files themselves, something I'm reluctant to do in a production environment on a regular basis.
It doesn't help that I'm not especially familiar with the Windows server platforms, but I would like to know if there is an alternative way I can retrieve the inputs and outputs remotely? Is there a Universal Forwarder SNMP MIB for example? Or does this configuration get stored in the registry somewhere?
I'd also like to know if there's anything else I should be checking to give a reliable confirmation that the Universal Forwarder is operating as expected.
Thanks for your help.

[dineshraj9 · Accepted solution]

You can open the management port(default 8089) on the forwarder, but to access this port you need to change the default admin password on the forwarder from "changeme" to something different. Once you have done that, you can access the apps on the forwarder using REST endpoint and get information on inputs and outputs.

Change password - ./splunk edit user admin -password foo -role admin -auth admin:changeme

Restart forwarder

Access rest endpoint - https://forwarder1.mycompany.com:8089/services/data/inputs/ and enter admin credentials or

OR use CURL command - curl -k -u admin:<password> https://forwarder1.mycompany.com:8089/services/data/inputs/

[jtsplunk1]

Hi Dineshraj9
I wasn't aware of this API - I'm new to Splunk. Hopefully I can get our standard build guys to agree to opening up the API, but this looks very promising. Looking forward to giving it a go ...
Thanks for your answer.

[gjanders]

Have you considered using port 8089 / the REST API of the forwarders to remotely determine what inputs they have?
I believe there is a sub-section of the REST API that will list the inputs, there should also be one for the outputs somewhere.

Other alternatives would be to ingest the config files into Splunk, run the splunk btool command, or retrieve the files themselves...

[jtsplunk1]

Hi Garethatiag
I like the sound of the REST API - didn't know that existed.
Thanks for your input.

[xavierashe]

What do you use for configuration management? You could get this done with Tripwire, BigFix, Tanium, Puppet, etc.

[jtsplunk1]

Hi Xavierashe,
Yes, I'm sure I could get it done with those, but this is a large global organisation with all the tools already in place. I need to extend our security configuration compliance auditing system to retrieve the input and output configuration of the Splunk Universal Forwarders, either by
1) copying the inputs.conf and outputs.conf files (not my preferred solution, hence my original question)
2) some other way (hence my original question).
This extension will take the form of a plugin which I will have to develop.
Thanks.

[adonio]

hello jstsplunk1,
how do you control your forwarders today? how do you distribute configurations today? do you use MC (or DMC) to look for missing forwarders?

[jtsplunk1]

Hi Adonio,
Splunk is deployed as part of a standard server build which is controlled by another part of the organisation, so I can't answer your questions. I have simply been tasked with finding a way to retrieve these Splunk configurations as part of a wider configuration auditing program - it's not so much about auditing the Splunk deployment as a whole across the organisation, rather auditing the Splunk Forwarder configuration on selected servers as part of a broader list of configuration items.
That said, if we are using MC or DMC, and one or other of these was polling each Univeral Forwarder at regular intervals, would I be right in thinking that I could retrieve the required information from the [D]MC via an API of some sort?
Thanks

[adonio]

just throwing an offer here in the air since it seems like you are not using Splunk Deployment Server.
you can create a simple scripted inputs app that has a script that executes on a set interval the command:
./splunk list monitor (read here more) https://docs.splunk.com/Documentation/Splunk/6.5.3/Data/MonitorfilesanddirectoriesusingtheCLI
now you will have all the inputs on each forwarder in splunk.

[jtsplunk1]

Hi Adonio
Sorry, only just saw your suggestion.
Nice idea, although getting CLI access to any kind of centralised server belonging to another group is going to be tricky, especially when that server is responsible for controlling resources across the enterprise.
As it happens the REST API approach sounds perfect, so I'm going to play with that and see how it goes.
Thanks for the idea.