Getting Data In

Is there a way to retrieve Universal Forwarder configuration remotely for security configuration compliance auditing?

jtsplunk1
Engager

Hi,
I am developing a plugin for my organisation's security configuration compliance auditing system, and some Windows Server-based devices have come into scope which are using the Splunk Universal Forwarder to monitor privileged access events. As part of the auditing process for these devices, I will need to verify that Splunk is collecting the correct events and sending them to the correct destination. So far I have come to the conclusion that the audit criteria should be:
1) that inputs.conf includes all necessary logfiles and that disable = 0 for each,
2) that outputs.conf is sending the log digests to the right destination, and
3) the SplunkForwarder service is running and configured to start automatically.
Checking the service is easily done using the svSvc table in the lmmib2 (LanMgr MIB). But I'm struggling to find a way to retrieve the contents of inputs.conf and outputs.conf without literally retrieving the files themselves, something I'm reluctant to do in a production environment on a regular basis.
It doesn't help that I'm not especially familiar with the Windows server platforms, but I would like to know if there is an alternative way I can retrieve the inputs and outputs remotely? Is there a Universal Forwarder SNMP MIB for example? Or does this configuration get stored in the registry somewhere?
I'd also like to know if there's anything else I should be checking to give a reliable confirmation that the Universal Forwarder is operating as expected.
Thanks for your help.

0 Karma
1 Solution

dineshraj9
Builder

You can open the management port(default 8089) on the forwarder, but to access this port you need to change the default admin password on the forwarder from "changeme" to something different. Once you have done that, you can access the apps on the forwarder using REST endpoint and get information on inputs and outputs.

Change password - ./splunk edit user admin -password foo -role admin -auth admin:changeme

Restart forwarder

Access rest endpoint - https://forwarder1.mycompany.com:8089/services/data/inputs/ and enter admin credentials or

OR use CURL command - curl -k -u admin:<password> https://forwarder1.mycompany.com:8089/services/data/inputs/

View solution in original post

dineshraj9
Builder

You can open the management port(default 8089) on the forwarder, but to access this port you need to change the default admin password on the forwarder from "changeme" to something different. Once you have done that, you can access the apps on the forwarder using REST endpoint and get information on inputs and outputs.

Change password - ./splunk edit user admin -password foo -role admin -auth admin:changeme

Restart forwarder

Access rest endpoint - https://forwarder1.mycompany.com:8089/services/data/inputs/ and enter admin credentials or

OR use CURL command - curl -k -u admin:<password> https://forwarder1.mycompany.com:8089/services/data/inputs/

jtsplunk1
Engager

Hi Dineshraj9
I wasn't aware of this API - I'm new to Splunk. Hopefully I can get our standard build guys to agree to opening up the API, but this looks very promising. Looking forward to giving it a go ...
Thanks for your answer.

0 Karma

gjanders
SplunkTrust
SplunkTrust

Have you considered using port 8089 / the REST API of the forwarders to remotely determine what inputs they have?
I believe there is a sub-section of the REST API that will list the inputs, there should also be one for the outputs somewhere.

Other alternatives would be to ingest the config files into Splunk, run the splunk btool command, or retrieve the files themselves...

jtsplunk1
Engager

Hi Garethatiag
I like the sound of the REST API - didn't know that existed.
Thanks for your input.

0 Karma

xavierashe
Contributor

What do you use for configuration management? You could get this done with Tripwire, BigFix, Tanium, Puppet, etc.

0 Karma

jtsplunk1
Engager

Hi Xavierashe,
Yes, I'm sure I could get it done with those, but this is a large global organisation with all the tools already in place. I need to extend our security configuration compliance auditing system to retrieve the input and output configuration of the Splunk Universal Forwarders, either by
1) copying the inputs.conf and outputs.conf files (not my preferred solution, hence my original question)
2) some other way (hence my original question).
This extension will take the form of a plugin which I will have to develop.
Thanks.

0 Karma

adonio
Ultra Champion

hello jstsplunk1,
how do you control your forwarders today? how do you distribute configurations today? do you use MC (or DMC) to look for missing forwarders?

0 Karma

jtsplunk1
Engager

Hi Adonio,
Splunk is deployed as part of a standard server build which is controlled by another part of the organisation, so I can't answer your questions. I have simply been tasked with finding a way to retrieve these Splunk configurations as part of a wider configuration auditing program - it's not so much about auditing the Splunk deployment as a whole across the organisation, rather auditing the Splunk Forwarder configuration on selected servers as part of a broader list of configuration items.
That said, if we are using MC or DMC, and one or other of these was polling each Univeral Forwarder at regular intervals, would I be right in thinking that I could retrieve the required information from the [D]MC via an API of some sort?
Thanks

0 Karma

adonio
Ultra Champion

just throwing an offer here in the air since it seems like you are not using Splunk Deployment Server.
you can create a simple scripted inputs app that has a script that executes on a set interval the command:
./splunk list monitor (read here more) https://docs.splunk.com/Documentation/Splunk/6.5.3/Data/MonitorfilesanddirectoriesusingtheCLI
now you will have all the inputs on each forwarder in splunk.

0 Karma

jtsplunk1
Engager

Hi Adonio
Sorry, only just saw your suggestion.
Nice idea, although getting CLI access to any kind of centralised server belonging to another group is going to be tricky, especially when that server is responsible for controlling resources across the enterprise.
As it happens the REST API approach sounds perfect, so I'm going to play with that and see how it goes.
Thanks for the idea.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...