Getting Data In

Output and deployment client configs constantly overwritten

whar_garbl
Path Finder

I have a single-instance Splunk setup with a handful of Universal Forwarders sending in data. There was previously a different architecture on this network, but this is a new build from the ground up - everything is new builds and fresh installs (all version 8.2.2.1; server is RHEL 8; clients are Windows 10). 

My UFs are installed with command line options to set the forwarding server and deployer (the same place). However, periodically, the clients' outputs.conf and deploymentclient.conf are being overwritten, and I cannot for the life of me figure out why. The settings being pushed in are for the old architecture, none of which remains on the network. Also, notably, it seems to only be the Windows UFs that are getting their settings overwritten - my *nix boxes do not appear to be affected as of now. 

I have attached a ProcMon to monitor the file edits. The changes are coming from splunkd.exe via the REST API: 

C:\Program Files\SplunkUniversalForwarder\bin\splunkd rest --noauth POST /services/data/outputs/tcp/server/ name=wrong_server.domain.com:9997
C:\Program Files\SplunkUniversalForwarder\bin\splunkd rest --noauth POST /services/admin/deploymentclient/deployment-client/ targetUri=wrong_deployer.domain.com:8089

I haven't yet found a way to manually elicit this change, and the update interval seems to vary from just a few minutes to every couple of hours. I've scoured my Group Policy and have not found any relevant settings there. 

I'm stumped. Does anyone have any ideas as to what may be doing this?

Labels (1)
0 Karma
1 Solution

whar_garbl
Path Finder

I figured it out.

Someone had set a compliance baseline in SCCM to check and overwrite the conf files if they didn't match a specified value. Somehow that had been lingering for a couple of years without being disabled. Oops.

View solution in original post

0 Karma

whar_garbl
Path Finder

I figured it out.

Someone had set a compliance baseline in SCCM to check and overwrite the conf files if they didn't match a specified value. Somehow that had been lingering for a couple of years without being disabled. Oops.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

You've ruled out GPO.  Does your company have another management utility, like BigFix, Puppet, or Ansible?

---
If this reply helps you, Karma would be appreciated.

whar_garbl
Path Finder

Nope, I don't have anything like that. 

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Why Splunk Customers Should Attend Cisco Live 2026 Las Vegas

Why Splunk Customers Should Attend Cisco Live 2026 Las Vegas     Cisco Live 2026 is almost here, and this ...

What Is the Name of the USB Key Inserted by Bob Smith? (BOTS Hint, Not the Answer)

Hello Splunkers,   So you searched, “what is the name of the usb key inserted by bob smith?”  Not gonna lie… ...

Automating Threat Operations and Threat Hunting with Recorded Future

    Automating Threat Operations and Threat Hunting with Recorded Future June 29, 2026 | Register   Is your ...