Deployment Server Uri different than expected?

[AHBrook]

Hey everyone!

I'm in the process of investigating a Splunk instance that I have inherited. I've got a decent handle on things, but I am seeing that the majority of our index is being eaten up by logs from our multiple Active Directory controllers.

Digging around, I see that the local inputs.conf file for the universal forwarder on the DCs is empty, and btool confirms they are not pulling in config from other places. There is, however, a deploymentclient.conf file, with a single targetUri in it.

What's interesting, though, is that the listed TargetUri is not a server name that is present in our environment. It's close, but not exact. Further, I see no signs that this particular domain controller has ever checked in with our deployment server.

I know for a fact that we manually installed the Universal Forwarder on the domain controller. I also know that the correct Deployment Server and Indexer were provided at install time.

So what might have caused the targetUri to change? I'm thinking it may be something in the deployment server itself, but I don't know where to look for that setting or how the deployment server might have updated it. I'm still getting my head wrapped around just what the deployment server itself is doing, in fact. But I am worried that with a full throttle, out of the box universal forwarder, we are likely collecting way more information than we actually want.

[Stefanie]

In $SPLUNK_HOME/etc/deployment-apps/ there could be an app that contains a script.
Also, is your deploymentclient.conf file in $SPLUNK_HOME/etc/system/local or in an app?

[AHBrook]

Sorry, missed this one.

The deploymentclient.conf I referenced was on the universal forwarder in $SPLUNK_HOME/etc/system/local. The deployment-apps folder just has a README.

On our deployment server, there are 22 apps in $SPLUNK_HOME/etc/deployment-apps.

[Stefanie]

No worries, one of those 22 apps may have a script that is modifying the Forwarder's deploymentclient.conf.

Typically it would be in a folder called 'bin' in the app's folder. Do any of the names of the apps in /deployment-apps/ sound like it could have configuration files for the forwarders that connect?

What apps were pushed to the server with the Universal Forwarder on it? It will be in $SPLUNK_HOME/etc/apps/

You could look in the apps on there for a script. It should be something like:  $SPLUNK_HOME/etc/apps/(appname)/bin/script.ps1 

[AHBrook]

On the client, there are 6 elements in the apps folder:

• introspection_generator_addon
• learned
• search
• splunkhttpinput
• splunk_internal_metrics
• SplunkUniversalForwarder

The only one with a bin is the introspection generator, with collector.path.

I do see a bunch of .cmd files in $SPLUNK_HOME/etc/system/bin, but those look like they set up the admon, perfmon, powershell, event log, etc.

That said, the previous admin did throw the deployment server's configs into our git instance, so I'm gonna go spelunking into that and see if I can find this very particular reference.

Edit: And a quick search shows the only place with the URI that I'm looking for in our gitlab... is in a few ansible files. I suspect this change may be something outside of splunk.

I really do greatly appreciate the help figuring this out!

[Stefanie]

Is it possible your Deployment Server has a script that is pointing the deploymentclient.conf to that other TargetUri?

[AHBrook]

This is actually the exact scenario I'm trying to hunt down, but I don't know where to look. I've confirmed we have an SCCM deployment for the universal forwarder that was developed but never deployed, and it has the right settings. So it really feels like these servers manually checked in, then were pointed somewhere else. I just.. can't find where that might be coming from.