Deployment Architecture

Deployment Server Uri different than expected?

AHBrook
Loves-to-Learn

Hey everyone!

I'm in the process of investigating a Splunk instance that I have inherited. I've got a decent handle on things, but I am seeing that the majority of our index is being eaten up by logs from our multiple Active Directory controllers.

Digging around, I see that the local inputs.conf file for the universal forwarder on the DCs is empty, and btool confirms they are not pulling in config from other places. There is, however, a deploymentclient.conf file, with a single targetUri in it.

What's interesting, though, is that the listed TargetUri is not a server name that is present in our environment. It's close, but not exact. Further, I see no signs that this particular domain controller has ever checked in with our deployment server.

I know for a fact that we manually installed the Universal Forwarder on the domain controller. I also know that the correct Deployment Server and Indexer were provided at install time.

So what might have caused the targetUri to change? I'm thinking it may be something in the deployment server itself, but I don't know where to look for that setting or how the deployment server might have updated it. I'm still getting my head wrapped around just what the deployment server itself is doing, in fact. But I am worried that with a full throttle, out of the box universal forwarder, we are likely collecting way more information than we actually want.

 

Labels (1)
0 Karma

Stefanie
Communicator

In $SPLUNK_HOME/etc/deployment-apps/ there could be an app that contains a script.
Also, is your deploymentclient.conf file in $SPLUNK_HOME/etc/system/local or in an app?

0 Karma

AHBrook
Loves-to-Learn

Sorry, missed this one.

 

The deploymentclient.conf I referenced was on the universal forwarder in $SPLUNK_HOME/etc/system/local. The deployment-apps folder just has a README.

 

On our deployment server, there are 22 apps in $SPLUNK_HOME/etc/deployment-apps.

0 Karma

Stefanie
Communicator

No worries, one of those 22 apps may have a script that is modifying the Forwarder's deploymentclient.conf.

Typically it would be in a folder called 'bin' in the app's folder. Do any of the names of the apps in /deployment-apps/ sound like it could have configuration files for the forwarders that connect?

What apps were pushed to the server with the Universal Forwarder on it? It will be in $SPLUNK_HOME/etc/apps/

You could look in the apps on there for a script. It should be something like:  $SPLUNK_HOME/etc/apps/(appname)/bin/script.ps1 

 

 

 

0 Karma

AHBrook
Loves-to-Learn

On the client, there are 6 elements in the apps folder:

  • introspection_generator_addon
  • learned
  • search
  • splunkhttpinput
  • splunk_internal_metrics
  • SplunkUniversalForwarder

The only one with a bin is the introspection generator, with collector.path.

I do see a bunch of .cmd files in $SPLUNK_HOME/etc/system/bin, but those look like they set up the admon, perfmon, powershell, event log, etc.

 

That said, the previous admin did throw the deployment server's configs into our git instance, so I'm gonna go spelunking into that and see if I can find this very particular reference.

 

Edit: And a quick search shows the only place with the URI that I'm looking for in our gitlab... is in a few ansible files. I suspect this change may be something outside of splunk.

I really do greatly appreciate the help figuring this out!

0 Karma

Stefanie
Communicator

Is it possible your Deployment Server has a script that is pointing the deploymentclient.conf to that other TargetUri?

0 Karma

AHBrook
Loves-to-Learn

This is actually the exact scenario I'm trying to hunt down, but I don't know where to look. I've confirmed we have an SCCM deployment for the universal forwarder that was developed but never deployed, and it has the right settings. So it really feels like these servers manually checked in, then were pointed somewhere else. I just.. can't find where that might be coming from.

Tags (1)
0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!