Deployment Architecture

Deployment Server Uri different than expected?

AHBrook
Path Finder

Hey everyone!

I'm in the process of investigating a Splunk instance that I have inherited. I've got a decent handle on things, but I am seeing that the majority of our index is being eaten up by logs from our multiple Active Directory controllers.

Digging around, I see that the local inputs.conf file for the universal forwarder on the DCs is empty, and btool confirms they are not pulling in config from other places. There is, however, a deploymentclient.conf file, with a single targetUri in it.

What's interesting, though, is that the listed TargetUri is not a server name that is present in our environment. It's close, but not exact. Further, I see no signs that this particular domain controller has ever checked in with our deployment server.

I know for a fact that we manually installed the Universal Forwarder on the domain controller. I also know that the correct Deployment Server and Indexer were provided at install time.

So what might have caused the targetUri to change? I'm thinking it may be something in the deployment server itself, but I don't know where to look for that setting or how the deployment server might have updated it. I'm still getting my head wrapped around just what the deployment server itself is doing, in fact. But I am worried that with a full throttle, out of the box universal forwarder, we are likely collecting way more information than we actually want.

 

Labels (1)
0 Karma

Stefanie
Builder

In $SPLUNK_HOME/etc/deployment-apps/ there could be an app that contains a script.
Also, is your deploymentclient.conf file in $SPLUNK_HOME/etc/system/local or in an app?

0 Karma

AHBrook
Path Finder

Sorry, missed this one.

 

The deploymentclient.conf I referenced was on the universal forwarder in $SPLUNK_HOME/etc/system/local. The deployment-apps folder just has a README.

 

On our deployment server, there are 22 apps in $SPLUNK_HOME/etc/deployment-apps.

0 Karma

Stefanie
Builder

No worries, one of those 22 apps may have a script that is modifying the Forwarder's deploymentclient.conf.

Typically it would be in a folder called 'bin' in the app's folder. Do any of the names of the apps in /deployment-apps/ sound like it could have configuration files for the forwarders that connect?

What apps were pushed to the server with the Universal Forwarder on it? It will be in $SPLUNK_HOME/etc/apps/

You could look in the apps on there for a script. It should be something like:  $SPLUNK_HOME/etc/apps/(appname)/bin/script.ps1 

 

 

 

0 Karma

AHBrook
Path Finder

On the client, there are 6 elements in the apps folder:

  • introspection_generator_addon
  • learned
  • search
  • splunkhttpinput
  • splunk_internal_metrics
  • SplunkUniversalForwarder

The only one with a bin is the introspection generator, with collector.path.

I do see a bunch of .cmd files in $SPLUNK_HOME/etc/system/bin, but those look like they set up the admon, perfmon, powershell, event log, etc.

 

That said, the previous admin did throw the deployment server's configs into our git instance, so I'm gonna go spelunking into that and see if I can find this very particular reference.

 

Edit: And a quick search shows the only place with the URI that I'm looking for in our gitlab... is in a few ansible files. I suspect this change may be something outside of splunk.

I really do greatly appreciate the help figuring this out!

0 Karma

Stefanie
Builder

Is it possible your Deployment Server has a script that is pointing the deploymentclient.conf to that other TargetUri?

0 Karma

AHBrook
Path Finder

This is actually the exact scenario I'm trying to hunt down, but I don't know where to look. I've confirmed we have an SCCM deployment for the universal forwarder that was developed but never deployed, and it has the right settings. So it really feels like these servers manually checked in, then were pointed somewhere else. I just.. can't find where that might be coming from.

Tags (1)
0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...