Deployment Architecture

What takes precedence, index.conf on Universal Forwarder, or Forwarder Management

kmower
Communicator

I have just set up forwarder management, and I have noticed that while all my 'apps' are showing as deployed to my clients that have 'phoned home' and downloaded them, when I remote in to the UF machine(s). I am not seeing the updates from my 'inputs.conf' within my deployment server deployment-apps/{appName}/local/inputs.conf directory being reflected on the UF machine (web server) SplunkUniversalForwarder/etc/system/local/inputs.conf ...

So, according to the forwarder manager the changes in my deployment-apps/{appName}/local/inputs.conf have been deployed to my client(s) without error. So, where, if anywhere, should I be seeing the inputs.conf changes on the UF box? Thanks.

0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

The apps received from the deployment server will be found on the forwarders in the $SPLUNK_HOME/etc/apps directory. Deployment servers cannot touch $SPLUNK_HOME/etc/system.

To answer the question in subject line, $SPLUNK_HOME/etc/system/local takes precedence over the same settings in $SPLUNK_HOME/etc/apps/*.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

The apps received from the deployment server will be found on the forwarders in the $SPLUNK_HOME/etc/apps directory. Deployment servers cannot touch $SPLUNK_HOME/etc/system.

To answer the question in subject line, $SPLUNK_HOME/etc/system/local takes precedence over the same settings in $SPLUNK_HOME/etc/apps/*.

---
If this reply helps you, Karma would be appreciated.

kmower
Communicator

So, when setting up Forwarder Management, one should remove index.conf from the UF machine $SPLUNK_HOME/etc/system/local entirely? Thanks again.

0 Karma

ddrillic
Ultra Champion

Right, $SPLUNK_HOME/etc/system/local/indexes.conf shouldn't exist on the UF.

0 Karma

kmower
Communicator

Yeah, sorry. It is: C:\Program Files\SplunkUniversalForwarder\etc\system\local

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Put the contents of C:\Program Files\SplunkUniversalForwarder\etc\system\local into $SPLUNK_HOME\etc\deployment-apps\my_indexes\default\indexes.conf. Delete C:\Program Files\SplunkUniversalForwarder\etc\system\local. Add the my_indexes app to your UF server classes in Forwarder Management.

---
If this reply helps you, Karma would be appreciated.
0 Karma

kmower
Communicator

OK, the entire C:\Program Files\SplunkUniversalForwarder\etc\system\local directory? Just move/delete it, even deploymentclient.conf which seems to wire it up for management by a deployment server?

I have deleted inputs.conf and outputs.conf from the C:\Program Files\SplunkUniversalForwarder\etc\system\local restarted the service and even reloaded the deploy-server on the splunk indexer. Still only seeing the one W3SVC folder logs, and not even all of the files in there that are my ignoreOlderThan = 90d clause ... it is only grabbing the past week it seems.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

We're only discussing indexes.conf so only the file need be deleted from etc\system\local.
When you deleted inputs.conf and outputs.conf from etc\system\local, did you replace them with files in an app?
Do not reload deploy-server on an indexer - it must be done on the deployment server. An indexer should never serve as a deployment server.

---
If this reply helps you, Karma would be appreciated.
0 Karma

kmower
Communicator

Yes, thanks, that was it .... and a bit more. I did remove outputs.conf and inputs.conf from the Universal Forwarder machine. I have one 'master' app to push outputs.conf from my deployment server, while I have different flavor apps for inputs.conf. Once deployed those apps show up on the Universal Forwarder machines' etc/apps directories.

However, I was still not getting all my iis logs across across. So, in the end I logged a support call with Splunk, and as it turns out:

  1. The Windows Universal forwarder seems to ignore 'small' log files, and we have daily log files from iis; and
  2. There may have been an error with my other W3SVC directory logs - W3SVC2 where it wasn't escaping quotes properly and the Splunk indexer was throwing errors on those, so:

I added to extra lines to my inputs.conf:

initCrcLength = 2310
alwaysOpenFile = 1

This, along with placing a limits.conf file on the Universal Forwarder machine in C:\Program Files\SplunkUniversalForwarder\etc\system\local with the clause:

[thruput]
maxKBps = 0

Has cleared my log forwarding constipation. Everything within my ingoreOlderThan = 90d is now coming through 🙂

Thanks again for the help.

0 Karma

kmower
Communicator

Oh, yes, also note for the quote issue, and for changing GMT time to the Splunk Server's time settings I added these lines to props.conf on both the indexer and forwarer in etc/system/local/props.conf

[iis]
TIME_FORMAT = %Y-%m-%d %H:%M:%S
TZ = GMT
FIELD_QUOTE = none

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In September, the Splunk Threat Research Team had two releases of new security content via the Enterprise ...

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...