topic Re: What takes precedence, index.conf on Universal Forwarder, or Forwarder Management in Deployment Architecture

What takes precedence, index.conf on Universal Forwarder, or Forwarder Management

kmower — Fri, 15 Mar 2019 03:23:54 GMT

I have just set up forwarder management, and I have noticed that while all my 'apps' are showing as deployed to my clients that have 'phoned home' and downloaded them, when I remote in to the UF machine(s). I am not seeing the updates from my 'inputs.conf' within my deployment server deployment-apps/{appName}/local/inputs.conf directory being reflected on the UF machine (web server) SplunkUniversalForwarder/etc/system/local/inputs.conf ...

So, according to the forwarder manager the changes in my deployment-apps/{appName}/local/inputs.conf have been deployed to my client(s) without error. So, where, if anywhere, should I be seeing the inputs.conf changes on the UF box? Thanks.

Re: What takes precedence, index.conf on Universal Forwarder, or Forwarder Management

richgalloway — Tue, 29 Sep 2020 23:45:03 GMT

The apps received from the deployment server will be found on the forwarders in the $SPLUNK_HOME/etc/apps directory. Deployment servers cannot touch $SPLUNK_HOME/etc/system.

To answer the question in subject line, $SPLUNK_HOME/etc/system/local takes precedence over the same settings in $SPLUNK_HOME/etc/apps/*.

Re: What takes precedence, index.conf on Universal Forwarder, or Forwarder Management

kmower — Sun, 17 Mar 2019 21:09:09 GMT

So, when setting up Forwarder Management, one should remove index.conf from the UF machine $SPLUNK_HOME/etc/system/local entirely? Thanks again.

Re: What takes precedence, index.conf on Universal Forwarder, or Forwarder Management

ddrillic — Sun, 17 Mar 2019 22:01:59 GMT

Right, $SPLUNK_HOME/etc/system/local/indexes.conf shouldn't exist on the UF.

Re: What takes precedence, index.conf on Universal Forwarder, or Forwarder Management

kmower — Sun, 17 Mar 2019 23:38:52 GMT

Yeah, sorry. It is: C:\Program Files\SplunkUniversalForwarder\etc\system\local

Re: What takes precedence, index.conf on Universal Forwarder, or Forwarder Management

richgalloway — Tue, 29 Sep 2020 23:45:22 GMT

Put the contents of C:\Program Files\SplunkUniversalForwarder\etc\system\local into $SPLUNK_HOME\etc\deployment-apps\my_indexes\default\indexes.conf. Delete C:\Program Files\SplunkUniversalForwarder\etc\system\local. Add the my_indexes app to your UF server classes in Forwarder Management.

Re: What takes precedence, index.conf on Universal Forwarder, or Forwarder Management

kmower — Tue, 19 Mar 2019 00:14:58 GMT

OK, the entire C:\Program Files\SplunkUniversalForwarder\etc\system\local directory? Just move/delete it, even deploymentclient.conf which seems to wire it up for management by a deployment server?

I have deleted inputs.conf and outputs.conf from the C:\Program Files\SplunkUniversalForwarder\etc\system\local restarted the service and even reloaded the deploy-server on the splunk indexer. Still only seeing the one W3SVC folder logs, and not even all of the files in there that are my ignoreOlderThan = 90d clause ... it is only grabbing the past week it seems.

Re: What takes precedence, index.conf on Universal Forwarder, or Forwarder Management

richgalloway — Tue, 19 Mar 2019 12:46:36 GMT

We're only discussing indexes.conf so only the file need be deleted from etc\system\local.
When you deleted inputs.conf and outputs.conf from etc\system\local, did you replace them with files in an app?
Do not reload deploy-server on an indexer - it must be done on the deployment server. An indexer should never serve as a deployment server.

Re: What takes precedence, index.conf on Universal Forwarder, or Forwarder Management

kmower — Tue, 19 Mar 2019 21:37:51 GMT

Yes, thanks, that was it .... and a bit more. I did remove outputs.conf and inputs.conf from the Universal Forwarder machine. I have one 'master' app to push outputs.conf from my deployment server, while I have different flavor apps for inputs.conf. Once deployed those apps show up on the Universal Forwarder machines' etc/apps directories.

However, I was still not getting all my iis logs across across. So, in the end I logged a support call with Splunk, and as it turns out:

The Windows Universal forwarder seems to ignore 'small' log files, and we have daily log files from iis; and
There may have been an error with my other W3SVC directory logs - W3SVC2 where it wasn't escaping quotes properly and the Splunk indexer was throwing errors on those, so:

I added to extra lines to my inputs.conf:

initCrcLength = 2310
alwaysOpenFile = 1

This, along with placing a limits.conf file on the Universal Forwarder machine in C:\Program Files\SplunkUniversalForwarder\etc\system\local with the clause:

[thruput]
maxKBps = 0

Has cleared my log forwarding constipation. Everything within my ingoreOlderThan = 90d is now coming through 🙂

Thanks again for the help.

Re: What takes precedence, index.conf on Universal Forwarder, or Forwarder Management

kmower — Tue, 29 Sep 2020 23:44:02 GMT

Oh, yes, also note for the quote issue, and for changing GMT time to the Splunk Server's time settings I added these lines to props.conf on both the indexer and forwarer in etc/system/local/props.conf

[iis]
TIME_FORMAT = %Y-%m-%d %H:%M:%S
TZ = GMT
FIELD_QUOTE = none