What takes precedence, index.conf on Universal Forwarder, or Forwarder Management

[kmower]

I have just set up forwarder management, and I have noticed that while all my 'apps' are showing as deployed to my clients that have 'phoned home' and downloaded them, when I remote in to the UF machine(s). I am not seeing the updates from my 'inputs.conf' within my deployment server deployment-apps/{appName}/local/inputs.conf directory being reflected on the UF machine (web server) SplunkUniversalForwarder/etc/system/local/inputs.conf ...

So, according to the forwarder manager the changes in my deployment-apps/{appName}/local/inputs.conf have been deployed to my client(s) without error. So, where, if anywhere, should I be seeing the inputs.conf changes on the UF box? Thanks.

[richgalloway · Accepted solution]

The apps received from the deployment server will be found on the forwarders in the $SPLUNK_HOME/etc/apps directory. Deployment servers cannot touch $SPLUNK_HOME/etc/system.

To answer the question in subject line, $SPLUNK_HOME/etc/system/local takes precedence over the same settings in $SPLUNK_HOME/etc/apps/*.

[kmower]

So, when setting up Forwarder Management, one should remove index.conf from the UF machine $SPLUNK_HOME/etc/system/local entirely? Thanks again.

[ddrillic]

Right, $SPLUNK_HOME/etc/system/local/indexes.conf shouldn't exist on the UF.

[kmower]

Yeah, sorry. It is: C:\Program Files\SplunkUniversalForwarder\etc\system\local

[richgalloway]

Put the contents of C:\Program Files\SplunkUniversalForwarder\etc\system\local into $SPLUNK_HOME\etc\deployment-apps\my_indexes\default\indexes.conf. Delete C:\Program Files\SplunkUniversalForwarder\etc\system\local. Add the my_indexes app to your UF server classes in Forwarder Management.

[kmower]

OK, the entire C:\Program Files\SplunkUniversalForwarder\etc\system\local directory? Just move/delete it, even deploymentclient.conf which seems to wire it up for management by a deployment server?

I have deleted inputs.conf and outputs.conf from the C:\Program Files\SplunkUniversalForwarder\etc\system\local restarted the service and even reloaded the deploy-server on the splunk indexer. Still only seeing the one W3SVC folder logs, and not even all of the files in there that are my ignoreOlderThan = 90d clause ... it is only grabbing the past week it seems.

[richgalloway]

We're only discussing indexes.conf so only the file need be deleted from etc\system\local.
When you deleted inputs.conf and outputs.conf from etc\system\local, did you replace them with files in an app?
Do not reload deploy-server on an indexer - it must be done on the deployment server. An indexer should never serve as a deployment server.

[kmower]

Yes, thanks, that was it .... and a bit more. I did remove outputs.conf and inputs.conf from the Universal Forwarder machine. I have one 'master' app to push outputs.conf from my deployment server, while I have different flavor apps for inputs.conf. Once deployed those apps show up on the Universal Forwarder machines' etc/apps directories.

However, I was still not getting all my iis logs across across. So, in the end I logged a support call with Splunk, and as it turns out:

1. The Windows Universal forwarder seems to ignore 'small' log files, and we have daily log files from iis; and
2. There may have been an error with my other W3SVC directory logs - W3SVC2 where it wasn't escaping quotes properly and the Splunk indexer was throwing errors on those, so:

I added to extra lines to my inputs.conf:

initCrcLength = 2310
alwaysOpenFile = 1

This, along with placing a limits.conf file on the Universal Forwarder machine in C:\Program Files\SplunkUniversalForwarder\etc\system\local with the clause:

[thruput]
maxKBps = 0

Has cleared my log forwarding constipation. Everything within my ingoreOlderThan = 90d is now coming through 🙂

Thanks again for the help.

[kmower]

Oh, yes, also note for the quote issue, and for changing GMT time to the Splunk Server's time settings I added these lines to props.conf on both the indexer and forwarer in etc/system/local/props.conf

[iis]
TIME_FORMAT = %Y-%m-%d %H:%M:%S
TZ = GMT
FIELD_QUOTE = none