How to disable system default INDEXED_EXTRACTIONS setting on 6.x Universal forwarders?

[t9445]

Hi, recently we upgraded all of our Universal forwaders (UFs) from various versions of 5.x to 6.1.4

We discovered that our field extractions of the IIS data were randonly broken now - e.g. c_ip and other fields were sporadically popluated
incorrectly, the _raw data was sporadically munged -- they had never had been these andom issues before (running for several years now and
no changes to the IIS props/transforms in several months...)

After allot of trouble-shooting, finally discoved that the INDEXED_EXTRACTIONS setting for iis by default is set to w3c on the Universal
forwarders is the culprit - however since it is set in system/default/pros.conf as = w3c -- there is no-apparent way to disable it (set it to off)
unless we setup something to update all of the etc/default/props.conf (not a good idea obvioulsy)

So we ended up using a new sourcetype in inputs.conf on our UFs, and then forcing the new sourcetype to change to souretype=iis on our indexers,
now our IIS field extractions are working again...

We tried setting it a number of ways on the UFs and it did not take (in-fact in some cases it stopped forwrding the IIS data on the UFs
when this setting was changed)

The question is - is there anyway to disable INDEXED_EXTRACTIONS?

Additonally any ETA of when this wil be fixed (if a known issue?)

thanks for any and all inputs

-tom

[t9445]

Hi thanks for the feeback,

Yes, we tried unsetting by setting it to blank and ("unset") - both actually caused iis to no longer be collected on the 6.1.4 (6.x) UFs

e.g. In our troubleshooting we distributed a new UFs props.conf to the 6.x UFs - both did not work and actually caused iis to stop being collected on the 6.x UFs.

---

[iis]

INDEXED_EXTRACTIONS =

and also tried

---

[iis]

INDEXED_EXTRACTIONS = unset

in both the cases the 6.x UFs actually stopped forwarding the IIS data due this setting being incorrect.

Additionally, since the only workaround appears to be using a custom sourcetype on the UFs then changing it at the indexers, the UFs (correctly) re-index all the data since it is a new sourcetype... (blew out our license over the weekend - no choice since alternative is to have randomly munged data where it never was before)

There must be a way to turn off INDEXED_EXTRACTIONS so this scenario does not play out with other data-types as well (e.g. access_combined, json, etc) - For Splunk users migrating to 6.x of the UFs (from 5.x and earlier) this is going to pose a huge issue since the INDEXED_EXTRACTIONS settings that are done by default on well known sourcetypes.

The munged _raw data from the UFs is seemingly random, in our shop about 0.5% of the rows coming in for iis sourcetype - just enough to get every user lighting torches and gathering pitchforks, since various reports/alerts/queries/etc now have bogus data peppered through them

Defaulting INDEXED_EXTRACTIONS on the 6.x UFs for those migrating from earlier versions is not pretty -- the only workarounds seems to be rewrite all of our working field-extractions, to setup a distributed application to update etc/system/default/props.conf on all the UFs (anytime one is spun up), or change the sourcetype name on the UFs and change it back on the indexers thus avoiding re-indexing the data (unless the default setting on the 6.x UFs for INDEXED_EXTRACTIONS can be turned back off somehow?)

Appreciate any and all inputs

thanks again

-tom

[yeahnah]

Anyone who comes across this issue please up vote the following idea for a configuration option to disable INDEXED_EXTRACTIONS via an app's local props.conf.

https://ideas.splunk.com/ideas/EID-I-2400

[yannK]

The indexed extractions are setup on the /etc/system/default/props.conf on forwarders and indexers.

If you want to disable them, rewrite the "iis" sourcetype in local/props.conf without them.


[iis]
INDEXED_EXTRACTIONS =

You may want to do that on forwarders AND indexers, otherwise I do not know what the indexers will try to do if they receive iis as unparsed and decide to parse it as INDEXED_EXTRACTIONS without having the header.

[martin_mueller]

What happens if you set

INDEXED_EXTRACTIONS = 

in your local/props.conf? Never tried that myself, but maybe that'll unset the default.