Yes you are on the right path, search time is done via the SH, even after its cooked and sent to the indexers. Here's some concepts: Search Time Extractions: This is where the RAW data extractions happen dynamically – done on the SH when the SPL is run – this is called scheme on the fly, not when the data is ingested – and designed so its fast than unlike traditional SQL databases. These have the TA’s installed to do one the fly extractions via props/trans and the right way for field extractions on the fly Forwarders: UF OR HF designed to forward data, attach meta data like sourcetype, host, index. The HF can further parse data and extract fields (Cooked) less work for the Indexers. Indexers, store the raw data and can also do parsing of the data if they are configured to do so (having TA’s) and mainly when sending data directly from UF’s. Forwarders such as HF affect how the data is pre-processed before it reaches the indexers, they apply sourcetype, host, index metadata. The UF's also apply metadata, but not parse unless some structured data types. HF = By performing parsing and field extraction at index time, the data is pre-formatted – this can help with the indexer not having to more performant. Applies metatdata UF = Sends the raw data to indexer, which means the indexer must handle the processing of the data and sometimes can become overloaded, so better to do it at the HF level if you can. Applies metatdata These can possible reasons for it not working, the upgrade should not do this, unless there was some config in the default props.conf file, and this has been overwritten, any custom code should be placed into the local props.conf file : Props/transforms changes (on the SH/Indexers) Changes to the TA may have been modified Inputs changes to metadata sourcetype name Overwritten props in default This may further help understand the Index time vs Search Time process. https://docs.splunk.com/Documentation/Splunk/9.2.1/Indexer/Indextimeversussearchtime Might be worth running btool on the search head and checking where the custom props is configured /opt/splunk/bin/splunk btool props list --debug my:sourcetype Typically is should look like the example below: /opt/splunk/etc/apps/My_Custom_app_props/local/props.conf or OR /opt/splunk/etc/system/local/props.conf
... View more