Missing search-time extractions after HFW upgrade

[eregon]

Hi all fellow Splunkthiasts, I need your help in understanding what could possibly go wrong in HFW upgrade.

We are collecting some logs through HFW and rely on search-time extractions for this sourcetype.  After upgrading HFW, extractions stopped working.

Data format is basically several Key=Value pairs separated by newline.

Investigating this, I found out:

1. There is no props.conf configuration for this sourcetype on upgraded HFW (and it wasn't there even before upgrade). All relevant configuration is on another instance serving as both indexer and search-head.
2. Props.conf in relevant sourcetype on Search-head has AUTO_KV_JSON=true, I don't see KV_MODE in splunk show config props / splunk btool props list (I suppose it takes default value "auto")
3. I have already realized that upgrade didn't take the right path (it was 7.2.6 upgraded to 9.2.1, without gradually upgrading through 8.2).

Except search-time extractions, everything seems to work as expected (data is flowing in, event breaking and timestamp extraction seems to be correct).

What I don't understand is how can HFW upgrade even affect search-time extractions on another instance. From there I am a bit clueless on what to focus on to fix this.

What I don't understand

[deepakc]

You say you didn’t have a TA (props and transforms on the HF before), normally as the HF is full Splunk instances you should have the TA there for parsing, also known as cooked data and before it reaches the indexer, if you was sending direct then the TA on Indexer will suffice. (Why it worked before the upgrade I don’t know, but the as to the upgrade path, you should always follow the path, as this can often introduce breaking changes, so could be a factor.

 I would try to deploy your custom TA(props code)  onto the HF and see if that makes a difference, as you already have this TA deployed to the current SH/IDX, you should be able to continue with normal field extractions, once it sees the sourcetype.   

So, ensure the code for this data source lives perhaps in a custom TA or copy the code as it is in the SH/IDX and config or deploy that to the HF + restart. Tip for consistency keep the code in one Custom TA app is best practise, otherwise use /local/props.conf.

[eregon]

Hello @deepakc, thank you for your effort to respond. Maybe I was unclear with my question, let me rephrase it.

TL;DR: how can changes on forwarder affect search-head's behavior during search-time? My primary goal is to understand how the extractions work particularly in this detail.

Longer version: The issue here is with search-time extractions. To my knowledge, these extractions are done by search-head and/or respective indexer when the data is being searched (= after all writing to the index is finished, no matter how the data was cooked). Forwarders are not involved in searching, therefore I don't understand how forwarder can affect the search. The events themselves seem to be indexed the same way before and after upgrade, I don't see any distinction in _raw field contents and generally no reason for search-time extractions to stop working.

What in my assumptions and knowledge above is not correct?

Your suggestions apply - as of my knowledge - to index-time extractions, however that is not the case here.

[deepakc]

Yes you are on the right path, search time is done via the SH, even after its cooked and sent to the indexers. 

Here's some concepts:

1. Search Time Extractions: This is where the RAW data extractions happen dynamically – done on the SH when the SPL is run – this is called scheme on the fly, not when the data is ingested – and designed so its fast than unlike traditional SQL databases. These have the TA’s installed to do one the fly extractions via props/trans and the right way for field extractions on the fly 
2. Forwarders: UF OR HF designed to forward data, attach meta data like sourcetype, host, index. The HF can further parse data and extract fields (Cooked) less work for the Indexers. 
3. Indexers, store the raw data and can also do parsing of the data if they are configured to do so (having TA’s)  and mainly when sending data directly from UF’s.

Forwarders such as HF  affect how the data is pre-processed before it reaches the indexers, they apply sourcetype, host, index metadata. The UF's also apply metadata, but not parse unless some structured data types.   

HF = By performing parsing and field extraction at index time, the data is pre-formatted – this can help with the indexer not having to more performant. Applies metatdata 

UF = Sends the raw data to indexer, which means the indexer must handle the processing of the data and sometimes can become overloaded, so better to do it at the HF level if you can. Applies metatdata 

These can possible reasons for it not working,  the upgrade should not do this, unless there was some config in the default props.conf file, and this has been overwritten, any custom code should be placed into the local props.conf file :

1. Props/transforms changes (on the SH/Indexers) 
2. Changes to the TA may have been modified
3. Inputs changes to metadata sourcetype name 
4. Overwritten props in default  

 This may further help understand the Index time vs Search Time process. 

https://docs.splunk.com/Documentation/Splunk/9.2.1/Indexer/Indextimeversussearchtime

Might be worth running btool on the search head and checking where the custom props is configured

/opt/splunk/bin/splunk btool props list --debug my:sourcetype 

Typically is should look like the example below: 

/opt/splunk/etc/apps/My_Custom_app_props/local/props.conf or

OR 

 /opt/splunk/etc/system/local/props.conf