Why are search time field extractions happening even though there are no props for the same sourcetype in the search head ?
Or the index field extractions happening at the heavy forwarder and again at indexer level ? .. but there are no props in the indexer !!
The field extractions are normal if we are indexing and searching the data in a stand alone instance but not at a distributed environment.
I am using the following props for another JSON data . That too only at the heavy forwarder level and the extraction is happening only once.
SHOULD_LINEMERGE = false
TIME_FORMAT = %F %T.%3N
TIME_PREFIX = "time"
MAX_TIMESTAMP_LOOKAHEAD = 25
TRUNCATE = 0
TZ = UTC
Have you tried setting KV_MODE=none and/or AUTO_KV_JSON=false for these sourcetypes on your search heads? if you don't set something at your search layer, I think splunk will use the default values (e.g. AUTO_KV_JSON=true) , which would explain the double extraction
Configuring search-time settings at your parsing layer won't carry through to your search layer...they need set at the search layer.
Thanks for the answer, I haven't set the parameters you have mentioned even for the second case mentioned in the question for the search head , but there are no double field extractions.
I have edited the props.conf for the first case with #INDEXED_EXTRACTIONS = json and KV_MODE = none in the heavy forwarder itself and the extractions are working normally.
Two things I want to know :
1. How are field extractions happening even though no extraction parameters are mentioned .
2. Just modifying the heavy forwarder props have solved the issue, why did it cause double extractions.
It's important to understand the phases of data in Splunk and where different setting apply. The field extractions you were seeing were coming from two places.
The INDEXED_EXTRACTIONS setting is done during the input phase. When you use that setting, the fields are indexed with the data. So that's where the first set of fields is coming from.
The next set of extractions are coming from your search head during the search phase. As you mentioned, you didn't configure any settings on the search head for that sourcetype. So that means your sourcetype will be using the default settings. The default value for AUTO_KV_JSON is true. So when you are running searches against that sourcetype, splunk is parsing the json again.
It's important to note that setting kv modes on heavy forwarder are not going to have any effect when searching from your search head. Your search head only knows about its setting. The settings do not carry over from phase to the next.
So when you commented out the indexed extractions in the first case (just like it was in the second case), you eliminated the first set of extractions, leaving just the search-time extractions.