Splunk Enterprise

Double field extraction for the JSON data

nawazns5038
Builder

I am using only the following props in the heavy forwarder and no other props any where for the data. Yet I am getting double field extractions for the data.
How to solve this problem !!

[xxxxx]
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y-%m-%d %T
TIME_PREFIX = date
MAX_TIMESTAMP_LOOKAHEAD = 30
TRUNCATE = 0
TZ = UTC
NO_BINARY_CHECK=true

KV_MODE = none

LINE_BREAKER = }([\r\n]+|){
INDEXED_EXTRACTIONS = json
AUTO_KV_JSON = false
TRANSFORMS-hostoverriede = hostoverride
EXTRACT-KVPS = (?:args":")?(?<_KEY_1>[^"=]+)=(?:\")?(?<_VAL_1>(\d+|[^\"]+))

Why are search time field extractions happening even though there are no props for the same sourcetype in the search head ?
Or the index field extractions happening at the heavy forwarder and again at indexer level ? .. but there are no props in the indexer !!
The field extractions are normal if we are indexing and searching the data in a stand alone instance but not at a distributed environment.

I am using the following props for another JSON data . That too only at the heavy forwarder level and the extraction is happening only once.
[json_time]
SHOULD_LINEMERGE = false
TIME_FORMAT = %F %T.%3N
TIME_PREFIX = "time"
MAX_TIMESTAMP_LOOKAHEAD = 25
TRUNCATE = 0
TZ = UTC

INDEXED_EXTRACTIONS = json

NO_BINARY_CHECK=true
KV_MODE = none
LINE_BREAKER = ([\r\n]+){
AUTO_KV_JSON = false

So how are the field extractions happening for this data ? without any extraction parameters set ?

Thanks

maciep
Champion

Have you tried setting KV_MODE=none and/or AUTO_KV_JSON=false for these sourcetypes on your search heads? if you don't set something at your search layer, I think splunk will use the default values (e.g. AUTO_KV_JSON=true) , which would explain the double extraction

Configuring search-time settings at your parsing layer won't carry through to your search layer...they need set at the search layer.

nawazns5038
Builder

Hi @maciep,

Thanks for the answer, I haven't set the parameters you have mentioned even for the second case mentioned in the question for the search head , but there are no double field extractions.

I have edited the props.conf for the first case with #INDEXED_EXTRACTIONS = json and KV_MODE = none in the heavy forwarder itself and the extractions are working normally.

Two things I want to know :
1. How are field extractions happening even though no extraction parameters are mentioned .
2. Just modifying the heavy forwarder props have solved the issue, why did it cause double extractions.

0 Karma

maciep
Champion

It's important to understand the phases of data in Splunk and where different setting apply. The field extractions you were seeing were coming from two places.

The INDEXED_EXTRACTIONS setting is done during the input phase. When you use that setting, the fields are indexed with the data. So that's where the first set of fields is coming from.

The next set of extractions are coming from your search head during the search phase. As you mentioned, you didn't configure any settings on the search head for that sourcetype. So that means your sourcetype will be using the default settings. The default value for AUTO_KV_JSON is true. So when you are running searches against that sourcetype, splunk is parsing the json again.

It's important to note that setting kv modes on heavy forwarder are not going to have any effect when searching from your search head. Your search head only knows about its setting. The settings do not carry over from phase to the next.

So when you commented out the indexed extractions in the first case (just like it was in the second case), you eliminated the first set of extractions, leaving just the search-time extractions.

Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!