Why am I getting duplicate values for all fields from JSON events?

[wegscd]

I have some JSON that I am indexing, using 6.3.0.1. If I index it as sourcetype=_json, all is well.

I defined a new source type, using the information from https://answers.splunk.com/answers/223095/why-is-my-sourcetype-configuration-for-json-events.html, specifically AUTO_KV_JSON = false and KV_MODE = none.

I'm getting duplicate values for all the fields in the JSON when I index it using this sourcetype and search it. What did I miss? I've had and solved this before, but I sure can't figure out what I missed here....

inputs.conf

[monitor:///home/splunk/cloudlock/*]
disabled = 0
sourcetype = cloudlock:incident_aggregate

props.conf

[cloudlock:incident_aggregate]
pulldown_type = true
INDEXED_EXTRACTIONS = json
KV_MODE = none
AUTO_KV_JSON = false
category = Structured
description = Cloudlock incident aggregate

I confirmed that the settings are as expected with splunk btool props list--debug

/opt/splunk/etc/apps/all_whirlpool_transforms/local/props.conf    [cloudlock:incident_aggregate]
/opt/splunk/etc/system/default/props.conf                         ANNOTATE_PUNCT = True
/opt/splunk/etc/apps/all_whirlpool_transforms/local/props.conf    AUTO_KV_JSON = false
/opt/splunk/etc/system/default/props.conf                         BREAK_ONLY_BEFORE = 
/opt/splunk/etc/system/default/props.conf                         BREAK_ONLY_BEFORE_DATE = True
/opt/splunk/etc/system/default/props.conf                         CHARSET = UTF-8
/opt/splunk/etc/system/default/props.conf                         DATETIME_CONFIG = /etc/datetime.xml
/opt/splunk/etc/system/default/props.conf                         HEADER_MODE = 
/opt/splunk/etc/apps/all_whirlpool_transforms/local/props.conf    INDEXED_EXTRACTIONS = json
/opt/splunk/etc/apps/all_whirlpool_transforms/local/props.conf    KV_MODE = none
/opt/splunk/etc/system/default/props.conf                         LEARN_SOURCETYPE = true
/opt/splunk/etc/system/default/props.conf                         LINE_BREAKER_LOOKBEHIND = 100
/opt/splunk/etc/system/default/props.conf                         MAX_DAYS_AGO = 2000
/opt/splunk/etc/system/default/props.conf                         MAX_DAYS_HENCE = 2
/opt/splunk/etc/system/default/props.conf                         MAX_DIFF_SECS_AGO = 3600
/opt/splunk/etc/system/default/props.conf                         MAX_DIFF_SECS_HENCE = 604800
/opt/splunk/etc/system/default/props.conf                         MAX_EVENTS = 256
/opt/splunk/etc/system/default/props.conf                         MAX_TIMESTAMP_LOOKAHEAD = 128
/opt/splunk/etc/system/default/props.conf                         MUST_BREAK_AFTER = 
/opt/splunk/etc/system/default/props.conf                         MUST_NOT_BREAK_AFTER = 
/opt/splunk/etc/system/default/props.conf                         MUST_NOT_BREAK_BEFORE = 
/opt/splunk/etc/system/default/props.conf                         SEGMENTATION = indexing
/opt/splunk/etc/system/default/props.conf                         SEGMENTATION-all = full
/opt/splunk/etc/system/default/props.conf                         SEGMENTATION-inner = inner
/opt/splunk/etc/system/default/props.conf                         SEGMENTATION-outer = outer
/opt/splunk/etc/system/default/props.conf                         SEGMENTATION-raw = none
/opt/splunk/etc/system/default/props.conf                         SEGMENTATION-standard = standard
/opt/splunk/etc/system/default/props.conf                         SHOULD_LINEMERGE = True
/opt/splunk/etc/system/default/props.conf                         TRANSFORMS = 
/opt/splunk/etc/system/default/props.conf                         TRUNCATE = 10000
/opt/splunk/etc/apps/all_whirlpool_transforms/local/props.conf    category = Structured
/opt/splunk/etc/apps/all_whirlpool_transforms/local/props.conf    description = Cloudlock incident aggregate
/opt/splunk/etc/system/default/props.conf                         detect_trailing_nulls = false
/opt/splunk/etc/system/default/props.conf                         maxDist = 100
/opt/splunk/etc/system/default/props.conf                         priority = 
/opt/splunk/etc/apps/all_whirlpool_transforms/local/props.conf    pulldown_type = true
/opt/splunk/etc/system/default/props.conf                         sourcetype = 

[Nicolo_Figiani]

Hello,
I went through the same issue. As far as I understood, if you really need indexed extraction, make sure to have your props.conf configured like:

[yoursourcetype]
...
INDEXED_EXTRACTIONS = JSON
KV_MODE = none
AUTO_KV_JSON = false
..

parameters explanation (from props.conf specs):

• KV_MODE = none: Used for search-time field extractions only. Specifies the field/value extraction @ search time mode for the data. none: if you want no field/value extraction to take place. Defaults to auto: extracts field/value pairs separated by equal signs.
• AUTO_KV_JSON = false: Used for search-time field extractions only. Specifies whether to try json extraction automatically. Defaults to true.

To have a successful field extraction you should change both KV_MODE and AUTO_KV_JSON as explained above. Chanign just one of the 2 will lead to a field extraction misconfiguration, aka events look like doubled.

Hope this will help, at least for me the above configuration make it sorted..

Additional ref: https://answers.splunk.com/answers/223095/why-is-my-sourcetype-configuration-for-json-events.html

[markusspitzli]

There are two different props.conf you have to create.

This line belongs to the universalforwarder

[yoursourcetype]
INDEXED_EXTRACTIONS = JSON

The other lines belong to the searchhead

[yoursourcetype]
 KV_MODE = none
 AUTO_KV_JSON = false

You probably want to create another props.conf on your indexer (or where you do the parsing). On this splunk instance you should extract the timestamp by yourself. It helps to speed up the parsing so that Splunk doesnt have to find the timestamp by itself.

[yoursourcetype]
TIMESTAMP_FIELDS = date
TIME_FORMAT = %Y%m%d

[ssharma09]

well explained.

[freedomson]

Brilliant worked perfectly!

[wegscd · Accepted solution]

Settings are/were correct; what I missed was setting the correct permissions on the app containing them.

The app wasn't set for global access. Had to add this to local.meta:

[]
access = read : [ * ], write : [ admin, power ]
export = system

[woodcock]

Please elaborate; I do not see how this could cause what you were experiencing.

[jtm7x2]

I just ran into the same issue. The "app" being referenced is the one deploying to the search heads. So, if the props.conf going to the search heads has app/TA only permissions (we name ours 'XXX_TA_logtype'), which doesn't extend to the Search app, they won't work.

INDEXED_EXTRACTIONS = JSON ----> Universal Forwarder

KV_MODE = NONE
AUTO_KV_JSON = FALSE -----> Search Head with permissions set correctly

[woodcock]

Ah, so you were using both settings, but you didn't think that you were.

[satishsdange]

Please see this https://answers.splunk.com/answers/227596/why-am-i-seeing-a-mismatch-between-key-value-and-c.html

[wegscd]

That answer recommends KV_MODE = none, which is already done.

[woodcock]

Because you are using both INDEXED_EXTRACTIONS and KV_MODE=JSON. You should only be using one, probably the former.

[wegscd]

I'm not seeing where I have KV_MODE = JSON. I'm seeing KV_MODE = none.