Splunk Search

Why am I getting duplicate values for all fields from JSON events?

wegscd
Contributor

I have some JSON that I am indexing, using 6.3.0.1. If I index it as sourcetype=_json, all is well.

I defined a new source type, using the information from https://answers.splunk.com/answers/223095/why-is-my-sourcetype-configuration-for-json-events.html, specifically AUTO_KV_JSON = false and KV_MODE = none.

I'm getting duplicate values for all the fields in the JSON when I index it using this sourcetype and search it. What did I miss? I've had and solved this before, but I sure can't figure out what I missed here....

inputs.conf

[monitor:///home/splunk/cloudlock/*]
disabled = 0
sourcetype = cloudlock:incident_aggregate

props.conf

[cloudlock:incident_aggregate]
pulldown_type = true
INDEXED_EXTRACTIONS = json
KV_MODE = none
AUTO_KV_JSON = false
category = Structured
description = Cloudlock incident aggregate

I confirmed that the settings are as expected with splunk btool props list--debug

/opt/splunk/etc/apps/all_whirlpool_transforms/local/props.conf    [cloudlock:incident_aggregate]
/opt/splunk/etc/system/default/props.conf                         ANNOTATE_PUNCT = True
/opt/splunk/etc/apps/all_whirlpool_transforms/local/props.conf    AUTO_KV_JSON = false
/opt/splunk/etc/system/default/props.conf                         BREAK_ONLY_BEFORE = 
/opt/splunk/etc/system/default/props.conf                         BREAK_ONLY_BEFORE_DATE = True
/opt/splunk/etc/system/default/props.conf                         CHARSET = UTF-8
/opt/splunk/etc/system/default/props.conf                         DATETIME_CONFIG = /etc/datetime.xml
/opt/splunk/etc/system/default/props.conf                         HEADER_MODE = 
/opt/splunk/etc/apps/all_whirlpool_transforms/local/props.conf    INDEXED_EXTRACTIONS = json
/opt/splunk/etc/apps/all_whirlpool_transforms/local/props.conf    KV_MODE = none
/opt/splunk/etc/system/default/props.conf                         LEARN_SOURCETYPE = true
/opt/splunk/etc/system/default/props.conf                         LINE_BREAKER_LOOKBEHIND = 100
/opt/splunk/etc/system/default/props.conf                         MAX_DAYS_AGO = 2000
/opt/splunk/etc/system/default/props.conf                         MAX_DAYS_HENCE = 2
/opt/splunk/etc/system/default/props.conf                         MAX_DIFF_SECS_AGO = 3600
/opt/splunk/etc/system/default/props.conf                         MAX_DIFF_SECS_HENCE = 604800
/opt/splunk/etc/system/default/props.conf                         MAX_EVENTS = 256
/opt/splunk/etc/system/default/props.conf                         MAX_TIMESTAMP_LOOKAHEAD = 128
/opt/splunk/etc/system/default/props.conf                         MUST_BREAK_AFTER = 
/opt/splunk/etc/system/default/props.conf                         MUST_NOT_BREAK_AFTER = 
/opt/splunk/etc/system/default/props.conf                         MUST_NOT_BREAK_BEFORE = 
/opt/splunk/etc/system/default/props.conf                         SEGMENTATION = indexing
/opt/splunk/etc/system/default/props.conf                         SEGMENTATION-all = full
/opt/splunk/etc/system/default/props.conf                         SEGMENTATION-inner = inner
/opt/splunk/etc/system/default/props.conf                         SEGMENTATION-outer = outer
/opt/splunk/etc/system/default/props.conf                         SEGMENTATION-raw = none
/opt/splunk/etc/system/default/props.conf                         SEGMENTATION-standard = standard
/opt/splunk/etc/system/default/props.conf                         SHOULD_LINEMERGE = True
/opt/splunk/etc/system/default/props.conf                         TRANSFORMS = 
/opt/splunk/etc/system/default/props.conf                         TRUNCATE = 10000
/opt/splunk/etc/apps/all_whirlpool_transforms/local/props.conf    category = Structured
/opt/splunk/etc/apps/all_whirlpool_transforms/local/props.conf    description = Cloudlock incident aggregate
/opt/splunk/etc/system/default/props.conf                         detect_trailing_nulls = false
/opt/splunk/etc/system/default/props.conf                         maxDist = 100
/opt/splunk/etc/system/default/props.conf                         priority = 
/opt/splunk/etc/apps/all_whirlpool_transforms/local/props.conf    pulldown_type = true
/opt/splunk/etc/system/default/props.conf                         sourcetype = 
0 Karma
1 Solution

wegscd
Contributor

Settings are/were correct; what I missed was setting the correct permissions on the app containing them.

The app wasn't set for global access. Had to add this to local.meta:

[]
access = read : [ * ], write : [ admin, power ]
export = system

View solution in original post

Nicolo_Figiani
Path Finder

Hello,
I went through the same issue. As far as I understood, if you really need indexed extraction, make sure to have your props.conf configured like:

[yoursourcetype]
...
INDEXED_EXTRACTIONS = JSON
KV_MODE = none
AUTO_KV_JSON = false
..

parameters explanation (from props.conf specs):

  • KV_MODE = none: Used for search-time field extractions only. Specifies the field/value extraction @ search time mode for the data. none: if you want no field/value extraction to take place. Defaults to auto: extracts field/value pairs separated by equal signs.
  • AUTO_KV_JSON = false: Used for search-time field extractions only. Specifies whether to try json extraction automatically. Defaults to true.

To have a successful field extraction you should change both KV_MODE and AUTO_KV_JSON as explained above. Chanign just one of the 2 will lead to a field extraction misconfiguration, aka events look like doubled.

Hope this will help, at least for me the above configuration make it sorted..

Additional ref: https://answers.splunk.com/answers/223095/why-is-my-sourcetype-configuration-for-json-events.html

0 Karma

markusspitzli
Communicator

There are two different props.conf you have to create.

This line belongs to the universalforwarder

[yoursourcetype]
INDEXED_EXTRACTIONS = JSON

The other lines belong to the searchhead

[yoursourcetype]
 KV_MODE = none
 AUTO_KV_JSON = false

You probably want to create another props.conf on your indexer (or where you do the parsing). On this splunk instance you should extract the timestamp by yourself. It helps to speed up the parsing so that Splunk doesnt have to find the timestamp by itself.

[yoursourcetype]
TIMESTAMP_FIELDS = date
TIME_FORMAT = %Y%m%d

ssharma09
Explorer

well explained.

0 Karma

freedomson
Explorer

Brilliant worked perfectly!

0 Karma

wegscd
Contributor

Settings are/were correct; what I missed was setting the correct permissions on the app containing them.

The app wasn't set for global access. Had to add this to local.meta:

[]
access = read : [ * ], write : [ admin, power ]
export = system

woodcock
Esteemed Legend

Please elaborate; I do not see how this could cause what you were experiencing.

0 Karma

jtm7x2
Explorer

I just ran into the same issue. The "app" being referenced is the one deploying to the search heads. So, if the props.conf going to the search heads has app/TA only permissions (we name ours 'XXX_TA_logtype'), which doesn't extend to the Search app, they won't work.

INDEXED_EXTRACTIONS = JSON ----> Universal Forwarder

KV_MODE = NONE
AUTO_KV_JSON = FALSE -----> Search Head with permissions set correctly

0 Karma

woodcock
Esteemed Legend

Ah, so you were using both settings, but you didn't think that you were.

0 Karma

wegscd
Contributor

That answer recommends KV_MODE = none, which is already done.

0 Karma

woodcock
Esteemed Legend

Because you are using both INDEXED_EXTRACTIONS and KV_MODE=JSON. You should only be using one, probably the former.

wegscd
Contributor

I'm not seeing where I have KV_MODE = JSON. I'm seeing KV_MODE = none.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...