Why does json data get duplicated after tabling the events?

[kathhuynh]

I have a Splunk On Call webhook that is using a POST request to send data to my index and sourcetype. Anytime a user enters a chat message for an incident, it will fire the webhook and data immediately gets added to that sourcetype.

My issue: The raw events in the index and sourcetype show one event. However, when I table data, the values in each field gets duplicated with the same data as a multivalue field.

Based on other Splunk Community questions, I've made some changes to the sourcetype settings:

[mysourcetype]
AUTO_KV_JSON = false
INDEXED_EXTRACTIONS = json
KV_MODE = none
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Structured
disabled = false
pulldowntype = true

This did not fix the issue like it has for others.

I have tried creating sourcetypes a few different ways:

1. Going into Settings > Sourcetypes > selecting "New Source Type" and updating the settings.
2. Cloning _json sourcetype that Splunk has so I can keep the settings, but am still getting duplicate values when I table.
3 Going into Settings > Data Inputs > HTTP Event Collector > selecting "New Token" > creating a new sourcetype in "Input Settings"

I also noticed that the json events does not highlight syntax by default. Is this due to the KV_MODE being set to none?
Can I set it to json without duplicating my data?

[kathhuynh · Accepted solution]

I tried replicating the sourcetype settings here and it did not work for my case:

https://community.splunk.com/t5/Getting-Data-In/Why-is-my-sourcetype-configuration-for-JSON-events-w...

However, since this data is coming from the Splunk On Call webhook, I found a solution:

I removed the AUTO_KV_JSON and INDEXED EXTRACTIONS settings and KV_MODE=json and the data is coming in with the correct format and not duplicating.

[mysourcetype]
KV_MODE = json
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Structured
disabled = false
pulldowntype = true

I'm guessing the reason why it didn't work before is that INDEXED_EXTRACTIONS "settings change the defaults for other settings in this subsection to appropriate values". It must have conflicted with my inputs.props and the other sourcetype settings somehow.

[PickleRick]

Also, show us the raw event and how it's parsed into fields by splunk.

[kathhuynh]

[PickleRick]

That is really interesting. Can you show us the parsed fields view as well? Because it looks as if the whole event was really being parsed twice creating mvfields of any fields...

[kathhuynh]

Sorry - what do you mean by parsed fields "view"? 

[PickleRick]

What you showed is a raw event (can be switched between raw view and syntax highlited view). Below that you should have a drop down menu with Event Actions and lower yet you should have a list of fields that are parsed from the event.

And do your search in verbose mode to get all fields!

[kathhuynh]

Thank you. Also confirmed I am searching in Verbose. Here's a snippet of the parsed fields.

[PickleRick]

As you can see in the field values - the fields are indeed parsed twice. So it's not a matter of the table command but splunk does see the fields with two values in each of them. I'd hazard a guess that there are some duplicated entries in your configs somewhere but it's hard to say where without going through all your props...

[richgalloway]

Please share the SPL that produced the table.

[kathhuynh]

index=dev_training sourcetype=dev_alerts_logs
| table *