topic Re: Duplicate fields in Splunk events in Splunk Search

Duplicate fields in Splunk events

Yashvik — Tue, 21 May 2024 08:48:44 GMT

Hi All,

When we doing a splunk search in our application (sh_app1), we noticed some fields are duplicated / double up (refer: sample_logs.png)

if we do the same search in another application (sh_welcome_app_ui), we do not see any duplication for the same fields.

cid	Perf-May06-9-151xxx
level	INFO
node_name	aks-application-xxx

SPL being used.

The event count remains same if we search inside that app or any other app, only some fields are duplicated. We couldn't figure out where the actual issue is.
Can someone help?

Re: Duplicate fields in Splunk events

gcusello — Tue, 21 May 2024 05:33:51 GMT

Hi @Yashvik,

this probably depends on the data you're using,

anyway, try to group your ata by a common key usingstats instead table command, something like this:

Ciao.

Giuseppe

Re: Duplicate fields in Splunk events

Yashvik — Tue, 21 May 2024 05:55:07 GMT

Hi @gcusello
Thanks for the reply. using stats helps in removing the duplicate values in "statistics" tab. However, the duplicate fields are still appearing in "Events" tab. I don't understand how it's happening.

Ps. Due to unknown reasons I can't attach images.

Re: Duplicate fields in Splunk events

gcusello — Tue, 21 May 2024 06:02:04 GMT

Hi @Yashvik ,

events are the ones you have, if you don't want duplicated events also in the Events tab use the dedup command (https://docs.splunk.com/Documentation/SCS/current/SearchReference/DedupCommandOverview) to remome the duplicated ones.

Ciao.

Giuseppe

Re: Duplicate fields in Splunk events

Yashvik — Tue, 21 May 2024 08:17:17 GMT

Hi @gcusello
Thanks, however actual issue is fields duplication. Please find the attached screenshot and you will see some fields contains duplicate values (cid, cluster, container_id, container_name etc).
I'd like to understand why they are showing 2 values instead of one.

Re: Duplicate fields in Splunk events

gcusello — Tue, 21 May 2024 08:24:49 GMT

Hi @Yashvik,

as I said, these are your logs and we cannot change them, you can only display them only one time to avoid unuseful duplications.

In addition, this is very frequent having json logs.

For this reason, I hint to use stats to display your logs in Statistics (and dashboard Panels) even if, in the raw logs you have duplicated values in some fields.

You shouldn't modify your logs, they are as they are and you use them displaying what you need.

Ciao.

Giuseppe

Re: Duplicate fields in Splunk events

Yashvik — Tue, 21 May 2024 08:57:25 GMT

Hello @gcusello
But source doesn't contain any duplicate fields while sending to Splunk & they are appearing only if we search within particular app.
As said earlier, If I run the same query outside the app, I don't see these duplicate field values. My users don't have permissions to run the searches outside their app so they see duplicate entries every time.

Re: Duplicate fields in Splunk events

gcusello — Tue, 21 May 2024 10:33:03 GMT

Hi @Yashvik,

your data seems to be json, that usually has duplicated field values.

Anyway, could you share a sample of your data (please not a screenshot)?

About the behaviour in a particoular app, maybe there are some calculated fields that elaborate your values.

Ciao.

Giuseppe

Re: Duplicate fields in Splunk events

Yashvik — Tue, 21 May 2024 12:49:04 GMT

Sure @gcusello

Sample event:

{ [-]
   application: uslcc-nonprod
   cluster: AKS-SYD-NPDI1-ESE-2
   container_id: 9ae09dba5f0ca4c75dfxxxxxxb6b1824ec753663f02d832cf5bfb6f0dxxxxxxx
   container_image: acrsydnpdi1ese.azurecr.io/ms-usl-acct-maint:snapshot-a23584a1221b57xxxxxb437d80xxxxxxb6e65
   container_name: ms-usl-acct-maint
   level: INFO
   log: 2024-05-06 11:08:40.385 INFO 26 --- [org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1] [CID:Perf-May06-9-151615] l.AccountCreditLimitChangedKafkaListener : message="xxxxx listener 'account credit limit event enrichment'", elapsed_time_ms="124"
   namespace: uslcc-nonprod
   node_name: aks-application-3522xxxxx-vmss0000xl
   pod_ip: 10.209.82.xxx
   pod_name: ms-usl-acct-maint-ppte-7dc7xxxxxx-2fc58
   tenant: uslcc
   timestamp: 2024-05-06 11:08:40.385
}

Raw:
{"log":"2024-05-06 11:08:40.385 INFO 26 --- [org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1] [CID:Perf-May06-9-151615] l.AccountCreditLimitChangedxxxxxListener : message=\"xxxxx listener 'account credit limit event enrichment'\", elapsed_time_ms=\"124\"","application":"uslcc-nonprod","cluster":"AKS-SYD-NPDI1-ESE-2","namespace":"uslcc-nonprod","tenant":"uslcc","timestamp":"2024-05-06 11:08:40.385","level":"INFO","container_id":"9ae09dba5xxxxxfd2724b6b1824ec753663f02dxxxxxf0d55d59940","container_name":"ms-usl-acct-maint","container_image":"acrsydnpdi1ese.azurecr.io/ms-usl-acct-maint:snapshot-a23584a1221b5749xxxxxd803eb2aabaxxxxx5","pod_name":"ms-usl-acct-maint-ppte-7dc7c9xxxxc58","pod_ip":"10.209.82.xxx","node_name":"aks-application-35229300-vmssxxxxxl"}

Re: Duplicate fields in Splunk events

gcusello — Wed, 22 May 2024 07:02:31 GMT

Hi @Yashvik,

as I said, check the caculated fields in your app.

Ciao.

Giuseppe

Re: Duplicate fields in Splunk events

Yashvik — Wed, 22 May 2024 08:22:52 GMT

Thanks @gcusello will get it checked.

Re: Duplicate fields in Splunk events

gcusello — Wed, 22 May 2024 08:41:58 GMT

Hi @Yashvik,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉

Re: Duplicate fields in Splunk events

Eider — Wed, 25 Sep 2024 17:13:13 GMT

In my case I was sending TCP info (JSON) through API REST, I had to recreate my source type configuration like this:

Name: Whatever
Description: Whatever
Destination App: Whatever
Category: Whatever
Indexed extractions: json
Next in the Advanced TAB, you need to add this extra setting: KV_MODE = none

The reason is that the json I send via API already contains the event attribute in the splunk expected way, so KV_MODE (key value mode) should be set to none, like this way you avoid double parsing the event json data.

{ "sourcetype": "MyCustomSourceType", "index": "index-name", "event": { "a": "aa", "n": 1, ..... } }