topic Splunk Search of a JSON array event provides incorrect field values and count. in Splunk Search

Splunk Search of a JSON array event provides incorrect field values and count.

kk1231 — Wed, 14 Aug 2024 21:12:52 GMT

We have a huge json array event, when I search for that event, search results shows a few missing values for a field. Any suggestion how to fix this issue, and have all values displayed for the field.

Re: Splunk Search of a JSON array event provides incorrect field values and count.

ITWhisperer — Wed, 14 Aug 2024 22:06:52 GMT

This is difficult to diagnose without sight of your events and the search you are currently using. It is possible that you are hitting some sort of limit but where that might be is almost impossible to determine without further information.

Re: Splunk Search of a JSON array event provides incorrect field values and count.

kk1231 — Thu, 15 Aug 2024 16:40:19 GMT

Here is the JSON event, when I hover over on the interested field "LogController_LogMerticsAsync_request.loggerData{}.adType", I am only getting 3 values, as shown in the table, but I see 5 entries of "adTypes" in the raw event.

Values	Count	%
Completes	1	100%
Midpoints	1	100%
Third Quartiles	1	100%

here is the sample json, It is a huge json event, but truncated some data.

{ "@t": "2024-08-14T13:34:42.1718458Z", "@mt": "{className}{methodName}{transactionId}{logLevel}@{LogController_LogMetricsAsync_request}", "className": "D:\\CW\\uploader\\Service\\LogController.cs_152", "methodName": "LogMetricsAsync", "transactionId": "d8e8e141-e9fc749abb0f", "logLevel": "Information", "LogController_LogMetricsAsync_request": { "action": "Device", "event": "Info", "loggerData": [ { "schema": "1.0", "bv": "1.3.41", "dt": "CS", "act": "PlaybackAd", "adType": "Midpoints", "content": "Episode", "adId": "676697" }, { "schema": "1.0", "bv": "1.3.41", "dt": "CS", "act": "PlaybackAd", "adType": "Third Quartiles", "content": "Episode", "adId": "676697" }, { "schema": "1.0", "bv": "1.3.41", "dt": "CS", "act": "PlaybackAd", "adType": "Completes", "adId": "676697" }, { "schema": "1.0", "bv": "1.3.41", "dt": "CS", "act": "NetworkBalance", "data": { "connectionType": "Wi-Fi", "routerInfo": "ARRIS" } }, { "schema": "1.0", "bv": "1.3.41", "dt": "CS", "act": "NetworkBalance", "data": { "connectionType": "Wi-Fi", "routerInfo": "ARRIS" } }, { "schema": "1.0", "bv": "1.3.41", "dt": "CS", "act": "NetworkBalance", "data": { "connectionType": "Wi-Fi", "routerInfo": "ARRIS" } }, { "schema": "1.0", "bv": "1.3.41", "dt": "CS", "act": "NetworkBalance", "data": { "connectionType": "Wi-Fi", "routerInfo": "ARRIS" } }, { "schema": "1.0", "bv": "1.3.41", "dt": "CS", "act": "NetworkBalance", "data": { "connectionType": "Wi-Fi", "routerInfo": "ARRIS" } }, { "schema": "1.0", "bv": "1.3.41", "dt": "CS", "act": "NetworkBalance", "data": { "connectionType": "Wi-Fi", "routerInfo": "ARRIS" } }, { "schema": "1.0", "bv": "1.3.41", "dt": "CS", "act": "NetworkBalance", "data": { "connectionType": "Wi-Fi", "routerInfo": "ARRIS" } }, { "schema": "1.0", "bv": "1.3.41", "dt": "CS", "adType": "Midpoints", "content": "Episode", "adId": "CODE791" }, { "schema": "1.0", "bv": "1.3.41", "dt": "CS", "adType": "Third Quartiles", "content": "Episode", "adId": "CODE791" }, { "schema": "1.0", "bv": "1.3.41", "dt": "CS", "adType": "Completes", "content": "Episode", "adId": "CODE791" }, { "schema": "1.0", "bv": "1.3.41", "dt": "CS", "act": "NetworkBalance", "data": { "connectionType": "Wi-Fi", "routerInfo": "ARRIS" } }, { "schema": "1.0", "bv": "1.3.41", "dt": "CS", "act": "NetworkBalance", "data": { "connectionType": "Wi-Fi", "routerInfo": "ARRIS" } }, { "schema": "1.0", "bv": "1.3.41", "dt": "CS", "adType": "Start", "content": "Episode", "adId": "635897" } ] } }

Re: Splunk Search of a JSON array event provides incorrect field values and count.

yuanliu — Thu, 15 Aug 2024 17:22:38 GMT

Splunk will not automatically give a count or a percentage after search. You need to show the command you use to get those three values. This is why @ITWhisperer says you cannot find an answer without context.

This emulation shows what Splunk gets after raw search.

| makeresults | eval _raw = "{ \"@t\": \"2024-08-14T13:34:42.1718458Z\", \"@mt\": \"{className}{methodName}{transactionId}{logLevel}@{LogController_LogMetricsAsync_request}\", \"className\": \"D:\\\\CW\\\\uploader\\\\Service\\\\LogController.cs_152\", \"methodName\": \"LogMetricsAsync\", \"transactionId\": \"d8e8e141-e9fc749abb0f\", \"logLevel\": \"Information\", \"LogController_LogMetricsAsync_request\": { \"action\": \"Device\", \"event\": \"Info\", \"loggerData\": [ { \"schema\": \"1.0\", \"bv\": \"1.3.41\", \"dt\": \"CS\", \"act\": \"PlaybackAd\", \"adType\": \"Midpoints\", \"content\": \"Episode\", \"adId\": \"676697\" }, { \"schema\": \"1.0\", \"bv\": \"1.3.41\", \"dt\": \"CS\", \"act\": \"PlaybackAd\", \"adType\": \"Third Quartiles\", \"content\": \"Episode\", \"adId\": \"676697\" }, { \"schema\": \"1.0\", \"bv\": \"1.3.41\", \"dt\": \"CS\", \"act\": \"PlaybackAd\", \"adType\": \"Completes\", \"adId\": \"676697\" }, { \"schema\": \"1.0\", \"bv\": \"1.3.41\", \"dt\": \"CS\", \"act\": \"NetworkBalance\", \"data\": { \"connectionType\": \"Wi-Fi\", \"routerInfo\": \"ARRIS\" } }, { \"schema\": \"1.0\", \"bv\": \"1.3.41\", \"dt\": \"CS\", \"act\": \"NetworkBalance\", \"data\": { \"connectionType\": \"Wi-Fi\", \"routerInfo\": \"ARRIS\" } }, { \"schema\": \"1.0\", \"bv\": \"1.3.41\", \"dt\": \"CS\", \"act\": \"NetworkBalance\", \"data\": { \"connectionType\": \"Wi-Fi\", \"routerInfo\": \"ARRIS\" } }, { \"schema\": \"1.0\", \"bv\": \"1.3.41\", \"dt\": \"CS\", \"act\": \"NetworkBalance\", \"data\": { \"connectionType\": \"Wi-Fi\", \"routerInfo\": \"ARRIS\" } }, { \"schema\": \"1.0\", \"bv\": \"1.3.41\", \"dt\": \"CS\", \"act\": \"NetworkBalance\", \"data\": { \"connectionType\": \"Wi-Fi\", \"routerInfo\": \"ARRIS\" } }, { \"schema\": \"1.0\", \"bv\": \"1.3.41\", \"dt\": \"CS\", \"act\": \"NetworkBalance\", \"data\": { \"connectionType\": \"Wi-Fi\", \"routerInfo\": \"ARRIS\" } }, { \"schema\": \"1.0\", \"bv\": \"1.3.41\", \"dt\": \"CS\", \"act\": \"NetworkBalance\", \"data\": { \"connectionType\": \"Wi-Fi\", \"routerInfo\": \"ARRIS\" } }, { \"schema\": \"1.0\", \"bv\": \"1.3.41\", \"dt\": \"CS\", \"adType\": \"Midpoints\", \"content\": \"Episode\", \"adId\": \"CODE791\" }, { \"schema\": \"1.0\", \"bv\": \"1.3.41\", \"dt\": \"CS\", \"adType\": \"Third Quartiles\", \"content\": \"Episode\", \"adId\": \"CODE791\" }, { \"schema\": \"1.0\", \"bv\": \"1.3.41\", \"dt\": \"CS\", \"adType\": \"Completes\", \"content\": \"Episode\", \"adId\": \"CODE791\" }, { \"schema\": \"1.0\", \"bv\": \"1.3.41\", \"dt\": \"CS\", \"act\": \"NetworkBalance\", \"data\": { \"connectionType\": \"Wi-Fi\", \"routerInfo\": \"ARRIS\" } }, { \"schema\": \"1.0\", \"bv\": \"1.3.41\", \"dt\": \"CS\", \"act\": \"NetworkBalance\", \"data\": { \"connectionType\": \"Wi-Fi\", \"routerInfo\": \"ARRIS\" } }, { \"schema\": \"1.0\", \"bv\": \"1.3.41\", \"dt\": \"CS\", \"adType\": \"Start\", \"content\": \"Episode\", \"adId\": \"635897\" } ] } }" | spath ``` data emulation above ``` | table LogController_LogMetricsAsync_request.loggerData{}.adType

The table I get is

LogController_LogMetricsAsync_request.loggerData{}.adType

Midpoints

Third Quartiles

Completes

Midpoints

Third Quartiles

Completes

Start

There are seven values in this array. Play with this emulation and plug subsequent search commands and find out what's wrong in those. Or post your search to get volunteers to help.

Side note: Python, for one, will not accept \\ in JSON. Technically this is invalid in JSON document, but somehow Splunk takes it.

Re: Splunk Search of a JSON array event provides incorrect field values and count.

kk1231 — Thu, 15 Aug 2024 19:32:40 GMT

@yuanliu , I am not running any complex query, with the basic search when I hover over my mouse on the interested field "LogController_LogMerticsAsync_request.loggerData{}.adType", I am only getting top the 3 values instead of 5 values as you provided the table. The Json event I provided is a trauncated, the actual number of lines in JSON format is around 959 Lines. So Is there any limit setting on the search head to analyze whole event?

Re: Splunk Search of a JSON array event provides incorrect field values and count.

yuanliu — Thu, 15 Aug 2024 21:06:17 GMT

Not a search head limit, but an ingestion limit. If you look at raw events, you'll probably see one JSON document broken into multiple "events". The solution is in props.conf (or use Splunk Web to set MAX_EVENTS). Good thing you noticed line numbers. It took me like 2 years. See my post in Getting Data In.

Re: Splunk Search of a JSON array event provides incorrect field values and count.

kk1231 — Mon, 19 Aug 2024 15:53:50 GMT

@yuanliu , I see the whole event in a single line when I search for that event and on the indexer I have this props.

[load_server]
TRUNCATE=999999

Re: Splunk Search of a JSON array event provides incorrect field values and count.

yuanliu — Tue, 20 Aug 2024 04:57:27 GMT

@yuanliu , I see the whole event in a single line when I search for that event and on the indexer I have

Does this conflict with the following?

trauncated, the actual number of lines in JSON format is around 959 Lines. So Is there any limit setting on the search head to analyze whole event?

Could you elaborate, maybe with some real examples? (Anonymize as needed.)