Hi, am I doing this correct or is there another way to tabulate this JSON? I've seen many examples on the forums of people using mvexpand and mvzip to tabulate their JSON but this is working with just a few fields rather than a handful and not to any depths. I have events pulled from the splunk add-on for microsoft cloud services. As you may already know, this pulls body.records{}->[Many subitems in an array]... for individual events: One (redacted) event in splunk may look like so: { body: { records: [ { backendPoolName: XXXX-pool backendSettingName: XXXX-backend-setting category: ApplicationGatewayAccessLog listenerName: XXXX-ssl-listener operationName: ApplicationGatewayAccess properties: { clientIP: 4.5.6.7 clientPort: 56522 host: xxxx httpMethod: GET httpStatus: 302 httpVersion: HTTP/1.1 instanceId: appgw_1 originalHost: XXXX originalRequestUriWithArgs: /XXXX receivedBytes: 386 requestQuery: client_id=XXXX requestUri: /XXXX sentBytes: 1404 serverResponseLatency: 0.012 serverRouted: XXXX serverStatus: 302 sslCipher: ECDHE-RSA-AES256-GCM-SHA384 sslClientCertificateFingerprint: sslClientCertificateIssuerName: sslClientVerify: NONE sslEnabled: on sslProtocol: TLSv1.2 timeTaken: 0.013 transactionId: XXXX userAgent: Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/) } resourceId: /SUBSCRIPTIONS/XXXX/RESOURCEGROUPS/XXXX/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/XXXX ruleName: XXXX-ruleset timeStamp: 2021-03-05T23:53:01+00:00 } { backendPoolName: XXXX-pool backendSettingName: XXXX-backend-setting category: ApplicationGatewayAccessLog listenerName: XXXX-ssl-listener operationName: ApplicationGatewayAccess properties: { clientIP: 1.2.3.4 clientPort: 52478 host: XXXX httpMethod: GET httpStatus: 200 httpVersion: HTTP/1.1 instanceId: appgw_1 originalHost: XXXX originalRequestUriWithArgs: /session/XXXX receivedBytes: 2087 requestQuery: XXXX requestUri: /session/XXXX sentBytes: 6081 serverResponseLatency: 0.024 serverRouted: XXXX serverStatus: 200 sslCipher: ECDHE-RSA-AES256-GCM-SHA384 sslClientCertificateFingerprint: sslClientCertificateIssuerName: sslClientVerify: NONE sslEnabled: on sslProtocol: TLSv1.2 timeTaken: 0.028 transactionId: XXXX userAgent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/87.0.4280.20 Safari/537.36 } resourceId: /SUBSCRIPTIONS/XXXX/RESOURCEGROUPS/XXXX-PRGATEWAY/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/GATEWAY-PR-AG ruleName: XXXX-ruleset timeStamp: 2021-03-05T23:53:03+00:00 } ] } x-opt-enqueued-time: 1614988459311 x-opt-offset: 29218690517200 x-opt-sequence-number: 19033991 } First I had issues with empty values, so i have resolved this. Some of these events are so big they wont render in the web gui, so its hard to tell what's going on, we have a plan to break them down using props/SEDCMD. This is my query: index=eventhub AND source="*C113*" AND body.records{}.resourceId="/subscriptions/XXXXX/resourcegroups/XXXXX-prgateway/providers/microsoft.network/applicationgateways/gateway-pr-ag" | eval _raw = replace (_raw, "\":\"\",","\":\"N/A\",") | spath | rename body.records{}.timeStamp as timeStamp_ | rename body.records{}.properties.receivedBytes as receivedBytes_ | rename body.records{}.properties.clientIP as clientIP_ | rename body.records{}.properties.clientPort as clientPort_ | rename body.records{}.properties.host as host_ | rename body.records{}.properties.httpMethod as httpMethod_ | rename body.records{}.properties.httpStatus as httpStatus_ | rename body.records{}.properties.originalHost as originalHost_ | rename body.records{}.properties.originalRequestUriWithArgs as originalRequestUriWithArgs_ | rename body.records{}.properties.timeTaken as timeTaken_ | rename body.records{}.properties.userAgent as userAgent_ | eval x = mvzip(timeStamp_, clientIP_, ";;") | mvexpand x | eval x = split(x, ";;") | eval timeStamp2 = mvindex(x, 0) | eval clientIP2 = mvindex(x, 1) | eval x = mvzip(clientIP_, receivedBytes_, ";;") | mvexpand x | eval x = split(x, ";;") | eval clientIP2 = mvindex(x, 0) | eval receivedBytes2 = mvindex(x, 1) | eval x = mvzip(clientIP_, clientPort_, ";;") | mvexpand x | eval x = split(x, ";;") | eval clientIP2 = mvindex(x, 0) | eval clientPort2 = mvindex(x, 1) | eval x = mvzip(clientIP_, host_, ";;") | mvexpand x | eval x = split(x, ";;") | eval clientIP2 = mvindex(x, 0) | eval host2 = mvindex(x, 1) | eval x = mvzip(clientIP_, httpMethod_, ";;") | mvexpand x | eval x = split(x, ";;") | eval clientIP2 = mvindex(x, 0) | eval httpMethod2 = mvindex(x, 1) | eval x = mvzip(clientIP_, httpStatus_, ";;") | mvexpand x | eval x = split(x, ";;") | eval clientIP2 = mvindex(x, 0) | eval httpStatus2 = mvindex(x, 1) | eval x = mvzip(clientIP_, originalHost_, ";;") | mvexpand x | eval x = split(x, ";;") | eval clientIP2 = mvindex(x, 0) | eval originalHost2 = mvindex(x, 1) | eval x = mvzip(clientIP_, originalRequestUriWithArgs_, ";;") | mvexpand x | eval x = split(x, ";;") | eval clientIP2 = mvindex(x, 0) | eval originalRequestUriWithArgs2 = mvindex(x, 1) | eval x = mvzip(clientIP_, timeTaken_, ";;") | mvexpand x | eval x = split(x, ";;") | eval clientIP2 = mvindex(x, 0) | eval timeTaken2 = mvindex(x, 1) | eval x = mvzip(clientIP_, userAgent_, ";;") | mvexpand x | eval x = split(x, ";;") | eval clientIP2 = mvindex(x, 0) | eval userAgent2 = mvindex(x, 1) | table timeStamp2, receivedBytes2, clientIP2, clientPort2, host2, httpMethod2, httpStatus2, originalHost2, originalRequestUriWithArgs2, timeTaken2, userAgent2 | dedup timeStamp2, receivedBytes2, clientIP2, clientPort2, host2, httpMethod2, httpStatus2, originalHost2, originalRequestUriWithArgs2, timeTaken2, userAgent2 I'm seeing exponentially more results in my table, the more fields I add. limits.conf is being mentioned for result truncation with mvexpand being used. I'm seeing user agents which, when I check an individual record manually, does not map, where an event has just one entry for a clientIP it's spawning many lines in my resulting table which appear to cover all combinations. Any help on how to query this slightly advanced JSON dataset would be good, or perhaps point out where im going wrong? p.s. we have historical data we need to process, event breaking is on the horizon but we cant really get out of this one that easily...
... View more