Getting Data In

JSON log extraction at index time

saiynv
New Member

Hi, I am trying to extract a JSON log file at index time. The log structure has a nested key(key,value) pairs.
Like
field 1, field 2 field 3{field 4=value 4 field5=value5 .......}fieldn=valuen.
The nested field value pairs are not uniform
Below is the nested field I'm looking to extract at run time.

"msg":"allowed=true reason=\"UNDER_RATE_LIMIT\" elapsed=1 visitorId=\"18b5b2c1-2201-4dbc-9f6f-9c7aa1941072\" deviceClassification=\"10\" userAgent=\"Mozilla/5.0 (X11; CrOS x86_64 12105.100.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.144 Safari/537.36\" ip=\"xx.xx.xx.xxx\" city=\"sykesville\" region=\"MD\" country=\"US\" lat=\"39.393\" lng=\"-76.931\" asn=\"7922\" application=\"xx-xx\" method=\"post\" endpoint=\"/pdp/graphql/" requestLimit=\"5\" requestLimitWindow=\"5000\" accessToken=\"18b5b2c1-2201-4dbc-9f6f-9c7aa1941072\" remainingRequests=\"4\"","v":1

I've tried with extract at searchtime but I need to get it on index time. I have tried with the regex (?P<_KEY_1>\w+)=(?P<_VAL_1>[^\s]+). But it gives partial results. Please help me out on this

0 Karma

splunk_sv
Path Finder

@saiynv

At the time of indexing, you can specify the type of data you are trying to index which is JSON in this case,
So, that can only get you key-value pairs in which the "msg" will be a field with the rest of the string as its value.
And to use the properties present in that string as general key-value pairs you need to split that string and have to create new key-value pairs from that string in Splunk Query itself.

0 Karma

jawaharas
Motivator

Try below configuration in props.conf for index time extraction.

[SOURCETYPE_NAME]
INDEXED_EXTRACTIONS = json
category = Structured
pulldown_type = 1
detect_trailing_nulls = false
AUTO_KV_JSON = false
KV_MODE = none
0 Karma

saiynv
New Member

It still isn't dividing the whole event.

0 Karma

jawaharas
Motivator

Sample JSON content (full content with all nested structure) and JSON key field that you look to extract will be helpful to assist you.

0 Karma

saiynv
New Member

Below is the complete raw event, which I am trying to extract

{"level":30,"time":1566851259952,"app":{"version":"master-133-0dcbc5f0f3"},"req":{"id":"33bc357d3ca1","method":"post","url":"/pdp/graphql","headers":{"x-ha-session-id":"200480ea-fcf1-4a58-97ef-0b3b2fd2f431","x-ha-device-classification":"10","x-ha-location":"city=xxxx, region=x, country=US, lat=30.378, lng=-97.640, asn=11427","x-ha-visitor-id":"1973a681-6655-4af5-8291-50a5af682eb6","x-ha-device-id":"123abs31-21sf32531-351d5s","x-hat-trace-id":"531685512685316553a35168","x-xxxx-xxxxx":"33bc357d-3ca1-42a5-8a1d-4ae737e2f69e"},"remoteAddress":"xx.xx.xx.xxx","remotePort":xxxxx},"tags":["info","rate-limiter"],"msg":"allowed=true reason=\"UNDER_RATE_LIMIT\" elapsed=1 visitorId=\"1973a681-6655-4af5-8291-sldvkdmvslkd6866\" deviceClassification=\"10\" userAgent=\"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36\" ip=\"xx.xx.xx.xxx\" city=\"xxx\" region=\"xx\" country=\"xx\" lat=\"xx\" lng=\"-97.640\" asn=\"11427\" application=\"xy-ab-bc\" method=\"post\" endpoint=\"/pdp/graphql/abc_bcd\" requestLimit=\"5\" requestLimitWindow=\"5000\" accessToken=\"1973a681-6655-4af5-8291-50a5af682eb6\" remainingRequests=\"4\"","v":1}

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...