Getting Data In

JSON log extraction at index time

saiynv
New Member

Hi, I am trying to extract a JSON log file at index time. The log structure has a nested key(key,value) pairs.
Like
field 1, field 2 field 3{field 4=value 4 field5=value5 .......}fieldn=valuen.
The nested field value pairs are not uniform
Below is the nested field I'm looking to extract at run time.

"msg":"allowed=true reason=\"UNDER_RATE_LIMIT\" elapsed=1 visitorId=\"18b5b2c1-2201-4dbc-9f6f-9c7aa1941072\" deviceClassification=\"10\" userAgent=\"Mozilla/5.0 (X11; CrOS x86_64 12105.100.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.144 Safari/537.36\" ip=\"xx.xx.xx.xxx\" city=\"sykesville\" region=\"MD\" country=\"US\" lat=\"39.393\" lng=\"-76.931\" asn=\"7922\" application=\"xx-xx\" method=\"post\" endpoint=\"/pdp/graphql/" requestLimit=\"5\" requestLimitWindow=\"5000\" accessToken=\"18b5b2c1-2201-4dbc-9f6f-9c7aa1941072\" remainingRequests=\"4\"","v":1

I've tried with extract at searchtime but I need to get it on index time. I have tried with the regex (?P<_KEY_1>\w+)=(?P<_VAL_1>[^\s]+). But it gives partial results. Please help me out on this

0 Karma

splunk_sv
Path Finder

@saiynv

At the time of indexing, you can specify the type of data you are trying to index which is JSON in this case,
So, that can only get you key-value pairs in which the "msg" will be a field with the rest of the string as its value.
And to use the properties present in that string as general key-value pairs you need to split that string and have to create new key-value pairs from that string in Splunk Query itself.

0 Karma

jawaharas
Motivator

Try below configuration in props.conf for index time extraction.

[SOURCETYPE_NAME]
INDEXED_EXTRACTIONS = json
category = Structured
pulldown_type = 1
detect_trailing_nulls = false
AUTO_KV_JSON = false
KV_MODE = none
0 Karma

saiynv
New Member

It still isn't dividing the whole event.

0 Karma

jawaharas
Motivator

Sample JSON content (full content with all nested structure) and JSON key field that you look to extract will be helpful to assist you.

0 Karma

saiynv
New Member

Below is the complete raw event, which I am trying to extract

{"level":30,"time":1566851259952,"app":{"version":"master-133-0dcbc5f0f3"},"req":{"id":"33bc357d3ca1","method":"post","url":"/pdp/graphql","headers":{"x-ha-session-id":"200480ea-fcf1-4a58-97ef-0b3b2fd2f431","x-ha-device-classification":"10","x-ha-location":"city=xxxx, region=x, country=US, lat=30.378, lng=-97.640, asn=11427","x-ha-visitor-id":"1973a681-6655-4af5-8291-50a5af682eb6","x-ha-device-id":"123abs31-21sf32531-351d5s","x-hat-trace-id":"531685512685316553a35168","x-xxxx-xxxxx":"33bc357d-3ca1-42a5-8a1d-4ae737e2f69e"},"remoteAddress":"xx.xx.xx.xxx","remotePort":xxxxx},"tags":["info","rate-limiter"],"msg":"allowed=true reason=\"UNDER_RATE_LIMIT\" elapsed=1 visitorId=\"1973a681-6655-4af5-8291-sldvkdmvslkd6866\" deviceClassification=\"10\" userAgent=\"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36\" ip=\"xx.xx.xx.xxx\" city=\"xxx\" region=\"xx\" country=\"xx\" lat=\"xx\" lng=\"-97.640\" asn=\"11427\" application=\"xy-ab-bc\" method=\"post\" endpoint=\"/pdp/graphql/abc_bcd\" requestLimit=\"5\" requestLimitWindow=\"5000\" accessToken=\"1973a681-6655-4af5-8291-50a5af682eb6\" remainingRequests=\"4\"","v":1}

0 Karma
Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...