JSON log extraction at index time

[saiynv]

Hi, I am trying to extract a JSON log file at index time. The log structure has a nested key(key,value) pairs.
Like
field 1, field 2 field 3{field 4=value 4 field5=value5 .......}fieldn=valuen.
The nested field value pairs are not uniform
Below is the nested field I'm looking to extract at run time.

"msg":"allowed=true reason=\"UNDER_RATE_LIMIT\" elapsed=1 visitorId=\"18b5b2c1-2201-4dbc-9f6f-9c7aa1941072\" deviceClassification=\"10\" userAgent=\"Mozilla/5.0 (X11; CrOS x86_64 12105.100.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.144 Safari/537.36\" ip=\"xx.xx.xx.xxx\" city=\"sykesville\" region=\"MD\" country=\"US\" lat=\"39.393\" lng=\"-76.931\" asn=\"7922\" application=\"xx-xx\" method=\"post\" endpoint=\"/pdp/graphql/" requestLimit=\"5\" requestLimitWindow=\"5000\" accessToken=\"18b5b2c1-2201-4dbc-9f6f-9c7aa1941072\" remainingRequests=\"4\"","v":1

I've tried with extract at searchtime but I need to get it on index time. I have tried with the regex (?P<_KEY_1>\w+)=(?P<_VAL_1>[^\s]+). But it gives partial results. Please help me out on this

[splunk_sv]

@saiynv

At the time of indexing, you can specify the type of data you are trying to index which is JSON in this case,
So, that can only get you key-value pairs in which the "msg" will be a field with the rest of the string as its value.
And to use the properties present in that string as general key-value pairs you need to split that string and have to create new key-value pairs from that string in Splunk Query itself.

[jawaharas]

Try below configuration in props.conf for index time extraction.

[SOURCETYPE_NAME]
INDEXED_EXTRACTIONS = json
category = Structured
pulldown_type = 1
detect_trailing_nulls = false
AUTO_KV_JSON = false
KV_MODE = none

[saiynv]

It still isn't dividing the whole event.

[jawaharas]

Sample JSON content (full content with all nested structure) and JSON key field that you look to extract will be helpful to assist you.

[saiynv]

Below is the complete raw event, which I am trying to extract

{"level":30,"time":1566851259952,"app":{"version":"master-133-0dcbc5f0f3"},"req":{"id":"33bc357d3ca1","method":"post","url":"/pdp/graphql","headers":{"x-ha-session-id":"200480ea-fcf1-4a58-97ef-0b3b2fd2f431","x-ha-device-classification":"10","x-ha-location":"city=xxxx, region=x, country=US, lat=30.378, lng=-97.640, asn=11427","x-ha-visitor-id":"1973a681-6655-4af5-8291-50a5af682eb6","x-ha-device-id":"123abs31-21sf32531-351d5s","x-hat-trace-id":"531685512685316553a35168","x-xxxx-xxxxx":"33bc357d-3ca1-42a5-8a1d-4ae737e2f69e"},"remoteAddress":"xx.xx.xx.xxx","remotePort":xxxxx},"tags":["info","rate-limiter"],"msg":"allowed=true reason=\"UNDER_RATE_LIMIT\" elapsed=1 visitorId=\"1973a681-6655-4af5-8291-sldvkdmvslkd6866\" deviceClassification=\"10\" userAgent=\"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36\" ip=\"xx.xx.xx.xxx\" city=\"xxx\" region=\"xx\" country=\"xx\" lat=\"xx\" lng=\"-97.640\" asn=\"11427\" application=\"xy-ab-bc\" method=\"post\" endpoint=\"/pdp/graphql/abc_bcd\" requestLimit=\"5\" requestLimitWindow=\"5000\" accessToken=\"1973a681-6655-4af5-8291-50a5af682eb6\" remainingRequests=\"4\"","v":1}