topic Re: JSON log extraction at index time in Getting Data In

JSON log extraction at index time

saiynv — Wed, 30 Sep 2020 01:56:31 GMT

Hi, I am trying to extract a JSON log file at index time. The log structure has a nested key(key,value) pairs.
Like
field 1, field 2 field 3{field 4=value 4 field5=value5 .......}fieldn=valuen.
The nested field value pairs are not uniform
Below is the nested field I'm looking to extract at run time.

"msg":"allowed=true reason=\"UNDER_RATE_LIMIT\" elapsed=1 visitorId=\"18b5b2c1-2201-4dbc-9f6f-9c7aa1941072\" deviceClassification=\"10\" userAgent=\"Mozilla/5.0 (X11; CrOS x86_64 12105.100.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.144 Safari/537.36\" ip=\"xx.xx.xx.xxx\" city=\"sykesville\" region=\"MD\" country=\"US\" lat=\"39.393\" lng=\"-76.931\" asn=\"7922\" application=\"xx-xx\" method=\"post\" endpoint=\"/pdp/graphql/" requestLimit=\"5\" requestLimitWindow=\"5000\" accessToken=\"18b5b2c1-2201-4dbc-9f6f-9c7aa1941072\" remainingRequests=\"4\"","v":1

I've tried with extract at searchtime but I need to get it on index time. I have tried with the regex (?P<_KEY_1>\w+)=(?P<_VAL_1>[^\s]+). But it gives partial results. Please help me out on this

Re: JSON log extraction at index time

jawaharas — Thu, 29 Aug 2019 04:50:29 GMT

Try below configuration in props.conf for index time extraction.

[SOURCETYPE_NAME]
INDEXED_EXTRACTIONS = json
category = Structured
pulldown_type = 1
detect_trailing_nulls = false
AUTO_KV_JSON = false
KV_MODE = none

Re: JSON log extraction at index time

saiynv — Thu, 29 Aug 2019 15:03:57 GMT

It still isn't dividing the whole event.

Re: JSON log extraction at index time

jawaharas — Thu, 29 Aug 2019 23:57:54 GMT

Sample JSON content (full content with all nested structure) and JSON key field that you look to extract will be helpful to assist you.

Re: JSON log extraction at index time

saiynv — Wed, 30 Sep 2020 02:01:03 GMT

Below is the complete raw event, which I am trying to extract

{"level":30,"time":1566851259952,"app":{"version":"master-133-0dcbc5f0f3"},"req":{"id":"33bc357d3ca1","method":"post","url":"/pdp/graphql","headers":{"x-ha-session-id":"200480ea-fcf1-4a58-97ef-0b3b2fd2f431","x-ha-device-classification":"10","x-ha-location":"city=xxxx, region=x, country=US, lat=30.378, lng=-97.640, asn=11427","x-ha-visitor-id":"1973a681-6655-4af5-8291-50a5af682eb6","x-ha-device-id":"123abs31-21sf32531-351d5s","x-hat-trace-id":"531685512685316553a35168","x-xxxx-xxxxx":"33bc357d-3ca1-42a5-8a1d-4ae737e2f69e"},"remoteAddress":"xx.xx.xx.xxx","remotePort":xxxxx},"tags":["info","rate-limiter"],"msg":"allowed=true reason=\"UNDER_RATE_LIMIT\" elapsed=1 visitorId=\"1973a681-6655-4af5-8291-sldvkdmvslkd6866\" deviceClassification=\"10\" userAgent=\"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36\" ip=\"xx.xx.xx.xxx\" city=\"xxx\" region=\"xx\" country=\"xx\" lat=\"xx\" lng=\"-97.640\" asn=\"11427\" application=\"xy-ab-bc\" method=\"post\" endpoint=\"/pdp/graphql/abc_bcd\" requestLimit=\"5\" requestLimitWindow=\"5000\" accessToken=\"1973a681-6655-4af5-8291-50a5af682eb6\" remainingRequests=\"4\"","v":1}

Re: JSON log extraction at index time

splunk_sv — Fri, 30 Aug 2019 07:00:05 GMT

@saiynv

At the time of indexing, you can specify the type of data you are trying to index which is JSON in this case,
So, that can only get you key-value pairs in which the "msg" will be a field with the rest of the string as its value.
And to use the properties present in that string as general key-value pairs you need to split that string and have to create new key-value pairs from that string in Splunk Query itself.