About davidlang

davidlang · ‎04-10-2023

trying to do this at index time, not just search time

davidlang · ‎04-06-2023

sorry, my sample should have included quotes around "message" (perils of typing examples in rather than cut-n-paste actual samples) The original is fully valid json, but with one of the json fields being text that (once it's unescaped) is also valid json so, if you were to use the jq tool to parse the original jq .message would return the string {"foo": "bar"} which happens to also be valid json so you could then do a second pass like jq .message |jq .foo would return the string bar but jq .message.foo will fail as message is a string, not a structure. the actual contents of message are much more complex (a full nested json structure that has (almost) all the interesting fields that we care about) I was hoping that there was some way to define a sourcetype parser that would do the json parsing, and then allow me to specify to do the parsing again on the contents of a particular field (like I do with the jq example above)

davidlang · ‎04-06-2023

agreed, but when the data you are likely to be searching on can appear in multiple places in the raw message, being able to extract the field at index time so that at search time you are only retrieving the messages that have the search term in the right place can be a huge win. (I'm 'new member' with the account for my current company, but have been running Splunk since 2006, so very aware of the trade-offs, but thanks for raising the concern)

davidlang · ‎04-06-2023

not as elegant as I was hoping (as PickleRick notes, doing a sed on the entire message is risky, consider quotes embedded in a text field anywhere in the message, this would break the message)

davidlang · ‎04-05-2023

I believe that it's already doing a json extraction of the fields in the message, I just need to do a second extraction of the contents of one of those fields. If you were to ask for the contents of message, you would get well formatted json.

davidlang · ‎04-05-2023

I have found that heavy index time extraction speeds up searches compared to search time extraction (less data needs to be fetched due to the increased accuracy of the searches) I'd rather spend cpu as the data is arriving rather than as users are waiting for results (it significantly improves the perceived performance, even if it takes more total CPU, which I question)

davidlang · ‎04-05-2023

(edited to give a more accurate example) I have an input that is json, but then includes escaped json a much more complex version of the example below (many fields and nesting both outside of message and in the message string), so this isn't just a field extraction of a particular field, I need to tell splunk to extract the message string (removing the escaping) and then parse that as json { "message": "{\"foo\": \"bar\", \"baz\": {\"a\": \"b\", \"c\": \"d\"}}" } I can extract this at search time with rex and spath, but would prefer to do so at index time. parsing this message with jq .message -r |jq . gives: { "foo": "bar", "baz": { "a": "b", "c": "d" } } what I ideally want is to have it look ike: { "message": { "foo": "bar", "baz": { "a": "b", "c": "d" } } }

Posts	13
Solutions	0
Karma Given	0
Karma Received	0
Member Since	‎04-05-2023

Online Status	Offline
Date Last Visited	‎04-12-2023 03:33 PM

How to parse escaped json at index time?

Re: How to parse escaped json at index time?

Re: How to parse escaped json at index time?

Re: How to parse escaped json at index time?

Re: How to parse escaped json at index time?

Re: How to parse escaped json at index time?

Re: How to parse escaped json at index time?

How to parse escaped json at index time?

Are you a member of the Splunk Community?