Splunk Search
Highlighted

With regular expression how to auto extract JSON elements?

Communicator

I've created a Field Transform that attempts to extract all JSON key-value pairs, via the following regex:

(?:\"|\')([^"])(?:\"|\')(?=:)(?::\s)(?:\")?(true|false|[0-9a-zA-Z()\@:\,\/!+-.\$\ \\']*)(?:\")?

It's extracting ALL Json Key Value Pairs, except for Array's.

I'm okay not capturing arrays for now.

The problem I'm having is due to the one-size-fits-all approach of this RegEx, I need to include Comma's within the value matching for some of our error logging, however, that's resulting in the comma being captured after non-quoted numerical fields, as shown here:
Here's a screenshot showing the capture matches

Without the \, in the second capture group, I can't get the entire 'About' message, which includes a comma.

With it, I pick up the comma's on non-quoted numerical fields.

I haven't given up, but thought I'd crowd source an answer if possible because I'm a couple hours deep in this now and thought maybe someone knows what's missing.

Note: We can't use the KV_Mode Json Auto-extractions because JSON data is embedded within other log data in unexpected places, so this is a simple "catch all" match we apply to a handful of sourcetypes.

Thanks for your help! Feel free to head to the URL in that image to play with the expression directly.

Tags (2)
0 Karma
Highlighted

Re: With regular expression how to auto extract JSON elements?

Communicator

I think I got it with it in this version:

Version 5

Time to target those Arrays...

0 Karma
Highlighted

Re: With regular expression how to auto extract JSON elements?

Motivator

I appreciate your efforts! Can you provide some sample data to further assist you?

Also, on the side note, did you try using props.conf setting, INDEXED_EXTRACTIONS = JSON as described here.

0 Karma
Highlighted

Re: With regular expression how to auto extract JSON elements?

Communicator

@nittala_sirya,

Sample data has been provided in the regex101 mockup in the screenshot - I'm not permitted to paste links, so you'll need to type the url seen in the screenshots to pull up both my Regular Expression and Sample Data shown in the example.

I've resolved the issue, but what doesn't work is using INDEXEDEXTRACTIONS and/or KVMODE=JSON, as I tried to make clear in the post, because the events aren't purely JSON data; there's metadata before / after within the event that prevents Splunk's built-in automatic extraction methods from working.

My approach doesn't capture array's yet, but those aren't required for my use case; what matters is that once I applied my custom field-transform to a sourcetype, every single JSON key-value pair is extracted, regardless of where it sits within the event.

I don't understand why this isn't something offered out of the box.

0 Karma
Highlighted

Re: With regular expression how to auto extract JSON elements?

Motivator

Glad you've resolved it. Please post your solution in the "Answers" to help others with similar issue.

0 Karma
Highlighted

Re: With regular expression how to auto extract JSON elements?

Communicator

I'm new here. How do I tag my reply to my question as the correct response?

...or are you asking me to RE-post this as an Answers article?

0 Karma
Highlighted

Re: With regular expression how to auto extract JSON elements?

Motivator

Hover over to your comment (which you feel is the solution) and you will see a little gear wheel with More. Clicking on it will give you option to Convert to answer. This will move your comment to answer. Additionally, you can accept answer for your question. More info here.

0 Karma
Highlighted

Re: With regular expression how to auto extract JSON elements?

Communicator

It's not giving me that option

0 Karma
Highlighted

Re: With regular expression how to auto extract JSON elements?

Motivator

Ah, bummer. You have to answer your own question with the regex which worked for you.

On the side note: I am not sure if you've already seen this, but please take a look at [spath] splunk command. For starters, you can just append |spath at the end of your search query. More info, http://docs.splunk.com/Documentation/Splunk/7.1.2/SearchReference/Spath

0 Karma
Highlighted

Re: With regular expression how to auto extract JSON elements?

Communicator

spath's great for users familiar with SPL, and I've made great use of it, but I'm trying to get extractions for business users so they don't have to code.

Thanks for the tip!

0 Karma