I've created a Field Transform that attempts to extract all JSON key-value pairs, via the following regex:
It's extracting ALL Json Key Value Pairs, except for Array's.
I'm okay not capturing arrays for now.
The problem I'm having is due to the one-size-fits-all approach of this RegEx, I need to include Comma's within the value matching for some of our error logging, however, that's resulting in the comma being captured after non-quoted numerical fields, as shown here:
Without the \, in the second capture group, I can't get the entire 'About' message, which includes a comma.
With it, I pick up the comma's on non-quoted numerical fields.
I haven't given up, but thought I'd crowd source an answer if possible because I'm a couple hours deep in this now and thought maybe someone knows what's missing.
Note: We can't use the KV_Mode Json Auto-extractions because JSON data is embedded within other log data in unexpected places, so this is a simple "catch all" match we apply to a handful of sourcetypes.
Thanks for your help! Feel free to head to the URL in that image to play with the expression directly.
Sample data has been provided in the regex101 mockup in the screenshot - I'm not permitted to paste links, so you'll need to type the url seen in the screenshots to pull up both my Regular Expression and Sample Data shown in the example.
I've resolved the issue, but what doesn't work is using INDEXEDEXTRACTIONS and/or KVMODE=JSON, as I tried to make clear in the post, because the events aren't purely JSON data; there's metadata before / after within the event that prevents Splunk's built-in automatic extraction methods from working.
My approach doesn't capture array's yet, but those aren't required for my use case; what matters is that once I applied my custom field-transform to a sourcetype, every single JSON key-value pair is extracted, regardless of where it sits within the event.
I don't understand why this isn't something offered out of the box.
Glad you've resolved it. Please post your solution in the "Answers" to help others with similar issue.
I'm new here. How do I tag my reply to my question as the correct response?
...or are you asking me to RE-post this as an Answers article?
Hover over to your comment (which you feel is the solution) and you will see a little gear wheel with
More. Clicking on it will give you option to
Convert to answer. This will move your comment to answer. Additionally, you can accept answer for your question. More info here.
Ah, bummer. You have to answer your own question with the regex which worked for you.
On the side note: I am not sure if you've already seen this, but please take a look at
[spath] splunk command. For starters, you can just append
|spath at the end of your search query. More info, http://docs.splunk.com/Documentation/Splunk/7.1.2/SearchReference/Spath
spath's great for users familiar with SPL, and I've made great use of it, but I'm trying to get extractions for business users so they don't have to code.
Thanks for the tip!