topic Re: RegEx in Splunk Search

RegEx

CPrimoR — Wed, 11 Dec 2024 19:58:41 GMT

I am trying to regex out eligible with the answer field true, when i do it in the regex builder this works

eligible\\":(?<eligibility_status>[^,]+)

but when i do it in Splunk with adding the additional backslash to escape the quotation the query runs but the field is not there.

Name":null,"Id":null,"WaypointId":null}},"Body":{"APIServiceCall":{"ResponseStatusCode":"200","ResponsePayload":"{\"eligibilityIndicator\":[{\"service\":\"Mobile\",\"eligible\":true,\"successReasonCodes\":[],\"failureReasonCodes\":[]}]}"}}}

Re: RegEx

isoutamo — Wed, 11 Dec 2024 19:04:01 GMT

have you check that your raw event is what you are thinking? As it seems to be a JSON, it probably contains some other characters what you aren’t expecting!

Open event and select from event action “show source”. From there you see what event contains and then you can modify your rex to match it.

r. Ismo

Re: RegEx

PickleRick — Wed, 11 Dec 2024 20:55:27 GMT

What exactly do you mean by "when I do it in Splunk"?

Re: RegEx

ITWhisperer — Wed, 11 Dec 2024 21:01:48 GMT

When rex'ing backslashes, you need to quadruple them

| rex "eligible\\\\\":(?<eligibility_status>[^,]+)"

Re: RegEx

yuanliu — Thu, 12 Dec 2024 06:17:24 GMT

Another important point: Your raw data is in JSON. Do not treat structured data as plain strings. In other words, instead of using regex, use proper JSON tools Splunk has.

When showing structured data, it is important to post a compliant structure. Let me reconstruct a compliant JSON out of the illustrated fragment before giving your a shortcut.

{"isthiscorrect": {"somekey": {"Name":null,"Id":null,"WaypointId":null}},"Body":{"APIServiceCall":{"ResponseStatusCode":"200","ResponsePayload":"{\"eligibilityIndicator\":[{\"service\":\"Mobile\",\"eligible\":true,\"successReasonCodes\":[],\"failureReasonCodes\":[]}]}"}}}

If your raw events resemble the above in structure, Splunk would have given you a field named Body.APIServiceCall.ResponsePayload. Your illustrated fragment contains this value for that field:

{"eligibilityIndicator":[{"service":"Mobile","eligible":true,"successReasonCodes":[],"failureReasonCodes":[]}]}

All you need to do is to use an appropriate tool extract from this. But before you do, note that eligibilityIndicator is an array. You most likely want to split the array into their own events.

Putting this chain together:

| spath input=Body.APIServiceCall.ResponsePayload path=eligibilityIndicator{} | mvexpand eligibilityIndicator{} | spath input=eligibilityIndicator{}

The field you are trying to extract is now called eligible.

Here is an emulation with your fragment as reconstructed above.

| makeresults | eval _raw = "{\"isthiscorrect\": {\"somekey\": {\"Name\":null,\"Id\":null,\"WaypointId\":null}},\"Body\":{\"APIServiceCall\":{\"ResponseStatusCode\":\"200\",\"ResponsePayload\":\"{\\\"eligibilityIndicator\\\":[{\\\"service\\\":\\\"Mobile\\\",\\\"eligible\\\":true,\\\"successReasonCodes\\\":[],\\\"failureReasonCodes\\\":[]}]}\"}}}" | spath ``` data emulation above ```

These are the three fields extracted from eligibilityIndicator{}

eligible	service	successReasonCodes{}
true	Mobile

Re: RegEx

PickleRick — Thu, 12 Dec 2024 08:09:03 GMT

While I wholeheartedly agree about the "don't fiddle with structured data using regexes" point, it's worth noting that spath is not feasible for search-time extractions on which you'd want to base your searches because spath has to parse whole event (or a whole given field) as json event and has no notion about fields before that so you don't have any condition like "spath(whatever)=some_value". In other words, while for "first-order" jsons you can do the normal initial search filtering based on field=value conditions, it won't work with more deeply embedded json structures (regardless of whether they are included as strings within an "outer" json or if they are simply a part of a syslog-headered event).

Splunk still has to process all events from the preceeding pipeline, push them through spath and only then you can filter the data further.

One possible way around it is to limit your processed data by limiting your data in the initial search by searching for the literal value term. It will not help much with fields of low cardinality and terms common across many fields (like in this case - true/false is not a very well-limiting search term) but in other cases when you're searching for a fairly unique term it can mean loads of speedup.

Re: RegEx

CPrimoR — Fri, 13 Dec 2024 20:30:29 GMT

Thanks this worked like a charm!