topic Re: Show all possible values in Splunk Search

Show all possible values

weird_guy — Sat, 17 Aug 2024 16:18:40 GMT

Hello.

I have a lot of events. Each event contains similar string \"errorDetail\":\"possible_value\"

Please specify how to create new field \"errorDetail\" and stats all possible values? (There are more than 50 kinds of errorDetail)

For example:

\"errorDetail\":\"acctNumber\"
\"errorDetail\":\"Message Version higher"\
\"errorDetail\":\"email\"

Thank you.

Re: Show all possible values

ITWhisperer — Sat, 17 Aug 2024 16:25:22 GMT

This looks like JSON format data - if so, you should be extracting as JSON and using the JSON functions to manipulate the data. Please share your full event in raw format in a code block, anonymise your data as appropriate. This will enable volunteers to better guide you on a way forward.

Re: Show all possible values

weird_guy — Sat, 17 Aug 2024 22:33:49 GMT

Hello @ITWhisperer

Thank you for your response.

Here is the raw data:

{ "messageType": "Data", "status": "Error", "statusMessage": "invalid message fields, wrong message from ds:[{\"threeDSServerTransID\":\"123\",\"messageType\":\"Erro\",\"messageVersion\":\"2.2.0\",\"acsTransID\":\"123\",\"dsTransID\":\"123\",\"errorCode\":\"305\",\"errorComponent\":\"A\",\"errorDescription\":\"Transaction data not valid\",\"errorDetail\":\"No issuer found\",\"errorMessageType\":\"AReq\"}]; type[Erro] code[101] component[SERVER]" }

Re: Show all possible values

ITWhisperer — Sat, 17 Aug 2024 17:16:18 GMT

From your raw event you could do this

| spath | rex field=statusMessage "\[(?<ds_message>[^\]]+)" | spath input=ds_message | stats count by errorDetail

if you have already extracted statusMessage when the event was ingested, you can skip the first spath command

Re: Show all possible values

PickleRick — Sat, 17 Aug 2024 20:44:51 GMT

Ugh. That's a pretty example of ugly data. Technically your data is a json structure with a field containing a string. That string describes another json structure but from splunk's point of view it's just a string. That makes it very inconvenient and possibly inefficient to manipulate. It would be much better if you got this from your source as some more sane format.

Re: Show all possible values

weird_guy — Sun, 18 Aug 2024 09:12:50 GMT

Hello @ITWhisperer

Ive entered

INTERNAL_VALIDATION_FAILED| spath | rex field=statusMessage "\[(?<ds_message>[^\]]+)" | spath input=ds_message | stats count by errorDetail

And there is only "errorDetail\": + count of events without values.

Re: Show all possible values

ITWhisperer — Sun, 18 Aug 2024 09:06:49 GMT

Here is a runanywhere example using your original event data showing the solution working. If it is not working with your real data, this means that the sample you shared is not an accurate representation of your real data. Please share an updated accurate representation of your data.

Re: Show all possible values

weird_guy — Sun, 18 Aug 2024 09:15:24 GMT

Hello

Here is an updated accurate data.

Thank you.

3DS2 Server ARes Response: {"messageType":"ARes","status":"INTERNAL_VALIDATION_FAILED","statusMessage":"invalid message fields, wrong message from ds:[{\"threeDSServerTransID\":\"123\",\"messageType\":\"Erro\",\"messageVersion\":\"2.2.0\",\"acsTransID\":\"345\",\"dsTransID\":\"567\",\"errorCode\":\"305\",\"errorComponent\":\"A\",\"errorDescription\":\"Cardholder Account Number is not in a range belonging to Issuer\",\"errorDetail\":\"acctNumber\",\"errorMessageType\":\"AReq\"}]; type[Erro] code[101] component[SERVER]"}

Re: Show all possible values

ITWhisperer — Sun, 18 Aug 2024 10:02:54 GMT

Try this

| rex "Response: (?<response>\{.+\})" | spath input=response | rex field=statusMessage "ds:\[(?<ds_message>[^\]]+)" | spath input=ds_message | stats count by errorDetail

Re: Show all possible values

weird_guy — Sun, 18 Aug 2024 10:06:41 GMT

Magic. It works.

But small issue here. It shows \"errorDetail\":

Hmmm

Re: Show all possible values

ITWhisperer — Sun, 18 Aug 2024 10:09:46 GMT

Again, here is a runanywhere example with your sample data

| makeresults | eval _raw="3DS2 Server ARes Response: {\"messageType\":\"ARes\",\"status\":\"INTERNAL_VALIDATION_FAILED\",\"statusMessage\":\"invalid message fields, wrong message from ds:[{\\\"threeDSServerTransID\\\":\\\"123\\\",\\\"messageType\\\":\\\"Erro\\\",\\\"messageVersion\\\":\\\"2.2.0\\\",\\\"acsTransID\\\":\\\"345\\\",\\\"dsTransID\\\":\\\"567\\\",\\\"errorCode\\\":\\\"305\\\",\\\"errorComponent\\\":\\\"A\\\",\\\"errorDescription\\\":\\\"Cardholder Account Number is not in a range belonging to Issuer\\\",\\\"errorDetail\\\":\\\"acctNumber\\\",\\\"errorMessageType\\\":\\\"AReq\\\"}]; type[Erro] code[101] component[SERVER]\"}" | rex "Response: (?<response>\{.+\})" | spath input=response | rex field=statusMessage "ds:\[(?<ds_message>[^\]]+)" | spath input=ds_message | stats count by errorDetail

If it is not working for some of your real data, then your sample is not an accurate representation of said (failing) data.

Re: Show all possible values

yuanliu — Sun, 18 Aug 2024 20:06:52 GMT

There can be a small problem: the error message, or "invalid message fields, wrong message from ds" as prefaced in the raw message, is a JSON array. You want to handle that as an entity.

| rex "^[^{]+(?<response>.+)" | spath input=response | rename messageType as topMessageType ``` handle namespace conflict ``` | rex field=statusMessage "^[^\[]+(?<message_from_ds>[^\]]+\])" | spath input=message_from_ds path={} | mvexpand {} | spath input={} | dedup errorDetail | table errorDetail