topic Re: How to remove path from spath field names in Splunk Search

How to remove path from spath field names

j8lp — Mon, 09 Nov 2015 01:24:44 GMT

Hello,

So I love the spath command. With just one call, it will automatically extract and make searchable each and every field from each JSON log entries.

The only problem is that the spath command names each discovered field with that field's full path. This is a problem when trying to match fields across logs with different structures. For example, calling spth on the two log entries below will produce two different fields called "Request.Header.MessageID" and "Response.Header.MessageID"

{"Request":  {"Header":  {"MessageID":1234}}}
{"Response":  {"Header":  {"MessageID":1234}}}'

I actually am not going to know the exact message path or structure ahead of time. It could be Request.Header.MessageID, Request.MessageID, or even Request.Body.MessageID .

I'm looking for something that will just recognize that all of the fields that end with "MessageID" are referring to the same thing.

Is there a way I can do that? Does anyone know how I can remove the full path from the spath field names?

Re: How to remove path from spath field names

MuS — Mon, 09 Nov 2015 02:43:44 GMT

Hi j8lp,

you can use eval and coalesce in your search:

your base search here | spath | eval MessageID=coalesce("Request.Header.MessageID", "Request.MessageID", "Request.Body.MessageID"

This will use either of the three possible MessageID fields.

Hope this helps ...

cheers, MuS

Re: How to remove path from spath field names

j8lp — Mon, 09 Nov 2015 06:13:45 GMT

I actually am not going to know the exact path. I updated the question.

The idea could work if it was

rename "*.MessageID" AS MessageID

But Splunk doesn't seem to like wildcards in this command.

Re: How to remove path from spath field names

MuS — Mon, 09 Nov 2015 07:46:40 GMT

Updated the answer as well 😉

Re: How to remove path from spath field names

j8lp — Tue, 10 Nov 2015 15:42:55 GMT

Sorry if my question isn't clear, but I'm actually not going to know all the possible paths. Is there a way to get coalesce everything that ends with ".MessageID" into a single field?

Re: How to remove path from spath field names

MuS — Tue, 10 Nov 2015 19:25:55 GMT

Okay, in this case I'm not sure if you would need spath at all or if you're better off using a rex :

 your base search here | rex max_match=0 "[Mm]essage[IiDd]+":(?<MessageID>[^}]+)" | ...

tested on regex101.con and with pcregextest in Splunk with this examples:

{"Request":  {"Header":  {"MessageID":1234}}}
{"Response":  {"Header":  {"MessageID":1234}}}'
{"Response":  {"Header":  {"messageID":1234}}}'
{"Response":  {"Header":  {"messageiD":1234}}}'
{"Response":  {"Header":  {"MessageId":1234}}}'

Just in case MessageID could also contain lower case m, i and d 😉

cheers, MuS

Re: How to remove path from spath field names

bimord — Wed, 06 Oct 2021 02:42:27 GMT

This would work if you did it without the quotes

e.g. | rename *.MessageID as MessageID

Another beneficial use of this function would also work in the opposite direction where you want to remove all prefixes from all fields:

| rename Request.Header.* as *
| rename Response.Header.* as *