How to remove path from spath field names

[j8lp]

Hello,

So I love the spath command. With just one call, it will automatically extract and make searchable each and every field from each JSON log entries.

The only problem is that the spath command names each discovered field with that field's full path. This is a problem when trying to match fields across logs with different structures. For example, calling spth on the two log entries below will produce two different fields called "Request.Header.MessageID" and "Response.Header.MessageID"

{"Request":  {"Header":  {"MessageID":1234}}}
{"Response":  {"Header":  {"MessageID":1234}}}'

I actually am not going to know the exact message path or structure ahead of time. It could be Request.Header.MessageID, Request.MessageID, or even Request.Body.MessageID .

I'm looking for something that will just recognize that all of the fields that end with "MessageID" are referring to the same thing.

Is there a way I can do that? Does anyone know how I can remove the full path from the spath field names?

[MuS · Accepted solution]

Hi j8lp,

you can use eval and coalesce in your search:

your base search here | spath | eval MessageID=coalesce("Request.Header.MessageID", "Request.MessageID", "Request.Body.MessageID"

This will use either of the three possible MessageID fields.

Hope this helps ...

cheers, MuS

[j8lp]

Sorry if my question isn't clear, but I'm actually not going to know all the possible paths. Is there a way to get coalesce everything that ends with ".MessageID" into a single field?

[MuS]

Okay, in this case I'm not sure if you would need spath at all or if you're better off using a rex :

 your base search here | rex max_match=0 "[Mm]essage[IiDd]+":(?<MessageID>[^}]+)" | ...

tested on regex101.con and with pcregextest in Splunk with this examples:

{"Request":  {"Header":  {"MessageID":1234}}}
{"Response":  {"Header":  {"MessageID":1234}}}'
{"Response":  {"Header":  {"messageID":1234}}}'
{"Response":  {"Header":  {"messageiD":1234}}}'
{"Response":  {"Header":  {"MessageId":1234}}}'

Just in case MessageID could also contain lower case m, i and d 😉

cheers, MuS

[j8lp]

I actually am not going to know the exact path. I updated the question.

The idea could work if it was

rename "*.MessageID" AS MessageID

But Splunk doesn't seem to like wildcards in this command.

[bimord]

This would work if you did it without the quotes

e.g. | rename *.MessageID as MessageID

Another beneficial use of this function would also work in the opposite direction where you want to remove all prefixes from all fields:

| rename Request.Header.* as * 
| rename Response.Header.* as *

[MuS]

Updated the answer as well 😉