Splunk Search
Highlighted

How to remove path from spath field names

Explorer

Hello,

So I love the spath command. With just one call, it will automatically extract and make searchable each and every field from each JSON log entries.

The only problem is that the spath command names each discovered field with that field's full path. This is a problem when trying to match fields across logs with different structures. For example, calling spth on the two log entries below will produce two different fields called "Request.Header.MessageID" and "Response.Header.MessageID"

{"Request":  {"Header":  {"MessageID":1234}}}
{"Response":  {"Header":  {"MessageID":1234}}}'

I actually am not going to know the exact message path or structure ahead of time. It could be Request.Header.MessageID, Request.MessageID, or even Request.Body.MessageID .

I'm looking for something that will just recognize that all of the fields that end with "MessageID" are referring to the same thing.

Is there a way I can do that? Does anyone know how I can remove the full path from the spath field names?

0 Karma
Highlighted

Re: How to remove path from spath field names

SplunkTrust
SplunkTrust

Hi j8lp,

you can use eval and coalesce in your search:

your base search here | spath | eval MessageID=coalesce("Request.Header.MessageID", "Request.MessageID", "Request.Body.MessageID"

This will use either of the three possible MessageID fields.

Hope this helps ...

cheers, MuS

View solution in original post

0 Karma
Highlighted

Re: How to remove path from spath field names

Explorer

I actually am not going to know the exact path. I updated the question.

The idea could work if it was

rename "*.MessageID" AS MessageID

But Splunk doesn't seem to like wildcards in this command.

0 Karma
Highlighted

Re: How to remove path from spath field names

SplunkTrust
SplunkTrust

Updated the answer as well 😉

0 Karma
Highlighted

Re: How to remove path from spath field names

Explorer

Sorry if my question isn't clear, but I'm actually not going to know all the possible paths. Is there a way to get coalesce everything that ends with ".MessageID" into a single field?

0 Karma
Highlighted

Re: How to remove path from spath field names

SplunkTrust
SplunkTrust

Okay, in this case I'm not sure if you would need spath at all or if you're better off using a rex :

 your base search here | rex max_match=0 "[Mm]essage[IiDd]+":(?<MessageID>[^}]+)" | ...

tested on regex101.con and with pcregextest in Splunk with this examples:

{"Request":  {"Header":  {"MessageID":1234}}}
{"Response":  {"Header":  {"MessageID":1234}}}'
{"Response":  {"Header":  {"messageID":1234}}}'
{"Response":  {"Header":  {"messageiD":1234}}}'
{"Response":  {"Header":  {"MessageId":1234}}}'

Just in case MessageID could also contain lower case m, i and d 😉

cheers, MuS

0 Karma