Splunk Search

How to remove path from spath field names

j8lp
Explorer

Hello,

So I love the spath command. With just one call, it will automatically extract and make searchable each and every field from each JSON log entries.

The only problem is that the spath command names each discovered field with that field's full path. This is a problem when trying to match fields across logs with different structures. For example, calling spth on the two log entries below will produce two different fields called "Request.Header.MessageID" and "Response.Header.MessageID"

{"Request":  {"Header":  {"MessageID":1234}}}
{"Response":  {"Header":  {"MessageID":1234}}}'

I actually am not going to know the exact message path or structure ahead of time. It could be Request.Header.MessageID, Request.MessageID, or even Request.Body.MessageID .

I'm looking for something that will just recognize that all of the fields that end with "MessageID" are referring to the same thing.

Is there a way I can do that? Does anyone know how I can remove the full path from the spath field names?

0 Karma
1 Solution

MuS
SplunkTrust
SplunkTrust

Hi j8lp,

you can use eval and coalesce in your search:

your base search here | spath | eval MessageID=coalesce("Request.Header.MessageID", "Request.MessageID", "Request.Body.MessageID"

This will use either of the three possible MessageID fields.

Hope this helps ...

cheers, MuS

View solution in original post

0 Karma

MuS
SplunkTrust
SplunkTrust

Hi j8lp,

you can use eval and coalesce in your search:

your base search here | spath | eval MessageID=coalesce("Request.Header.MessageID", "Request.MessageID", "Request.Body.MessageID"

This will use either of the three possible MessageID fields.

Hope this helps ...

cheers, MuS

View solution in original post

0 Karma

j8lp
Explorer

Sorry if my question isn't clear, but I'm actually not going to know all the possible paths. Is there a way to get coalesce everything that ends with ".MessageID" into a single field?

0 Karma

MuS
SplunkTrust
SplunkTrust

Okay, in this case I'm not sure if you would need spath at all or if you're better off using a rex :

 your base search here | rex max_match=0 "[Mm]essage[IiDd]+":(?<MessageID>[^}]+)" | ...

tested on regex101.con and with pcregextest in Splunk with this examples:

{"Request":  {"Header":  {"MessageID":1234}}}
{"Response":  {"Header":  {"MessageID":1234}}}'
{"Response":  {"Header":  {"messageID":1234}}}'
{"Response":  {"Header":  {"messageiD":1234}}}'
{"Response":  {"Header":  {"MessageId":1234}}}'

Just in case MessageID could also contain lower case m, i and d 😉

cheers, MuS

0 Karma

j8lp
Explorer

I actually am not going to know the exact path. I updated the question.

The idea could work if it was

rename "*.MessageID" AS MessageID

But Splunk doesn't seem to like wildcards in this command.

0 Karma

bimord
Path Finder

This would work if you did it without the quotes

e.g. | rename *.MessageID as MessageID

Another beneficial use of this function would also work in the opposite direction where you want to remove all prefixes from all fields:

| rename Request.Header.* as * 
| rename Response.Header.* as *

0 Karma

MuS
SplunkTrust
SplunkTrust

Updated the answer as well 😉

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!