Splunk Search

How to extract fields from key/value, but separated with "|" symbol?

skender27
Contributor

Hi,

I have my syslog file writen as the following. I index these events in a syslog sourcetype.
What I need to extract are fields as PacketyType, PacketIndex, SkinTemperature, StepCounter, DELTADISTANCE and so on...

SocketTLMD:  --------------------   Client :1| PacketType : 6 | senderID : 1.0.0.5 | PacketIndex :26| BatteryVoltage :189| SkinTemperature :23.76| RSSI :78| StepCounter :1| FallCounter :0| AlmostFallCounter :0| MobilityIndex :42| userID : 132234 | CRC :202  | DISTANCE: -3825233.931520 | DELTADISTANCE: -0.000000 | DELTACOLARIES: -0.000000 | SPEED: -0.000000 | DELTASTEP: 0 --------------

Could you suggest a rex to extract only one of these fields?
Thanks,
Skender

0 Karma
1 Solution

skender27
Contributor

After I backed-up my .conf files, actually I am resolving it adding each extraction from the syslog:

EXTRACT-Distance = DISTANCE:\s(?<distance>\d+)
EXTRACT-Calories = Calories:\s(?<calories>\d+)
EXTRACT-PktType = PacketType :\s(?<pcktype>\d+)
EXTRACT-UserID = userID :\s(?<userTLMD>\d+)
EXTRACT-DeltaStep = DELTASTEP:\s(?<deltastep>\d+)\s\-
EXTRACT-DeltaDistance = DELTADISTANCE:\s(?<ddistance>\d+)
EXTRACT-DeltaCalories = DELTACOLARIES:\s(?<dcalories>\d+)
EXTRACT-Speed = SPEED:\s(?<speed>\d+)

View solution in original post

0 Karma

skender27
Contributor

After I backed-up my .conf files, actually I am resolving it adding each extraction from the syslog:

EXTRACT-Distance = DISTANCE:\s(?<distance>\d+)
EXTRACT-Calories = Calories:\s(?<calories>\d+)
EXTRACT-PktType = PacketType :\s(?<pcktype>\d+)
EXTRACT-UserID = userID :\s(?<userTLMD>\d+)
EXTRACT-DeltaStep = DELTASTEP:\s(?<deltastep>\d+)\s\-
EXTRACT-DeltaDistance = DELTADISTANCE:\s(?<ddistance>\d+)
EXTRACT-DeltaCalories = DELTACOLARIES:\s(?<dcalories>\d+)
EXTRACT-Speed = SPEED:\s(?<speed>\d+)
0 Karma

skender27
Contributor

it is better to use this syntax in order to extract integer numbers and not just digits:
(for example)

EXTRACT-Speed = SPEED:\s(?<speed>\d+\.\d+)
0 Karma

woodcock
Esteemed Legend

Be aware that this is 8x more expensive than my single-pass solution (which I tested and does work, after I re-edited it to remove a couple of mistakes).

0 Karma

skender27
Contributor

Thanks Woodcock,

I do not know why it is not working your settings in .conf files.
Probably because the events itself are not clean, not exactly as JSON input I put in the example, there are also other info, open and close of an external database, not so ordinary in the log I receive...

Anyway, thanks again,
Skender

0 Karma

woodcock
Esteemed Legend

Try this:

props.conf:

[YourSourcetypeHere]
REPORT-kvps = colon_pipe_kvps

transforms.conf

[colon_pipe_kvps]
FORMAT = $1::$2
MV_ADD = 1
REGEX = ([^\s:\|]+)\s*:\s*([^:\|]*)(?:\s*\||$)

skender27
Contributor

Hi,

I added the stanzas you suggested in the .conf files, but it is still not working.
Splunk in some way recognizes the JSON input format and so I'd prefer to make use of this...

This is how I see in the events in Splunk:

<30>Oct 8 12:46:19 SocketTLMD: (27082) JSON input: {"Client":1,"PacketType":"6","senderID":"1.0.0.4","PacketIndex":121,"BatteryVoltage":184,"SkinTemperature":22.48,"RSSI":92,"StepCounter":13,"FallCounter":0,"AlmostFallCounter":0,"MobilityIndex":58,"userID":"020901","CRC":33}

0 Karma

woodcock
Esteemed Legend

OK, then you need to switch from pipes to commas so use this RegEx instead:

 REGEX = ([^\s:,]+)\s*:\s*([^:,]*)(?:\s*,|$)
0 Karma

woodcock
Esteemed Legend

There were problems with my RegEx but I fixed and updated my answer. Try it again.

0 Karma

woodcock
Esteemed Legend

There was also a mismatch in my KO name so copy EVERYTHING fresh.

0 Karma

MuS
SplunkTrust
SplunkTrust

Hi skender27,

one hint would be extract http://docs.splunk.com/Documentation/Splunk/6.3.0/SearchReference/Extract another hint can be this answer http://answers.splunk.com/answers/214487/can-i-extract-a-field-with-a-regexed-dynamic-field.html .. in your use case you would split the key value pair with the |

cheers, MuS

skender27
Contributor

Hi MuS,
Thanks a lot.

I'd prefer insert the extraction in props.conf like (example for the PacketType field):

EXTRACT-pcktype = PacketType :  (?P<pcktype>\d+)\s\|

The problem is that I cannot extratc all the fields I need, changing slightly the precedent regex it seems it is not enough...

I will try again!

Thanks anyway,
Skender

0 Karma

MuS
SplunkTrust
SplunkTrust

try this regex (\w+)[\s:\-\s]*([^|\s]+) .. thanks for the kudos 🙂

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...