Splunk Search

How to extract fields from key/value, but separated with "|" symbol?

Contributor

Hi,

I have my syslog file writen as the following. I index these events in a syslog sourcetype.
What I need to extract are fields as PacketyType, PacketIndex, SkinTemperature, StepCounter, DELTADISTANCE and so on...

SocketTLMD:  --------------------   Client :1| PacketType : 6 | senderID : 1.0.0.5 | PacketIndex :26| BatteryVoltage :189| SkinTemperature :23.76| RSSI :78| StepCounter :1| FallCounter :0| AlmostFallCounter :0| MobilityIndex :42| userID : 132234 | CRC :202  | DISTANCE: -3825233.931520 | DELTADISTANCE: -0.000000 | DELTACOLARIES: -0.000000 | SPEED: -0.000000 | DELTASTEP: 0 --------------

Could you suggest a rex to extract only one of these fields?
Thanks,
Skender

0 Karma
1 Solution

Contributor

After I backed-up my .conf files, actually I am resolving it adding each extraction from the syslog:

EXTRACT-Distance = DISTANCE:\s(?<distance>\d+)
EXTRACT-Calories = Calories:\s(?<calories>\d+)
EXTRACT-PktType = PacketType :\s(?<pcktype>\d+)
EXTRACT-UserID = userID :\s(?<userTLMD>\d+)
EXTRACT-DeltaStep = DELTASTEP:\s(?<deltastep>\d+)\s\-
EXTRACT-DeltaDistance = DELTADISTANCE:\s(?<ddistance>\d+)
EXTRACT-DeltaCalories = DELTACOLARIES:\s(?<dcalories>\d+)
EXTRACT-Speed = SPEED:\s(?<speed>\d+)

View solution in original post

0 Karma

Contributor

After I backed-up my .conf files, actually I am resolving it adding each extraction from the syslog:

EXTRACT-Distance = DISTANCE:\s(?<distance>\d+)
EXTRACT-Calories = Calories:\s(?<calories>\d+)
EXTRACT-PktType = PacketType :\s(?<pcktype>\d+)
EXTRACT-UserID = userID :\s(?<userTLMD>\d+)
EXTRACT-DeltaStep = DELTASTEP:\s(?<deltastep>\d+)\s\-
EXTRACT-DeltaDistance = DELTADISTANCE:\s(?<ddistance>\d+)
EXTRACT-DeltaCalories = DELTACOLARIES:\s(?<dcalories>\d+)
EXTRACT-Speed = SPEED:\s(?<speed>\d+)

View solution in original post

0 Karma

Contributor

it is better to use this syntax in order to extract integer numbers and not just digits:
(for example)

EXTRACT-Speed = SPEED:\s(?<speed>\d+\.\d+)
0 Karma

Esteemed Legend

Be aware that this is 8x more expensive than my single-pass solution (which I tested and does work, after I re-edited it to remove a couple of mistakes).

0 Karma

Contributor

Thanks Woodcock,

I do not know why it is not working your settings in .conf files.
Probably because the events itself are not clean, not exactly as JSON input I put in the example, there are also other info, open and close of an external database, not so ordinary in the log I receive...

Anyway, thanks again,
Skender

0 Karma

Esteemed Legend

Try this:

props.conf:

[YourSourcetypeHere]
REPORT-kvps = colon_pipe_kvps

transforms.conf

[colon_pipe_kvps]
FORMAT = $1::$2
MV_ADD = 1
REGEX = ([^\s:\|]+)\s*:\s*([^:\|]*)(?:\s*\||$)

Contributor

Hi,

I added the stanzas you suggested in the .conf files, but it is still not working.
Splunk in some way recognizes the JSON input format and so I'd prefer to make use of this...

This is how I see in the events in Splunk:

<30>Oct 8 12:46:19 SocketTLMD: (27082) JSON input: {"Client":1,"PacketType":"6","senderID":"1.0.0.4","PacketIndex":121,"BatteryVoltage":184,"SkinTemperature":22.48,"RSSI":92,"StepCounter":13,"FallCounter":0,"AlmostFallCounter":0,"MobilityIndex":58,"userID":"020901","CRC":33}

0 Karma

Esteemed Legend

OK, then you need to switch from pipes to commas so use this RegEx instead:

 REGEX = ([^\s:,]+)\s*:\s*([^:,]*)(?:\s*,|$)
0 Karma

Esteemed Legend

There were problems with my RegEx but I fixed and updated my answer. Try it again.

0 Karma

Esteemed Legend

There was also a mismatch in my KO name so copy EVERYTHING fresh.

0 Karma

SplunkTrust
SplunkTrust

Hi skender27,

one hint would be extract http://docs.splunk.com/Documentation/Splunk/6.3.0/SearchReference/Extract another hint can be this answer http://answers.splunk.com/answers/214487/can-i-extract-a-field-with-a-regexed-dynamic-field.html .. in your use case you would split the key value pair with the |

cheers, MuS

Contributor

Hi MuS,
Thanks a lot.

I'd prefer insert the extraction in props.conf like (example for the PacketType field):

EXTRACT-pcktype = PacketType :  (?P<pcktype>\d+)\s\|

The problem is that I cannot extratc all the fields I need, changing slightly the precedent regex it seems it is not enough...

I will try again!

Thanks anyway,
Skender

0 Karma

SplunkTrust
SplunkTrust

try this regex (\w+)[\s:\-\s]*([^|\s]+) .. thanks for the kudos 🙂

0 Karma