How to extract fields from key/value, but separated with "|" symbol?

[skender27]

Hi,

I have my syslog file writen as the following. I index these events in a syslog sourcetype.
What I need to extract are fields as PacketyType, PacketIndex, SkinTemperature, StepCounter, DELTADISTANCE and so on...

SocketTLMD:  --------------------   Client :1| PacketType : 6 | senderID : 1.0.0.5 | PacketIndex :26| BatteryVoltage :189| SkinTemperature :23.76| RSSI :78| StepCounter :1| FallCounter :0| AlmostFallCounter :0| MobilityIndex :42| userID : 132234 | CRC :202  | DISTANCE: -3825233.931520 | DELTADISTANCE: -0.000000 | DELTACOLARIES: -0.000000 | SPEED: -0.000000 | DELTASTEP: 0 --------------

Could you suggest a rex to extract only one of these fields?
Thanks,
Skender

[skender27 · Accepted solution]

After I backed-up my .conf files, actually I am resolving it adding each extraction from the syslog:

EXTRACT-Distance = DISTANCE:\s(?<distance>\d+)
EXTRACT-Calories = Calories:\s(?<calories>\d+)
EXTRACT-PktType = PacketType :\s(?<pcktype>\d+)
EXTRACT-UserID = userID :\s(?<userTLMD>\d+)
EXTRACT-DeltaStep = DELTASTEP:\s(?<deltastep>\d+)\s\-
EXTRACT-DeltaDistance = DELTADISTANCE:\s(?<ddistance>\d+)
EXTRACT-DeltaCalories = DELTACOLARIES:\s(?<dcalories>\d+)
EXTRACT-Speed = SPEED:\s(?<speed>\d+)

[skender27]

it is better to use this syntax in order to extract integer numbers and not just digits:
(for example)

EXTRACT-Speed = SPEED:\s(?<speed>\d+\.\d+)

[woodcock]

Be aware that this is 8x more expensive than my single-pass solution (which I tested and does work, after I re-edited it to remove a couple of mistakes).

[skender27]

Thanks Woodcock,

I do not know why it is not working your settings in .conf files.
Probably because the events itself are not clean, not exactly as JSON input I put in the example, there are also other info, open and close of an external database, not so ordinary in the log I receive...

Anyway, thanks again,
Skender

[woodcock]

Try this:

props.conf:

[YourSourcetypeHere]
REPORT-kvps = colon_pipe_kvps

transforms.conf

[colon_pipe_kvps]
FORMAT = $1::$2
MV_ADD = 1
REGEX = ([^\s:\|]+)\s*:\s*([^:\|]*)(?:\s*\||$)

[skender27]

Hi,

I added the stanzas you suggested in the .conf files, but it is still not working.
Splunk in some way recognizes the JSON input format and so I'd prefer to make use of this...

This is how I see in the events in Splunk:

<30>Oct 8 12:46:19 SocketTLMD: (27082) JSON input: {"Client":1,"PacketType":"6","senderID":"1.0.0.4","PacketIndex":121,"BatteryVoltage":184,"SkinTemperature":22.48,"RSSI":92,"StepCounter":13,"FallCounter":0,"AlmostFallCounter":0,"MobilityIndex":58,"userID":"020901","CRC":33}

[woodcock]

OK, then you need to switch from pipes to commas so use this RegEx instead:

 REGEX = ([^\s:,]+)\s*:\s*([^:,]*)(?:\s*,|$)

[woodcock]

There were problems with my RegEx but I fixed and updated my answer. Try it again.

[woodcock]

There was also a mismatch in my KO name so copy EVERYTHING fresh.

[MuS]

Hi skender27,

one hint would be extract http://docs.splunk.com/Documentation/Splunk/6.3.0/SearchReference/Extract another hint can be this answer http://answers.splunk.com/answers/214487/can-i-extract-a-field-with-a-regexed-dynamic-field.html .. in your use case you would split the key value pair with the |

cheers, MuS

[skender27]

Hi MuS,
Thanks a lot.

I'd prefer insert the extraction in props.conf like (example for the PacketType field):

EXTRACT-pcktype = PacketType :  (?P<pcktype>\d+)\s\|

The problem is that I cannot extratc all the fields I need, changing slightly the precedent regex it seems it is not enough...

I will try again!

Thanks anyway,
Skender

[MuS]

try this regex (\w+)[\s:\-\s]*([^|\s]+) .. thanks for the kudos 🙂