topic Re: How to extract fields from key/value, but separated with "|" symbol? in Splunk Search

How to extract fields from key/value, but separated with "|" symbol?

skender27 — Thu, 08 Oct 2015 08:37:35 GMT

Hi,

I have my syslog file writen as the following. I index these events in a syslog sourcetype.
What I need to extract are fields as PacketyType, PacketIndex, SkinTemperature, StepCounter, DELTADISTANCE and so on...

SocketTLMD:  --------------------   Client :1| PacketType : 6 | senderID : 1.0.0.5 | PacketIndex :26| BatteryVoltage :189| SkinTemperature :23.76| RSSI :78| StepCounter :1| FallCounter :0| AlmostFallCounter :0| MobilityIndex :42| userID : 132234 | CRC :202  | DISTANCE: -3825233.931520 | DELTADISTANCE: -0.000000 | DELTACOLARIES: -0.000000 | SPEED: -0.000000 | DELTASTEP: 0 --------------

Could you suggest a rex to extract only one of these fields?
Thanks,
Skender

Re: How to extract fields from key/value, but separated with "|" symbol?

MuS — Thu, 08 Oct 2015 08:45:48 GMT

Hi skender27,

one hint would be extract http://docs.splunk.com/Documentation/Splunk/6.3.0/SearchReference/Extract another hint can be this answer http://answers.splunk.com/answers/214487/can-i-extract-a-field-with-a-regexed-dynamic-field.html .. in your use case you would split the key value pair with the |

cheers, MuS

Re: How to extract fields from key/value, but separated with "|" symbol?

skender27 — Thu, 08 Oct 2015 08:59:31 GMT

Hi MuS,
Thanks a lot.

I'd prefer insert the extraction in props.conf like (example for the PacketType field):

EXTRACT-pcktype = PacketType :  (?P<pcktype>\d+)\s\|

The problem is that I cannot extratc all the fields I need, changing slightly the precedent regex it seems it is not enough...

I will try again!

Thanks anyway,
Skender

Re: How to extract fields from key/value, but separated with "|" symbol?

MuS — Thu, 08 Oct 2015 09:17:54 GMT

try this regex (\w+)[\s:\-\s]*([^|\s]+) .. thanks for the kudos 🙂

Re: How to extract fields from key/value, but separated with "|" symbol?

woodcock — Thu, 08 Oct 2015 11:38:52 GMT

Try this:

props.conf:

[YourSourcetypeHere]
REPORT-kvps = colon_pipe_kvps

transforms.conf

[colon_pipe_kvps]
FORMAT = $1::$2
MV_ADD = 1
REGEX = ([^\s:\|]+)\s*:\s*([^:\|]*)(?:\s*\||$)

Re: How to extract fields from key/value, but separated with "|" symbol?

skender27 — Thu, 08 Oct 2015 12:32:23 GMT

Hi,

I added the stanzas you suggested in the .conf files, but it is still not working.
Splunk in some way recognizes the JSON input format and so I'd prefer to make use of this...

This is how I see in the events in Splunk:

<30>Oct 8 12:46:19 SocketTLMD: (27082) JSON input: {"Client":1,"PacketType":"6","senderID":"1.0.0.4","PacketIndex":121,"BatteryVoltage":184,"SkinTemperature":22.48,"RSSI":92,"StepCounter":13,"FallCounter":0,"AlmostFallCounter":0,"MobilityIndex":58,"userID":"020901","CRC":33}

Re: How to extract fields from key/value, but separated with "|" symbol?

woodcock — Thu, 08 Oct 2015 13:29:56 GMT

There were problems with my RegEx but I fixed and updated my answer. Try it again.

Re: How to extract fields from key/value, but separated with "|" symbol?

woodcock — Thu, 08 Oct 2015 13:31:25 GMT

There was also a mismatch in my KO name so copy EVERYTHING fresh.

Re: How to extract fields from key/value, but separated with "|" symbol?

skender27 — Thu, 08 Oct 2015 16:09:16 GMT

After I backed-up my .conf files, actually I am resolving it adding each extraction from the syslog:

EXTRACT-Distance = DISTANCE:\s(?<distance>\d+)
EXTRACT-Calories = Calories:\s(?<calories>\d+)
EXTRACT-PktType = PacketType :\s(?<pcktype>\d+)
EXTRACT-UserID = userID :\s(?<userTLMD>\d+)
EXTRACT-DeltaStep = DELTASTEP:\s(?<deltastep>\d+)\s\-
EXTRACT-DeltaDistance = DELTADISTANCE:\s(?<ddistance>\d+)
EXTRACT-DeltaCalories = DELTACOLARIES:\s(?<dcalories>\d+)
EXTRACT-Speed = SPEED:\s(?<speed>\d+)

Re: How to extract fields from key/value, but separated with "|" symbol?

woodcock — Thu, 08 Oct 2015 16:16:58 GMT

Be aware that this is 8x more expensive than my single-pass solution (which I tested and does work, after I re-edited it to remove a couple of mistakes).

Re: How to extract fields from key/value, but separated with "|" symbol?

woodcock — Thu, 08 Oct 2015 16:18:38 GMT

OK, then you need to switch from pipes to commas so use this RegEx instead:

 REGEX = ([^\s:,]+)\s*:\s*([^:,]*)(?:\s*,|$)

Re: How to extract fields from key/value, but separated with "|" symbol?

skender27 — Fri, 09 Oct 2015 08:50:40 GMT

Thanks Woodcock,

I do not know why it is not working your settings in .conf files.
Probably because the events itself are not clean, not exactly as JSON input I put in the example, there are also other info, open and close of an external database, not so ordinary in the log I receive...

Anyway, thanks again,
Skender

Re: How to extract fields from key/value, but separated with "|" symbol?

skender27 — Mon, 12 Oct 2015 16:08:34 GMT

it is better to use this syntax in order to extract integer numbers and not just digits:
(for example)

EXTRACT-Speed = SPEED:\s(?<speed>\d+\.\d+)