Field Extract Syslog Data: How to extract the Syslog Message and KV pairs?

spelunkingsplnk — Fri, 10 Jun 2022 05:40:14 GMT

I feel I'm getting lost in the sauce. I'm working on creating a props.conf for some syslog data on ingest (not search time) where the syslog message has it's standard syslog content, and then my message will start off with a statement followed by colon delimited fields with new lines. So like this message below. NOTE: the bold "normal" text changes depending on the message type, so this part is dynamic.

<priority>timestamp data1 data2 this is a normal message:
key:val
key1:val1
key2:val2

---
key_n:val_n

So I want to parse the the first line and pull different values from the syslog message, and then after that just use a delimiter so I don't have to specify each field (because there are a lot of fields, up to 50 different key:value lines).

First, not sure how to specify to Splunk: parse line 1 one way, and then use a delimiter on every other line. I'm sure there is a way?

I've looked into attributes for structured data here. I want to treat the first line almost like a header (different from the rest of the log), but not like the FIELD_HEADER properties as this isn't a delimited header I'm attempting to extract (like csv data).

How can I parse just the first line of my syslog (probably with some regex to grab everything appropriately), and then for the rest of my content use the delimiter? Maybe I could use FIELD_DELIMITER=: ? Additionally, I'm thinking I might have to use the transforms DELIM property here. I'm thinking something like this: DELIMS = "\r\n", ":"

Re: Field Extract Syslog Data: Extract the Syslog Message and KV pairs

spelunkingsplnk — Wed, 08 Jun 2022 20:04:49 GMT

Figured it out with the help of this post: Using DELIMS to extract FIX data.

I created a props to extract fields on search-time. This allows me to extract all the fields on my first line of data.

props.conf

[sourcetypeInfo]
EXTRACT-syslogmsg = ^<(?<priority>[\d]+)>(?<timestamp>[a-zA-Z]{3}\s\s?[\d]{1,2}\sthis is a\s(?<message_type>[a-zA-Z]+)\sseverity:[\r\n\s]+(?<key_value_list>.*)
REPORT-syslog_key_value_list = syslog_key_value_list

Within my EXTRACT, I want to grab all the field I care about, as well as all the key-value pairs in it's own field. To separate my key-value pairs, I use a group name called key_value_list (located at the very end of my EXTRACT-syslogmsg) which I will use in my transforms.

transforms.conf

[syslog_key_value_list]
SOURCE_KEY = key_value_list
DELIMS = "\r\n", ":"

This is how it all comes together. In my transforms, I use the SOURCE_KEY to inform Splunk what Key (aka field) will be used. At this point, we can simply use the DELIMS property.

topic Field Extract Syslog Data: How to extract the Syslog Message and KV pairs? in Knowledge Management

Field Extract Syslog Data: How to extract the Syslog Message and KV pairs?

Re: Field Extract Syslog Data: Extract the Syslog Message and KV pairs