Knowledge Management

Field Extract Syslog Data: How to extract the Syslog Message and KV pairs?

spelunkingsplnk
Splunk Employee
Splunk Employee

I feel I'm getting lost in the sauce. I'm working on creating a props.conf for some syslog data on ingest (not search time) where the syslog message has it's standard syslog content, and then my message will start off with a statement followed by colon delimited fields with new lines. So like this message below. NOTE: the bold "normal" text changes depending on the message type, so this part is dynamic.

<priority>timestamp data1 data2 this is a normal message:
key:val
key1:val1
key2:val2

---
key_n:val_n

 

So I want to parse the the first line and pull different values from the syslog message, and then after that just use a delimiter so I don't have to specify each field (because there are a lot of fields, up to 50 different key:value lines).

First, not sure how to specify to Splunk: parse line 1 one way, and then use a delimiter on every other line. I'm sure there is a way?

I've looked into attributes for structured data here. I want to treat the first line almost like a header (different from the rest of the log), but not like the FIELD_HEADER properties as this isn't a delimited header I'm attempting to extract (like csv data).

How can I parse just the first line of my syslog (probably with some regex to grab everything appropriately), and then for the rest of my content use the delimiter? Maybe I could use FIELD_DELIMITER=: ? Additionally, I'm thinking I might have to use the transforms DELIM property here. I'm thinking something like this: DELIMS = "\r\n", ":"

Labels (1)
0 Karma
1 Solution

spelunkingsplnk
Splunk Employee
Splunk Employee

Figured it out with the help of this post: Using DELIMS to extract FIX data.

I created a props to extract fields on search-time. This allows me to extract all the fields on my first line of data.

props.conf

[sourcetypeInfo]
EXTRACT-syslogmsg = ^<(?<priority>[\d]+)>(?<timestamp>[a-zA-Z]{3}\s\s?[\d]{1,2}\sthis is a\s(?<message_type>[a-zA-Z]+)\sseverity:[\r\n\s]+(?<key_value_list>.*)
REPORT-syslog_key_value_list = syslog_key_value_list

Within my EXTRACT, I want to grab all the field I care about, as well as all the key-value pairs in it's own field. To separate my key-value pairs, I use a group name called key_value_list (located at the very end of my EXTRACT-syslogmsg) which I will use in my transforms.

transforms.conf

[syslog_key_value_list]
SOURCE_KEY = key_value_list
DELIMS = "\r\n", ":"

This is how it all comes together. In my transforms, I use the SOURCE_KEY to inform Splunk what Key (aka field) will be used. At this point, we can simply use the DELIMS property.

View solution in original post

0 Karma

spelunkingsplnk
Splunk Employee
Splunk Employee

Figured it out with the help of this post: Using DELIMS to extract FIX data.

I created a props to extract fields on search-time. This allows me to extract all the fields on my first line of data.

props.conf

[sourcetypeInfo]
EXTRACT-syslogmsg = ^<(?<priority>[\d]+)>(?<timestamp>[a-zA-Z]{3}\s\s?[\d]{1,2}\sthis is a\s(?<message_type>[a-zA-Z]+)\sseverity:[\r\n\s]+(?<key_value_list>.*)
REPORT-syslog_key_value_list = syslog_key_value_list

Within my EXTRACT, I want to grab all the field I care about, as well as all the key-value pairs in it's own field. To separate my key-value pairs, I use a group name called key_value_list (located at the very end of my EXTRACT-syslogmsg) which I will use in my transforms.

transforms.conf

[syslog_key_value_list]
SOURCE_KEY = key_value_list
DELIMS = "\r\n", ":"

This is how it all comes together. In my transforms, I use the SOURCE_KEY to inform Splunk what Key (aka field) will be used. At this point, we can simply use the DELIMS property.

0 Karma
Get Updates on the Splunk Community!

What’s new on Splunk Lantern in August

This month’s Splunk Lantern update gives you the low-down on all of the articles we’ve published over the past ...

Welcome to the Future of Data Search & Exploration

You have more data coming at you than ever before. Over the next five years, the total amount of digital data ...

This Week's Community Digest - Splunk Community Happenings [8.3.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...