Knowledge Management

Field Extract Syslog Data: How to extract the Syslog Message and KV pairs?

spelunkingsplnk
Splunk Employee
Splunk Employee

I feel I'm getting lost in the sauce. I'm working on creating a props.conf for some syslog data on ingest (not search time) where the syslog message has it's standard syslog content, and then my message will start off with a statement followed by colon delimited fields with new lines. So like this message below. NOTE: the bold "normal" text changes depending on the message type, so this part is dynamic.

<priority>timestamp data1 data2 this is a normal message:
key:val
key1:val1
key2:val2

---
key_n:val_n

 

So I want to parse the the first line and pull different values from the syslog message, and then after that just use a delimiter so I don't have to specify each field (because there are a lot of fields, up to 50 different key:value lines).

First, not sure how to specify to Splunk: parse line 1 one way, and then use a delimiter on every other line. I'm sure there is a way?

I've looked into attributes for structured data here. I want to treat the first line almost like a header (different from the rest of the log), but not like the FIELD_HEADER properties as this isn't a delimited header I'm attempting to extract (like csv data).

How can I parse just the first line of my syslog (probably with some regex to grab everything appropriately), and then for the rest of my content use the delimiter? Maybe I could use FIELD_DELIMITER=: ? Additionally, I'm thinking I might have to use the transforms DELIM property here. I'm thinking something like this: DELIMS = "\r\n", ":"

Labels (1)
0 Karma
1 Solution

spelunkingsplnk
Splunk Employee
Splunk Employee

Figured it out with the help of this post: Using DELIMS to extract FIX data.

I created a props to extract fields on search-time. This allows me to extract all the fields on my first line of data.

props.conf

[sourcetypeInfo]
EXTRACT-syslogmsg = ^<(?<priority>[\d]+)>(?<timestamp>[a-zA-Z]{3}\s\s?[\d]{1,2}\sthis is a\s(?<message_type>[a-zA-Z]+)\sseverity:[\r\n\s]+(?<key_value_list>.*)
REPORT-syslog_key_value_list = syslog_key_value_list

Within my EXTRACT, I want to grab all the field I care about, as well as all the key-value pairs in it's own field. To separate my key-value pairs, I use a group name called key_value_list (located at the very end of my EXTRACT-syslogmsg) which I will use in my transforms.

transforms.conf

[syslog_key_value_list]
SOURCE_KEY = key_value_list
DELIMS = "\r\n", ":"

This is how it all comes together. In my transforms, I use the SOURCE_KEY to inform Splunk what Key (aka field) will be used. At this point, we can simply use the DELIMS property.

View solution in original post

0 Karma

spelunkingsplnk
Splunk Employee
Splunk Employee

Figured it out with the help of this post: Using DELIMS to extract FIX data.

I created a props to extract fields on search-time. This allows me to extract all the fields on my first line of data.

props.conf

[sourcetypeInfo]
EXTRACT-syslogmsg = ^<(?<priority>[\d]+)>(?<timestamp>[a-zA-Z]{3}\s\s?[\d]{1,2}\sthis is a\s(?<message_type>[a-zA-Z]+)\sseverity:[\r\n\s]+(?<key_value_list>.*)
REPORT-syslog_key_value_list = syslog_key_value_list

Within my EXTRACT, I want to grab all the field I care about, as well as all the key-value pairs in it's own field. To separate my key-value pairs, I use a group name called key_value_list (located at the very end of my EXTRACT-syslogmsg) which I will use in my transforms.

transforms.conf

[syslog_key_value_list]
SOURCE_KEY = key_value_list
DELIMS = "\r\n", ":"

This is how it all comes together. In my transforms, I use the SOURCE_KEY to inform Splunk what Key (aka field) will be used. At this point, we can simply use the DELIMS property.

0 Karma
Get Updates on the Splunk Community!

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...

DevSecOps: Why You Should Care and How To Get Started

 WATCH NOW In this Tech Talk we will talk about what people mean by DevSecOps and deep dive into the different ...

Introducing Ingest Actions: Filter, Mask, Route, Repeat

WATCH NOW Ingest Actions (IA) is the best new way to easily filter, mask and route your data in Splunk® ...