Splunk Search

Multi key value combinations in a single line

tonniea
Explorer

Hi,
I'm trying to perform some field extractions in a log containing availability reports of segments in our system similar to the one below:

<SSEL><ET>2011-07-02T12:11:29.676Z</ET><LSID>12000</LSID><SEID>1007</SEID><S>OUT-OF-SERVICE</S><LSID>12000</LSID><SEID>1010</SEID><S>OUT-OF-SERVICE</S></SSEL>
etc. followed by from 10 up to 200 combinations of <LSID><SEID><S> tags and ending in </SSEL>

The syntax of the logline is like:

<DateTime><Logical ID><Segment ID><Status>....<Logical ID><Segment ID><Status>

In human language: per logline we have 1 datetime and multiple status reports for parts of the system identified by <Logical ID><Segment ID>.

I'm looking to extract the combinations of LSID, SEID and S and have each combination extracted with the EventTime () that is present at the start
of the logline. The goal is to create an overview of the status (S) for combinations of LSID,SEID during the day.

I've tried multi-value extraction using props.conf and transforms.conf like below but that fails for 2 reasons:
- the time is only found once in the logline
- the relation between LSID, SEID and S is broken as Splunk extracts all LSID's in the logline individually. Same for all SEID's and all S's.

My props.conf and transforms.conf:

[OL]
pulldown_type = 1
REPORT-r1 = ol_logfile

[ol_logfile]
REGEX=(\d{4}-\d{2}-\d{2})T(\d{2}:\d{2}:\d{2}\.\d{3})(.*?)<LSID>(\d*?)\</LSID>.*?<SID>(\d*?)\</SID>.*?<S>(.*?)\</S>
FORMAT=date::$1,time::$2,garbage::$3,LSID::$4,SEID::$5,S::$6
MV_ADD=true

I'm thinking now of writing a Python preprocessor to deliver the logline to Splunk like this:

<ET>2011-07-02T12:11:29.676Z</ET><LSID>12000</LSID><SEID>1007</SEID><S>OUT-OF-SERVICE</S>
<ET>2011-07-02T12:11:29.676Z</ET><LSID>12000</LSID><SEID>1010</SEID><S>OUT-OF-SERVICE</S>
....

But I'd rather use built-in Splunk routines as the preprocessor adds complexity and maintenance.

What would be the most Splunkish way to import the data in Splunk for my report? I'm not happy with the preprocessor, but I also don't see how to extract
the multivalue part.

Any hints are greatly appreciated, thanks!

0 Karma
1 Solution

rakesh_498115
Motivator

Hi tonniea,

You can use the below query to get wat u want . change the index and sourtype accordingly.

index="indexname" sourcetype="samplename" | rex "ET>(?<ET>[^<]*)\<" | rex max_match=100  "LSID>(?<LSID>[^<]*)\<" | rex max_match=100  "SEID>(?<SEID>[^<]*)\<" | rex max_match=100  "S>(?<S>[^<]*)\<" | stats values(LSID) as LSID values(SEID) as SEID values(S) as S by ET

i am assuming 100 value per Event Time , you can configure to any no based on ur need.

Hope this Helps !

View solution in original post

rakesh_498115
Motivator

Hi tonniea,

You can use the below query to get wat u want . change the index and sourtype accordingly.

index="indexname" sourcetype="samplename" | rex "ET>(?<ET>[^<]*)\<" | rex max_match=100  "LSID>(?<LSID>[^<]*)\<" | rex max_match=100  "SEID>(?<SEID>[^<]*)\<" | rex max_match=100  "S>(?<S>[^<]*)\<" | stats values(LSID) as LSID values(SEID) as SEID values(S) as S by ET

i am assuming 100 value per Event Time , you can configure to any no based on ur need.

Hope this Helps !

tonniea
Explorer

Thanks for taking the time to answer! Your regex is certainly better, but the output I get is not what I'm looking for. I now get: ET, LSID, SEID and all and Statuses for all SEID's at that time.
With your solution I appear to lose the relation ET<->LSID<->SEID<->Status and that's what I need: the status of an SEID at a certain time.

Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...