Splunk Search

Multi key value combinations in a single line

tonniea
Explorer

Hi,
I'm trying to perform some field extractions in a log containing availability reports of segments in our system similar to the one below:

<SSEL><ET>2011-07-02T12:11:29.676Z</ET><LSID>12000</LSID><SEID>1007</SEID><S>OUT-OF-SERVICE</S><LSID>12000</LSID><SEID>1010</SEID><S>OUT-OF-SERVICE</S></SSEL>
etc. followed by from 10 up to 200 combinations of <LSID><SEID><S> tags and ending in </SSEL>

The syntax of the logline is like:

<DateTime><Logical ID><Segment ID><Status>....<Logical ID><Segment ID><Status>

In human language: per logline we have 1 datetime and multiple status reports for parts of the system identified by <Logical ID><Segment ID>.

I'm looking to extract the combinations of LSID, SEID and S and have each combination extracted with the EventTime () that is present at the start
of the logline. The goal is to create an overview of the status (S) for combinations of LSID,SEID during the day.

I've tried multi-value extraction using props.conf and transforms.conf like below but that fails for 2 reasons:
- the time is only found once in the logline
- the relation between LSID, SEID and S is broken as Splunk extracts all LSID's in the logline individually. Same for all SEID's and all S's.

My props.conf and transforms.conf:

[OL]
pulldown_type = 1
REPORT-r1 = ol_logfile

[ol_logfile]
REGEX=(\d{4}-\d{2}-\d{2})T(\d{2}:\d{2}:\d{2}\.\d{3})(.*?)<LSID>(\d*?)\</LSID>.*?<SID>(\d*?)\</SID>.*?<S>(.*?)\</S>
FORMAT=date::$1,time::$2,garbage::$3,LSID::$4,SEID::$5,S::$6
MV_ADD=true

I'm thinking now of writing a Python preprocessor to deliver the logline to Splunk like this:

<ET>2011-07-02T12:11:29.676Z</ET><LSID>12000</LSID><SEID>1007</SEID><S>OUT-OF-SERVICE</S>
<ET>2011-07-02T12:11:29.676Z</ET><LSID>12000</LSID><SEID>1010</SEID><S>OUT-OF-SERVICE</S>
....

But I'd rather use built-in Splunk routines as the preprocessor adds complexity and maintenance.

What would be the most Splunkish way to import the data in Splunk for my report? I'm not happy with the preprocessor, but I also don't see how to extract
the multivalue part.

Any hints are greatly appreciated, thanks!

0 Karma
1 Solution

rakesh_498115
Motivator

Hi tonniea,

You can use the below query to get wat u want . change the index and sourtype accordingly.

index="indexname" sourcetype="samplename" | rex "ET>(?<ET>[^<]*)\<" | rex max_match=100  "LSID>(?<LSID>[^<]*)\<" | rex max_match=100  "SEID>(?<SEID>[^<]*)\<" | rex max_match=100  "S>(?<S>[^<]*)\<" | stats values(LSID) as LSID values(SEID) as SEID values(S) as S by ET

i am assuming 100 value per Event Time , you can configure to any no based on ur need.

Hope this Helps !

View solution in original post

rakesh_498115
Motivator

Hi tonniea,

You can use the below query to get wat u want . change the index and sourtype accordingly.

index="indexname" sourcetype="samplename" | rex "ET>(?<ET>[^<]*)\<" | rex max_match=100  "LSID>(?<LSID>[^<]*)\<" | rex max_match=100  "SEID>(?<SEID>[^<]*)\<" | rex max_match=100  "S>(?<S>[^<]*)\<" | stats values(LSID) as LSID values(SEID) as SEID values(S) as S by ET

i am assuming 100 value per Event Time , you can configure to any no based on ur need.

Hope this Helps !

tonniea
Explorer

Thanks for taking the time to answer! Your regex is certainly better, but the output I get is not what I'm looking for. I now get: ET, LSID, SEID and all and Statuses for all SEID's at that time.
With your solution I appear to lose the relation ET<->LSID<->SEID<->Status and that's what I need: the status of an SEID at a certain time.

Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...