Multi key value combinations in a single line

tonniea — Wed, 12 Mar 2014 06:37:32 GMT

Hi,
I'm trying to perform some field extractions in a log containing availability reports of segments in our system similar to the one below:

<SSEL><ET>2011-07-02T12:11:29.676Z</ET><LSID>12000</LSID><SEID>1007</SEID><S>OUT-OF-SERVICE</S><LSID>12000</LSID><SEID>1010</SEID><S>OUT-OF-SERVICE</S></SSEL>
etc. followed by from 10 up to 200 combinations of <LSID><SEID><S> tags and ending in </SSEL>

The syntax of the logline is like:

<DateTime><Logical ID><Segment ID><Status>....<Logical ID><Segment ID><Status>

In human language: per logline we have 1 datetime and multiple status reports for parts of the system identified by <Logical ID><Segment ID>.

I'm looking to extract the combinations of LSID, SEID and S and have each combination extracted with the EventTime () that is present at the start
of the logline. The goal is to create an overview of the status (S) for combinations of LSID,SEID during the day.

I've tried multi-value extraction using props.conf and transforms.conf like below but that fails for 2 reasons:
- the time is only found once in the logline
- the relation between LSID, SEID and S is broken as Splunk extracts all LSID's in the logline individually. Same for all SEID's and all S's.

My props.conf and transforms.conf:

[OL]
pulldown_type = 1
REPORT-r1 = ol_logfile

[ol_logfile]
REGEX=(\d{4}-\d{2}-\d{2})T(\d{2}:\d{2}:\d{2}\.\d{3})(.*?)<LSID>(\d*?)\</LSID>.*?<SID>(\d*?)\</SID>.*?<S>(.*?)\</S>
FORMAT=date::$1,time::$2,garbage::$3,LSID::$4,SEID::$5,S::$6
MV_ADD=true

I'm thinking now of writing a Python preprocessor to deliver the logline to Splunk like this:

<ET>2011-07-02T12:11:29.676Z</ET><LSID>12000</LSID><SEID>1007</SEID><S>OUT-OF-SERVICE</S>
<ET>2011-07-02T12:11:29.676Z</ET><LSID>12000</LSID><SEID>1010</SEID><S>OUT-OF-SERVICE</S>
....

But I'd rather use built-in Splunk routines as the preprocessor adds complexity and maintenance.

What would be the most Splunkish way to import the data in Splunk for my report? I'm not happy with the preprocessor, but I also don't see how to extract
the multivalue part.

Any hints are greatly appreciated, thanks!

Re: Multi key value combinations in a single line

rakesh_498115 — Wed, 12 Mar 2014 07:27:09 GMT

Hi tonniea,

You can use the below query to get wat u want . change the index and sourtype accordingly.

index="indexname" sourcetype="samplename" | rex "ET>(?<ET>[^<]*)\<" | rex max_match=100  "LSID>(?<LSID>[^<]*)\<" | rex max_match=100  "SEID>(?<SEID>[^<]*)\<" | rex max_match=100  "S>(?<S>[^<]*)\<" | stats values(LSID) as LSID values(SEID) as SEID values(S) as S by ET

i am assuming 100 value per Event Time , you can configure to any no based on ur need.

Hope this Helps !

Re: Multi key value combinations in a single line

tonniea — Wed, 12 Mar 2014 15:50:41 GMT

Thanks for taking the time to answer! Your regex is certainly better, but the output I get is not what I'm looking for. I now get: ET, LSID, SEID and all and Statuses for all SEID's at that time.
With your solution I appear to lose the relation ET<->LSID<->SEID<->Status and that's what I need: the status of an SEID at a certain time.

topic Re: Multi key value combinations in a single line in Splunk Search

Multi key value combinations in a single line

Re: Multi key value combinations in a single line

Re: Multi key value combinations in a single line