topic Re: How to extract fields delimited by semicolons and the last field with variable values from my sample data? in Splunk Search

How to extract fields delimited by semicolons and the last field with variable values from my sample data?

smudge797 — Mon, 19 Oct 2015 08:35:16 GMT

Using Splunk Enterpise 6.2, I'm trying to get the fields extracted using search-time props.conf / transforms.conf and the issue is a variable value in the last field. Can anyone tell me what the settings would be ensure these values are taken care of? The message that logs are delimited by semicolon, but unfortunately, the different messages have different amount of fields and almost constant in size 40 characters except the last field that is variable, but also not more than 300 characters.

Is there a way to take care of this situation, at search-time?

Log sample:

[1445004534] SERVICE ALERT: madmsc03j1;Status_Host_ID-0104;OK;HARD;2;PING OK - Packet loss = 0%, RTA = 47.89 ms
[1445004568] SERVICE ALERT: de2secimpmx001;Resource_System_Load_ID-0108;OK;HARD;1;4 CPU, average load 70.8% < 75% : OK
[1445004576] SERVICE ALERT: pt1secimpmx001;Resource_System_Load_ID-0108;CRITICAL;HARD;1;2 CPU, average load 91.0% > 90% : CRITICAL
[1445004576] SERVICE FLAPPING ALERT: pt1secimpmx001;Resource_System_Load_ID-0108;STARTED; Service appears to have started flapping (12.4% change >= 10.0% threshold)
[1445004586] SERVICE ALERT: it1sec02mibi-002;Resource_Memory_ID-0011;OK;SOFT;2;OK - Physical memory: 98.0% used (7.5GiB/7.7GiB)
[1445004592] SERVICE ALERT: it1sec02mibi-002;Status_Interfaces_ID-0017;OK;SOFT;2;OK - No valid historical dataset... <a href="https://mydomain.com/interfacetable_v3t/tables/it1sec02mibiQ2D002--Interfacetable.html" target="_self">[details]</a>
[1445004634] SERVICE FLAPPING ALERT: it1sec04mibi-001;Resource_System_Load_ID-0109;STOPPED; Service appears to have stopped flapping (0.0% change < 1.0% threshold)
[1445004673] SERVICE ALERT: nl2secsou3d001;Status_Process_SS_Min2_ID-0101;UNKNOWN;SOFT;1;ERROR: Alarm signal (Nagios time-out)

Currently I have props.conf:

TIME_FORMAT=%s
TIME_PREFIX=\[
TZ=GMT
MAX_TIMESTAMP_LOOKAHEAD=10
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
LINE_BREAKER=([\r\n]+)
TRUNCATE=10000

Cheers!

Re: How to extract fields delimited by semicolons and the last field with variable values from my sample data?

santiagoaloi — Mon, 19 Oct 2015 11:11:38 GMT

Hi!

I need to understand what is that you want to achieve. If I understood you correctly , you want to extract all the key fields that are separated by the semicolon, and then the values at the end of the event such as "4 CPU, average load 70.8% < 75% : OK".
I will help you to do this, although I need to understand better what you need 🙂

Re: How to extract fields delimited by semicolons and the last field with variable values from my sample data?

smudge797 — Mon, 19 Oct 2015 15:30:05 GMT

Yes Im trying to extract the key fields separated by ; and the values. But the problem is the last field is variable value but no more than 300 characters. Hope that helps clarify 🙂

Re: How to extract fields delimited by semicolons and the last field with variable values from my sample data?

santiagoaloi — Tue, 20 Oct 2015 09:02:16 GMT

So! I've done most of the field extractions. You cannot combine delimited data extractions with regex, so is either one or the other.

Regex is a little tricky to tune the way you want it to work, the regex based field extractions in Splunk not always works as we expect, so most of the times www.regex101.com becomes your best friend.

You will have to change the names of the fields because since I don't know where this dataset comes from I couldn't guess much 🙂

EXTRACT-description = \s-|\s(?[^;]+$) is the one you needed help with.

props.conf

TIME_FORMAT=%s
TIME_PREFIX=\[
TZ=GMT
MAX_TIMESTAMP_LOOKAHEAD=10
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
LINE_BREAKER=([\r\n]+)
TRUNCATE=10000

[yoursourcetype]
EXTRACT-service_type    = ^\[\d+\]\sSERVICE\s(?<service_type>\w+)
EXTRACT-service         = ^[^:\n]*:\s+(?P<service>\w+)
EXTRACT-objet_id        = _ID-(?<objet_id>\d+)
EXTRACT-status          = ^(?:[^;\n]*;){2}(?P<status>\w+)
EXTRACT-description     = \s\-|\s(?<description>[^;]+$)
EXTRACT-layer           = ^(?:[^;\n]*;){3}(?P<layer>\w+)
EXTRACT-other_status    = ^\[\d+\]\s+\w+\s+\w+:\s+\w+\d+\w+\d+\w+\d+;\w+_\w+_\w+_\w+\d+_\w+\-\d+;\w+;\w+;\d+;(?P<other_status>\w+)

Re: How to extract fields delimited by semicolons and the last field with variable values from my sample data?

smudge797 — Thu, 29 Oct 2015 06:20:24 GMT

So this is all in props.conf at search time on the search head, correct? I will try it today.

Thanks!