topic How to extract just one field from a log when there are multiple that carry the same attribute name? - Regular Expression in Splunk Search

How to extract just one field from a log when there are multiple that carry the same attribute name? - Regular Expression

evang_26 — Mon, 28 Sep 2020 18:57:58 GMT

Hi all,

I am filtering some logs came from Nessus in order to identify vulnerable machines based on their OS, and the issue I have is when a host's OS is not adequately identified resulting in many "os" fields. An example is the below:

start_time="Mon Feb 16 03:56:07 2015"
end_time="Mon Feb 16 03:57:42 2015"
os="Microsoft Windows 2000" os="Microsoft Windows XP for Embedded Systems" os="Microsoft
Windows XP"

The query that I created for that (which only works sufficiently when 1 OS is found) is the following:

sourcetype=nessus severity!=informational  | rex "start_time=\"(?<start>.*)\"\send.*\s\sos=\"(?<OS>.*)\""

What I would like ideally to do, is to just find a way to filter out the " (double quote" symbol from within the extracted field. This is because apart from Windows machines, there are other printers and access points that are interpreted as many other mixed OSs.

So, it should be something like this:

sourcetype=nessus severity!=informational  | rex "start_time=\"(?<start>.*)\"\send.*\s\sos=\"(?<OS>.*[^\"])\""

but it doesn't work.

Any thoughts?

Regards,
Evang

Update:

Hi MuS,

Okay, here we are.

I guess I haven't stated my problem correctly. I do not want to remove the double quotes, actually, I want to only keep the first occurence of OS field in the rare cases that more than one appears!

Here is what I managed to do with sed, but I am not there quite yet.

sourcetype=nessus severity!=informational earliest=-5w@w1 latest=now|rex field=os mode=sed "s/.*\(os=\"[^\"]*\"\).*$/\1/g" | rex ".*os=\"(?P<OS>.*)\"\s.*\""

Any suggestions? 🙂

Regards,
Evang

Re: How to extract just one field from a log when there are multiple that carry the same attribute name? - Regular Expression

richgalloway — Mon, 16 Feb 2015 18:17:25 GMT

You were so close. Try this.

... | rex "start_time=\"(?P<start>.*)\"\send.*\sos=\"(?P<OS>[^\"]*?)\""

Re: How to extract just one field from a log when there are multiple that carry the same attribute name? - Regular Expression

MuS — Mon, 16 Feb 2015 18:19:52 GMT

Hi evang_26,

Okay after reading the update, it makes sense, try this:

sourcetype=nessus severity!=informational  | rex max_match=1 "start_time=\"(?<start>.*)\"\send.*\s\sos=\"(?<OS>[\w+\s]+)\"

This will match only one occurrence for each field in the regex

Hope this helps ...

cheers, MuS

Re: How to extract just one field from a log when there are multiple that carry the same attribute name? - Regular Expression

ramdaspr — Tue, 17 Feb 2015 00:05:53 GMT

Can you add an example of how you would like the output to look like?
The solution provided by MuS would give you the output without any double quotes, but its not really clear if thats what your intention is or if you would like to create a multivalue field with all the different OS's added.

Re: How to extract just one field from a log when there are multiple that carry the same attribute name? - Regular Expression

evang_26 — Tue, 17 Feb 2015 10:58:09 GMT

Hi richgallowway,

I already tried this, placing the [] in front and on the end, no luck.

Regards,
Evang

Re: How to extract just one field from a log when there are multiple that carry the same attribute name? - Regular Expression

evang_26 — Tue, 17 Feb 2015 11:14:09 GMT

Hi MuS,

Okay, here we are.

Here is what I managed to do with sed, but I am not there quite yet.

sourcetype=nessus severity!=informational earliest=-5w@w1 latest=now|rex field=os mode=sed "s/.*\(os=\"[^\"]*\"\).*$/\1/g" | rex ".*os=\"(?P<OS>.*)\"\s.*\""

Any suggestions? 🙂

Regards,
Evang

Re: How to extract just one field from a log when there are multiple that carry the same attribute name? - Regular Expression

evang_26 — Tue, 17 Feb 2015 11:17:31 GMT

Hi ramdaspr,

To be honest, as I stated on MuS's answer, my problem wasn't sufficiently clarified. I want to keep just the first occurrence of OS field in case there are more than one. This way, the dashboard want look overwhelmed by huge tags, will look prettier and in fact, Nessus orders OS guesses based on probability.

Look below MuS's answer my comment and my attempt.

Regards,
Evang

Re: How to extract just one field from a log when there are multiple that carry the same attribute name? - Regular Expression

MuS — Tue, 17 Feb 2015 11:26:16 GMT

Okay, now it makes sense, try this:

sourcetype=nessus severity!=informational  | rex max_match=1 "start_time=\"(?<start>.*)\"\send.*\s\sos=\"(?<OS>[\w+\s]+)\"

This will match only one occurrence for each field in the regex

Re: How to extract just one field from a log when there are multiple that carry the same attribute name? - Regular Expression

evang_26 — Tue, 17 Feb 2015 12:23:00 GMT

Hi MuS,

It worked perfectly!

Thanks you very much!

Regards,
Evang

Re: How to extract just one field from a log when there are multiple that carry the same attribute name? - Regular Expression

MuS — Tue, 17 Feb 2015 12:35:05 GMT

you're welcome, I've updated your question and my answer so it makes sense 😉