How to extract just one field from a log when there are multiple that carry the same attribute name? - Regular Expression

[evang_26]

Hi all,

I am filtering some logs came from Nessus in order to identify vulnerable machines based on their OS, and the issue I have is when a host's OS is not adequately identified resulting in many "os" fields. An example is the below:

start_time="Mon Feb 16 03:56:07 2015"
end_time="Mon Feb 16 03:57:42 2015"
os="Microsoft Windows 2000" os="Microsoft Windows XP for Embedded Systems" os="Microsoft
Windows XP"

The query that I created for that (which only works sufficiently when 1 OS is found) is the following:

sourcetype=nessus severity!=informational  | rex "start_time=\"(?<start>.*)\"\send.*\s\sos=\"(?<OS>.*)\""

What I would like ideally to do, is to just find a way to filter out the " (double quote" symbol from within the extracted field. This is because apart from Windows machines, there are other printers and access points that are interpreted as many other mixed OSs.

So, it should be something like this:

sourcetype=nessus severity!=informational  | rex "start_time=\"(?<start>.*)\"\send.*\s\sos=\"(?<OS>.*[^\"])\""

but it doesn't work.

Any thoughts?

Regards,
Evang

Update:

Hi MuS,

Okay, here we are.

I guess I haven't stated my problem correctly. I do not want to remove the double quotes, actually, I want to only keep the first occurence of OS field in the rare cases that more than one appears!

Here is what I managed to do with sed, but I am not there quite yet.

sourcetype=nessus severity!=informational earliest=-5w@w1 latest=now|rex field=os mode=sed "s/.*\(os=\"[^\"]*\"\).*$/\1/g" | rex ".*os=\"(?P<OS>.*)\"\s.*\""

Any suggestions? 🙂

Regards,
Evang

[ramdaspr]

Can you add an example of how you would like the output to look like?
The solution provided by MuS would give you the output without any double quotes, but its not really clear if thats what your intention is or if you would like to create a multivalue field with all the different OS's added.

[evang_26]

Hi ramdaspr,

To be honest, as I stated on MuS's answer, my problem wasn't sufficiently clarified. I want to keep just the first occurrence of OS field in case there are more than one. This way, the dashboard want look overwhelmed by huge tags, will look prettier and in fact, Nessus orders OS guesses based on probability.

Look below MuS's answer my comment and my attempt.

Regards,
Evang

[MuS · Accepted solution]

Hi evang_26,

Okay after reading the update, it makes sense, try this:

sourcetype=nessus severity!=informational  | rex max_match=1 "start_time=\"(?<start>.*)\"\send.*\s\sos=\"(?<OS>[\w+\s]+)\"

This will match only one occurrence for each field in the regex

Hope this helps ...

cheers, MuS

[evang_26]

Hi MuS,

Okay, here we are.

I guess I haven't stated my problem correctly. I do not want to remove the double quotes, actually, I want to only keep the first occurence of OS field in the rare cases that more than one appears!

Here is what I managed to do with sed, but I am not there quite yet.

sourcetype=nessus severity!=informational earliest=-5w@w1 latest=now|rex field=os mode=sed "s/.*\(os=\"[^\"]*\"\).*$/\1/g" | rex ".*os=\"(?P<OS>.*)\"\s.*\""

Any suggestions? 🙂

Regards,
Evang

[MuS]

Okay, now it makes sense, try this:

sourcetype=nessus severity!=informational  | rex max_match=1 "start_time=\"(?<start>.*)\"\send.*\s\sos=\"(?<OS>[\w+\s]+)\"

This will match only one occurrence for each field in the regex

[evang_26]

Hi MuS,

It worked perfectly!

Thanks you very much!

Regards,
Evang

[MuS]

you're welcome, I've updated your question and my answer so it makes sense 😉

[richgalloway]

You were so close. Try this.

... | rex "start_time=\"(?P<start>.*)\"\send.*\sos=\"(?P<OS>[^\"]*?)\""

[evang_26]

Hi richgallowway,

I already tried this, placing the [] in front and on the end, no luck.

Regards,
Evang