topic Re: How to write a search to use regex to extract multiple fields from a log line and find the total count of all matched patterns? in Splunk Search

How to write a search to use regex to extract multiple fields from a log line and find the total count of all matched patterns?

sugitime — Fri, 14 Nov 2014 16:38:59 GMT

I've looked through several of the other questions related to this one, but they were either unanswered, or answered with a workaround specific to the question asked.

I am trying to write a search which will search all log lines in all indices, and grep for specific patterns.
Each log line may have between 0 and all patterns that match.
The logs format vary based on what they are (custom logs, apache logs, etc).
I am not able to edit the transforms.conf; this would have to be done within the search line.
What I am interested in is the total count of matched patterns.

My first attempt was:

index=* | regex("regexpattern1"="/regexpattern/") | regex("regexpattern2"="/regexpattern/") | regex("regexpattern3"="/regexpattern/")

But I think that this is causing the results of one regex to just be piped into the other regex, which is not exactly what I was looking for.

Could anyone provide me a bit of guidance as to where I might start looking for this?

Re: How to write a search to use regex to extract multiple fields from a log line and find the total count of all matched patterns?

ppablo — Fri, 14 Nov 2014 16:49:49 GMT

Hi @sugitime

Regex experts floating here on Answers will need you to provide some sample data and specify what parts you want to extract to really help you put the syntax together 🙂

Re: How to write a search to use regex to extract multiple fields from a log line and find the total count of all matched patterns?

somesoni2 — Fri, 14 Nov 2014 17:00:34 GMT

Without much information give this a try

index=* | eval type1=if(searchmatch("regexpattern1"="/regexpattern/") ,1,0)| eval type2=if(searchmatch("regexpattern2"="/regexpattern/"),1,0) | eval type3=if(searchmatch("regexpattern3"="/regexpattern/"),1,0) | stats count as Total Events sum(type1) as Type1Count sum(type2) as Type2Count sum(type3) as Type3Count

Re: How to write a search to use regex to extract multiple fields from a log line and find the total count of all matched patterns?

sugitime — Fri, 14 Nov 2014 17:08:03 GMT

I can understand that. However I cant really provide samples. There are just so many different types of logs that could exist.

I wrote a python script which goes to different log locations, opens each log, reads each log line and compares those log lines to regex patterns I have created. It records the results and moves on.

I am trying to replace this with Splunk. So the log type really cant matter; the regex pattern would need to be able to look at each and every log line and match only on pattern, with no consideration to the type of file, the format of the file, the layout of the file or thing else related to the file itself.

Re: How to write a search to use regex to extract multiple fields from a log line and find the total count of all matched patterns?

sugitime — Fri, 14 Nov 2014 17:24:39 GMT

I took a look at that. I got this error:
Error in 'eval' command: The arguments to the 'searchmatch' function are invalid.

I changed the searchmatch() to only use 1 argument (per the Splunk docs):
eval type1=if(searchmatch("/regexpattern/") ,1,0)

Then I piped those out to stats sum(type1) as Type1Count

Then I did this for the rest of the patterns, and it seems to be working 🙂
So my final search (thanks to your help) is:
index=* | eval type1=if(searchmatch("/regexpattern/") ,1,0)| eval type2=if(searchmatch("/regexpattern/"),1,0) | eval type3=if(searchmatch("/regexpattern/"),1,0) | stats sum(type1) as Type1Count sum(type2) as Type2Count sum(type3) as Type3Count

Thank you!!!