topic Re: How to create a custom field to match a particular string? in Getting Data In

How to create a custom field to match a particular string?

jwelters — Tue, 29 Sep 2020 12:54:51 GMT

I have an interesting use case, where I have a list of strings that I search for within our proxy logs to identify problem systems. This works quite well. Currently the list exists in both a CSV format, and a search Macro. The macro just specifies (phrase1 OR phrase2 OR phrase3) it's not complicated.

Recently I decided I wanted to add to our analysis and return the results but also figure out how to add a new field that is the particular string that was matched. For example if the phrase was infected, I could display a table with the _time, URL , and matched_string. This would make analysis a bit easier for us.

I've seen a couple ways folks have looked at doing something similar but I wasn't able to find a working solution for me, has anyone done something like this before ?

Re: How to create a custom field to match a particular string?

somesoni2 — Fri, 17 Feb 2017 20:29:31 GMT

How many keywords are there? I'm assuming there are low as you are using them in Macro. If that's still the case, you could write another macro, which will do like this. You'd end up maintaining two macros and have to update both when you add a new phrase.

 eval matched_string=case(searchmatch("phrase1"),"phrase1",searchmatch("phrase2"),"phrase2", searchmatch("phrase3"),"phrase3",...,1=1,"PhraseNotInMacro")

Re: How to create a custom field to match a particular string?

DalJeanis — Sat, 18 Feb 2017 00:07:33 GMT

Here's a sample of what you can do. This makes a list (you would substitute your lookup table here)

|makeresults |eval mylist="infected disturbed reallycrazy zombiefied"|makemv mylist | mvexpand mylist

this turns it into a tight regex

|rename mylist as search
|format "(?<resultfield>" "" 3 "" "|" ")" 
| rex field=search mode=sed "s/\>  /\>/g" 
| rex field=search mode=sed "s/  \|  /|/g"

The return value looks like this -

(?<resultfield\>infected|disturbed|reallycrazy|zombiefied )

and you can drop that into a rex command

| rex field=_raw max_match=5 [the above search using your lookup table as input]

to extract which of the lookup item(s) matched.

Re: How to create a custom field to match a particular string?

jwelters — Tue, 21 Feb 2017 13:28:26 GMT

There are a few hundred.

Re: How to create a custom field to match a particular string?

somesoni2 — Tue, 21 Feb 2017 15:45:12 GMT

My bet will be on creating a macro with case expressions similar to above, assuming the phrase can occur at any place in the raw data (no fixed position).

Re: How to create a custom field to match a particular string?

jwelters — Tue, 21 Feb 2017 16:28:53 GMT

It looks like this will work well, as there's already a script that creates the Search macro, and the lookup table... it's fairly easy to also have it create the macro for this.

Re: How to create a custom field to match a particular string?

aaraneta_splunk — Wed, 19 Apr 2017 23:39:27 GMT

@jwelters - Did one of the answers below help provide a solution your question? If yes, please click “Accept” below the best answer to resolve this post and upvote anything that was helpful. If no, please leave a comment with more feedback. Thanks.

Re: How to create a custom field to match a particular string?

jwelters — Thu, 20 Apr 2017 12:40:57 GMT

This was the best solution, it just wasn't a great one but it worked. I believe it's the best solution given the limitations of the core tool.