topic Re: Using a lookup for search in Splunk Search

Using a lookup for search

cthulhucalling — Wed, 30 Sep 2020 00:51:10 GMT

I have a small CSV file with common attack signatures in them that I have uploaded as a lookup called web_attack_signatures.csv:

uri_path, description
*/etc/passwd*, Passwd file access attempt
*phpadmin*, PHPadmin access attempt

Ignoring the fact that these signatures are inefficient, the idea is that the lookup feeds the search, looking for any attack signature matches matches in my web logs. The relevant field in my logs is also called uri_path. So far I have been able to pull relevant events using this query:

earliest=-1d index=web [| inputlookup web_attack_sigs.csv | fields uri_path]

But I cannot get get the description field to be added to the results. I've tried adding "description" to the fields command with 0 results, and tried piping the events to another lookup command but that didn't work either. Can anyone help?

Re: Using a lookup for search

MuS — Wed, 12 Jun 2019 00:55:52 GMT

Hi cthulhucalling,

try this:

earliest=-1d index=web [| inputlookup web_attack_sigs.csv | format]

by using format you will get a list of

( ( uri_path="*/etc/passwd*" AND description ="Passwd file access attempt" ) OR ( uri_path="*phpadmin*" AND description ="PHPadmin access attempt" ) )

That will be added to your base search. More details here https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Format

Hope this helps ...

cheers, MuS

Re: Using a lookup for search

cthulhucalling — Wed, 12 Jun 2019 13:58:37 GMT

I tried this query. Unfortunately, it returned 0 results.

Re: Using a lookup for search

FrankVl — Wed, 12 Jun 2019 14:05:40 GMT

When adding the description to the output of the subsearch, it will also use that for filtering, which is not what you want. If I understand correctly, you want to filter the events for the signatures and then add the description to the results, right?

I think the following should work (might require some configuration change on your lookup, to enable wildcard matches):

earliest=-1d index=web 
| lookup web_attack_sigs.csv uri_path OUTPUT description
| search description=*

Possibly the search runs more quickly if you keep the subsearch to do the initial filtering:

earliest=-1d index=web [| inputlookup web_attack_sigs.csv | fields uri_path]
| lookup web_attack_sigs.csv uri_path OUTPUT description

Re: Using a lookup for search

cthulhucalling — Wed, 12 Jun 2019 14:14:12 GMT

The first query returns anything with "description" in the event. The second one I had tried yesterday. It returns relevant results but the description line is not being added to the events.

Re: Using a lookup for search

cthulhucalling — Wed, 12 Jun 2019 14:33:01 GMT

The issue appears to be that I'm wildcarding the signatures. If I change /etc/passwd to just /etc/passwd, I do get the description when I do this:

earliest=-1d index=web [| inputlookup web_attack_sigs.csv | fields uri_path]
| lookup web_attack_sigs.csv uri_path OUTPUT description

I'm guessing that because of the wildcards, it's not an exact match when the lookup occurs, so it's not a hit and the description isn't added. The issue is that I get only results where the request is exactly for /etc/passwd.

Re: Using a lookup for search

FrankVl — Wed, 12 Jun 2019 14:38:02 GMT

Did you configure your lookup to allow wildcard matches on the uri_path field? Otherwise the lookup command will not work as expected.

Re: Using a lookup for search

cthulhucalling — Wed, 12 Jun 2019 14:42:14 GMT

I don't have shell access to that particular search head, so I can't do that.

Re: Using a lookup for search

FrankVl — Wed, 12 Jun 2019 14:53:43 GMT

You can configure that in the GUI in the advanced options of a lookup definition.

Re: Using a lookup for search

cthulhucalling — Wed, 12 Jun 2019 15:33:07 GMT

Adding WILDCARD(uri_path) to the lookup definition did the trick! Thank you!

Re: Using a lookup for search

jnudell_2 — Wed, 12 Jun 2019 19:31:17 GMT

Hi @cthulhucalling ,

As some of the others have said, you can have wildcards in your lookups, but you have to specify it in the advanced settings in the web interface (UI). Here is an example of how you would do that:

Once you have the lookup configured, you can use the following search logic:

| lookup web_attack_sigs uri_path OUTPUT description

Here is what it looks like:

Re: Using a lookup for search

MuS — Wed, 12 Jun 2019 20:25:29 GMT

Well, it also looks like I completely misunderstood your question 😉 Sorry for that.

cheers, MuS