Building for the Splunk Platform

How to add Boolean operators on lookup files?

bonnlbbelandres
Path Finder

I would like to have a lookup that categorizes events depending on detected keywords from a specific field and I'd like to have my splunk search query as simple as possible. In order to do this i have thought of creating a lookup file that does the categorization. My keyword criteria includes AND and OR Boolean such as Keyword1 category is different from Keyword1 AND Keyword2 category..
This is the most critical part of the lookup file I'm making.
I was told by a friend that I should use python to read booleans from the lookup.
Can I do this without doing an python coding (because i really find it difficult to code in python) or
if Python coding is the only option, can you tell me how to kickstart my development?

Here is the table that i have in mind.

Group................Keywords...................Area..... Cluster......Subcluster
A..................... boy AND girl................. A1........... C1.................. S1
B..........................boy............................A2..............C2...................S2

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

I suspect that a custom function -- whether in Python or another high-level language -- would meet your loong-term needs better than attempting to build a lookup-based assignment of category. My estimate here is based largely on the fact that you say you will use both AND and OR, depending on the category, any my observation that record processing order in splunk is not guaranteed under many circumstances.

The caution I would have to underling here is that not everything which CAN be done, SHOULD be done. It might be technically feasible to write something like you request, for example, woodcock's KVstore suggestion could work. However, the order of magnitude of such a solution can rapidly get out of hand. The optimal case for this is going to n log n, but I'd expect most programmers who are posting a question like this would end up with an implementation that is less than optimal, possibly exponential. (ie, twice as many records generate four or more times as many operations.

On the other hand, a custom function would be much more testable in terms of its operation, and take up a more-or-less a straight-line order of magnitude (with roughly n log n for the number of categories) on the lookup, which is the best you can hope for.

0 Karma

woodcock
Esteemed Legend

What you need to do is to use a KV-store based lookup which supports multi-valued fields. It would look like this:

|inputcsv YourCSVwithANDstuff | makemv delim=" AND " Keywords | makemv Keywords | outputlookup YourKVStoreBasedLookup

Then you can run a search against your lookup like this:

Your Search Here | lookup YourKVStoreBasedLookup | where Keywords="boy" AND Keywords="girl"

You could also use mvexpand on the end of the coverter search and use a file-based lookup and it would do the same thing but take up more space because each line with an " AND " will be a duplicated row, rather than a multi-valued field.

0 Karma

niketn
Legend

@bonnlbbelandres... is the example table here the lookup file? Once you have the lookup what is the key to correlate this from search and what should your final search look like?

Can you provide couple of searches you are trying to build with the lookup? You can give mock query also in case you can not share the exact query. Some sample data to test would also be great.

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma
Get Updates on the Splunk Community!

Using Machine Learning for Hunting Security Threats

WATCH NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more for ...

Observability Newsletter Highlights | March 2023

 March 2023 | Check out the latest and greatestSplunk APM's New Tag Filter ExperienceSplunk APM has updated ...

Security Newsletter Updates | March 2023

 March 2023 | Check out the latest and greatestUnify Your Security Operations with Splunk Mission Control The ...