Knowledge Management

Limiting search by lookup

PickleRick
Builder

Hello.

Maybe someone can point me in a good direction because I don't have a reasonable idea at the moment.

The situation is - the customer has many event sources (definitely over a thousand; probably closer to two). A subset of the sources is "categorized". To make long story short - there is a big excel table in which every source is described with various labels. For example:

HostSourceSiteCategory1Category2...
host1192.168.1.1site_aLANOffice...
host2192.168.2.3site_aDMZPlant...
host3WindowsEventLogsite_bLANPlant...
............... 

 

"Site" is an indexed field added at ingest time.

Every single event source is unambigously characterised as either site/source or site/host combination.

And now I need two things. One is relatively simple because the customer wants to have the categories pertaining to a given event source displayed along with events. That's quite easy  - a simple lookup does the trick; if I want to make customer's users' live easier, I can do a macro - no problem.

But the second one is where I'm stuck. The customer wants to be able to select events by using the categories of the sources. Of course with small event volumes it would be perfectly acceptable to do a simple query, run it through a lookup and then filter the result. With small set of sources I'd just tell the customer to filter by particular host and get on with his life. But in this case the amount of data we would have to prepare firstly, just to be filtered later would be huge, so it would be reasonable to filter it as early as we can (especially that both pairs of fields that can be used to distinguish event sources are indexed fields and don't have to be parsed from the events). And the subsets of the hosts we would need to be using are quite big.

I thought about using lookup in a subquery but don't know how to approach it so it makes sense.

Labels (1)
0 Karma
1 Solution

PickleRick
Builder

OK. So it seems that:

Firstly, I posted in bad subforum - should be in "Splunk Search" rather than "Knowledge Management" I think. Sorry for that.

Secondly - as always - I posted a question then had an epiphany 😆

In this case I need to indeed use a subsearch. I need to use inputlookup with a WHERE clause to limit results and use TABLE to limit output only to columns needed for pinpointing right combination of fields needed for event selection.

[| inputlookup test.csv where Category1=DMZ | table index,source,host ]

I can combine more than one condition in the subsearch but I can also make the searching "modular" by combining more than one such subsearch together. Like:

[| inputlookup test.csv where Category1=DMZ | table source,host,site ][| inputlookup test.csv where Category2=Office | table source,host,site ]

(I know that in this form it's not very pretty but if you define shorthand macros it can be quite useful this way)

View solution in original post

0 Karma

PickleRick
Builder

OK. So it seems that:

Firstly, I posted in bad subforum - should be in "Splunk Search" rather than "Knowledge Management" I think. Sorry for that.

Secondly - as always - I posted a question then had an epiphany 😆

In this case I need to indeed use a subsearch. I need to use inputlookup with a WHERE clause to limit results and use TABLE to limit output only to columns needed for pinpointing right combination of fields needed for event selection.

[| inputlookup test.csv where Category1=DMZ | table index,source,host ]

I can combine more than one condition in the subsearch but I can also make the searching "modular" by combining more than one such subsearch together. Like:

[| inputlookup test.csv where Category1=DMZ | table source,host,site ][| inputlookup test.csv where Category2=Office | table source,host,site ]

(I know that in this form it's not very pretty but if you define shorthand macros it can be quite useful this way)

View solution in original post

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!