Knowledge Management

Limiting search by lookup

PickleRick
SplunkTrust
SplunkTrust

Hello.

Maybe someone can point me in a good direction because I don't have a reasonable idea at the moment.

The situation is - the customer has many event sources (definitely over a thousand; probably closer to two). A subset of the sources is "categorized". To make long story short - there is a big excel table in which every source is described with various labels. For example:

HostSourceSiteCategory1Category2...
host1192.168.1.1site_aLANOffice...
host2192.168.2.3site_aDMZPlant...
host3WindowsEventLogsite_bLANPlant...
............... 

 

"Site" is an indexed field added at ingest time.

Every single event source is unambigously characterised as either site/source or site/host combination.

And now I need two things. One is relatively simple because the customer wants to have the categories pertaining to a given event source displayed along with events. That's quite easy  - a simple lookup does the trick; if I want to make customer's users' live easier, I can do a macro - no problem.

But the second one is where I'm stuck. The customer wants to be able to select events by using the categories of the sources. Of course with small event volumes it would be perfectly acceptable to do a simple query, run it through a lookup and then filter the result. With small set of sources I'd just tell the customer to filter by particular host and get on with his life. But in this case the amount of data we would have to prepare firstly, just to be filtered later would be huge, so it would be reasonable to filter it as early as we can (especially that both pairs of fields that can be used to distinguish event sources are indexed fields and don't have to be parsed from the events). And the subsets of the hosts we would need to be using are quite big.

I thought about using lookup in a subquery but don't know how to approach it so it makes sense.

Labels (1)
0 Karma
1 Solution

PickleRick
SplunkTrust
SplunkTrust

OK. So it seems that:

Firstly, I posted in bad subforum - should be in "Splunk Search" rather than "Knowledge Management" I think. Sorry for that.

Secondly - as always - I posted a question then had an epiphany 😆

In this case I need to indeed use a subsearch. I need to use inputlookup with a WHERE clause to limit results and use TABLE to limit output only to columns needed for pinpointing right combination of fields needed for event selection.

[| inputlookup test.csv where Category1=DMZ | table index,source,host ]

I can combine more than one condition in the subsearch but I can also make the searching "modular" by combining more than one such subsearch together. Like:

[| inputlookup test.csv where Category1=DMZ | table source,host,site ][| inputlookup test.csv where Category2=Office | table source,host,site ]

(I know that in this form it's not very pretty but if you define shorthand macros it can be quite useful this way)

View solution in original post

0 Karma

PickleRick
SplunkTrust
SplunkTrust

OK. So it seems that:

Firstly, I posted in bad subforum - should be in "Splunk Search" rather than "Knowledge Management" I think. Sorry for that.

Secondly - as always - I posted a question then had an epiphany 😆

In this case I need to indeed use a subsearch. I need to use inputlookup with a WHERE clause to limit results and use TABLE to limit output only to columns needed for pinpointing right combination of fields needed for event selection.

[| inputlookup test.csv where Category1=DMZ | table index,source,host ]

I can combine more than one condition in the subsearch but I can also make the searching "modular" by combining more than one such subsearch together. Like:

[| inputlookup test.csv where Category1=DMZ | table source,host,site ][| inputlookup test.csv where Category2=Office | table source,host,site ]

(I know that in this form it's not very pretty but if you define shorthand macros it can be quite useful this way)

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...