Splunk Search

What are recommendations for optimizing the performance of my current search?

hartfoml
Motivator

I have this search that is a very sparse search over 7 days and takes a long time to complete (more than ten hours) .

eventtype=wineventlog_windows host_list=1 | eval Host=if(isnull(Host), upper(host), upper(Host)) | stats count by Host | eval _key = Host | outputlookup windows_event_hosts.csv

This search is using a lookup file to tag systems as being on the host_list
I checked and there are 147 systems tagged and more than three times as many that are not on the list.

how can I reduce the time to run this search?

Some ideas I had were to
limit to known indexes,
Change from eventtype to sourcetype,
run a subsearch to get the host names and do a search on just the host names

I don't quite know what will happen if I check "Accelerate this search" box or if the data will be returned correctly in the dashboards.
I don't know how accelerating the scheduled search will impact resources like CPU, Memory, Disk Space?

0 Karma

jacobwilkins
Communicator

To properly answer your question, we'd need to see the definition for the "wineventlog_windows" eventtype.

We'd also need to know more about host_list. Is that a field from a lookup? An auto-lookup? If so what is the structure of the lookup? How big is it in total?

Your question about a subsearch makes me question the overall logic of this endeavor. For a reasonable list of hosts, a subsearch isn't a bad approach... However, if your specific case I don't see how it would make sense...

The subsearch would have the effect of adding (host=foo OR host=bar OR host=etc) to your search... So you'd never see any hosts that weren't in that lookup.... Your outputlookup in this search would just create a lookup with the host of everything in the first list that had reported in...

You could probably do all of this with metasearch, but I don't think that is the best approach.

How about license logs?

index=_internal LicenseUsage type=Usage

Start with that. In those results, the host sending the event will be in the field called h, and the sourcetype of the event being reported will be in st.

0 Karma

aljohnson_splun
Splunk Employee
Splunk Employee

docs on report acceleration here and here.

are you searching in fast mode?
if you inspect your job with the search job inspector, what is the most costly part of execution?

From what I can tell, report acceleration would be reasonable here, however, as you mentioned, report acceleration DOES take up more disk space and processing resources.

0 Karma
Get Updates on the Splunk Community!

Introducing Splunk Enterprise 9.2

WATCH HERE! Watch this Tech Talk to learn about the latest features and enhancements shipped in the new Splunk ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...