Running search for 1000s of IOCs at once?

[Samantha]

I would like to create a dashboard which would run a search daily to check network traffic against a list of about 18,000 IP address. 

We created a lookup table with all the IP addresses and ran it, but the search times out. Then we tried to split the lookup tables into 8 different tables and each table was a panel in our dashboard. A few dashboards will run when we do it this way, but then the rest time out. 

An idea we had was to either create a drop down tab to only run the searches when we specify, or create a search that runs one lookup table and then will only start the next search when the other stops. 

Is there a simpler way to do this? Ideally it would all be one search but it just seems to be too much for our resources.

[gcusello]

Hi @Samantha ,

as also @PickleRick and @ITWhisperer said, this seems to be a job for a scheduled report.

If you want a dashboard, you could schedule a search (e.g. as an alert) running your search and sabing aggregated results in a summary index, then you could run the searches of your dashboard on this summary index.

Ciao.

Giuseppe

[PickleRick]

It depends on a use case. What and how you're searching.

Are you trying to search raw data or summarized datamodel? Are you using that lookup to generate search terms using a subsearch or are you using the lookup command? What amount of data are we talking about?

[ITWhisperer]

You could set up some scheduled reports to run on partial sets of addresses, then load the results from the searches in your dashboard. This assumes you can work with out of date data e.g. your report is based on yesterday's data and you don't need the very latest data.

Alternatively, as you said, you could "chain" your searches based on when a search completes, set a token which the next search is waiting for, and so on. (This is easier to do in SimpleXML, but still possible in Studio.)