We are running some splice searches to match on IOC's and they seem to be running very slow. We have a hybrid Splunk deployment with Indexers in Cloud and the Splice Search Head running ON Prem. Because of this we use |localop| in our searches so that Splice fetches all required components from the Search Head itself. Would this be a reason on why the searches run slow.
I am passing on an average about 400k events for a 10 min interval search if the search runs between 9 AM - 5PM. It takes about 20 min+ for splice to complete that search. Sometimes it just hangs up. Because of this its doesn't run on the schedule interval. Did anyone face the same issue?
The number of IOC's in Splice Mongo DB is about 115k now. We see the same slowness even when mongo db had only 40k records.
Has anyone faced this issue before. Any help would be appreciated.
SPLICE is a prototype and as any prototype, there are some limitations. One workaround would be to use the iocexportcsv command to create CSV lists of technical indicators that you would after refer via lookups or ES Threat List. And yes the command localop retrieve all the data the process it on the SH.