I want to analysis 100k targets using the same search command in the realtime,splunk will create 100k search jobs in the same time.
I thought the mission is impossilble.
How can I use splunk like spark or storm ?
Thank you very much!
Could you add a bit more context? I'm specifically wondering about your use of the word target and whether that is implying a lot of context that I am missing. I've strictly worked with Splunk in the last couple of years and am at a disadvantage in recognizing some of the terminology assumed by users of the other platforms.
Splunk can create a single search job that returns many events across multiple indexes. When it is reasonable to create custom fields at index time, these are very efficient to search at very high scale but come with lots of advisory notices since poor decisions on deploying these can be very troublesome to recover from.
The most efficient such statistical search would be based on tstats, which works great for statistics assuming you do not require the search output to contain the full event text but just a summary of counts of events matching various conditions and the like. tstats can not be run using a Splunk real-time search so you would likely use a scheduled job with a historic search across a recent time frame that is within your tolerance level for processing latency.