Splunk Search

How can I use splunk like spark or storm ?


I want to analysis 100k targets using the same search command in the realtime,splunk will create 100k search jobs in the same time.
I thought the mission is impossilble.
How can I use splunk like spark or storm ?
Thank you very much!

Tags (3)
0 Karma

Path Finder

Could you add a bit more context? I'm specifically wondering about your use of the word target and whether that is implying a lot of context that I am missing. I've strictly worked with Splunk in the last couple of years and am at a disadvantage in recognizing some of the terminology assumed by users of the other platforms.

Splunk can create a single search job that returns many events across multiple indexes. When it is reasonable to create custom fields at index time, these are very efficient to search at very high scale but come with lots of advisory notices since poor decisions on deploying these can be very troublesome to recover from.

The most efficient such statistical search would be based on tstats, which works great for statistics assuming you do not require the search output to contain the full event text but just a summary of counts of events matching various conditions and the like. tstats can not be run using a Splunk real-time search so you would likely use a scheduled job with a historic search across a recent time frame that is within your tolerance level for processing latency.

Configure index time field extractions: http://docs.splunk.com/Documentation/Splunk/latest/Data/Configureindex-timefieldextraction
tstats: http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Tstats

Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...