Splunk Search

How can I use splunk like spark or storm ?


I want to analysis 100k targets using the same search command in the realtime,splunk will create 100k search jobs in the same time.
I thought the mission is impossilble.
How can I use splunk like spark or storm ?
Thank you very much!

Tags (3)
0 Karma

Path Finder

Could you add a bit more context? I'm specifically wondering about your use of the word target and whether that is implying a lot of context that I am missing. I've strictly worked with Splunk in the last couple of years and am at a disadvantage in recognizing some of the terminology assumed by users of the other platforms.

Splunk can create a single search job that returns many events across multiple indexes. When it is reasonable to create custom fields at index time, these are very efficient to search at very high scale but come with lots of advisory notices since poor decisions on deploying these can be very troublesome to recover from.

The most efficient such statistical search would be based on tstats, which works great for statistics assuming you do not require the search output to contain the full event text but just a summary of counts of events matching various conditions and the like. tstats can not be run using a Splunk real-time search so you would likely use a scheduled job with a historic search across a recent time frame that is within your tolerance level for processing latency.

Configure index time field extractions: http://docs.splunk.com/Documentation/Splunk/latest/Data/Configureindex-timefieldextraction
tstats: http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Tstats

Get Updates on the Splunk Community!

Splunk Observability Cloud | Customer Survey!

If you use Splunk Observability Cloud, we invite you to share your valuable insights with us through a brief ...

.conf23 | Get Your Cybersecurity Defense Analyst Certification in Vegas

We’re excited to announce a new Splunk certification exam being released at .conf23! If you’re going to Las ...

Starting With Observability: OpenTelemetry Best Practices

Tech Talk Starting With Observability: OpenTelemetry Best Practices Tuesday, October 17, 2023   |  11AM PST / ...