- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Internal Mapreduce in Splunk
Hi,
I need to know how map functions and reduce functions are constructed using search string?
In one of the white paper i came across finding first reduce command (Which cannot be parallelized)...
How it happens in Splunk internally? Where the intermediate data will be shown?
What is pretimechart (Intermediate command for timechart??)????
- Arun
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Is this what you're looking for?
https://www.splunk.com/content/dam/splunk2/pdfs/technical-briefs/splunk-and-mapreduce.pdf
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@aweitzman,
Yes i went though this doc but here they mentioned something like unparallelizable command.
If i find unparallelizable command as second command in search with 100 commands then will all other 98 commands will be converted into reduce processes?
What is pretimechart (Intermediate command for timechart??)????
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
According to the document, "pretimechart" is a summarization command that can be run in a parallelizable way across all the data sets generated by "map" commands, so that when it comes time to "reduce," all the unnecessary data is already pruned.
As for your "what if" scenario, since most search-related commands don't fall into that bucket, you shouldn't have to worry about it very often. And if you do run into it, then there are usually creative ways to restructure your search so as to avoid it.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thank you Aweitzman.
Could you please have a look into my below posts :). Its been 2 weeks i didnt got any answers yet :(...
http://answers.splunk.com/answers/239222/when-to-use-django-approach-in-splunk-apps.html
http://answers.splunk.com/answers/237073/is-dynamic-query-construction-possble-in-splunk-wi-1.html
