With Hunk, verbose mode does not run a MapReduce job, so what does it do?

[jaredlaney]

I'm trying to understand Hunk verbose mode more. Here is a reference to verbose mode not running an MR job.

http://answers.splunk.com/answers/239341/in-hunk-verbose-mode-vs-smart-mode-for-vix-virtual.html

[dturnbull_splun · Accepted solution]

It's streaming data back from HDFS

[jaredlaney]

Using a hadoop streaming job? Or just a REST call to hdfs?

[kschon_splunk]

The Hadoop file system interface (usually backed by HDFS, but sometimes by S3, GPFS, etc.) allows a client to open an input stream to any file it contains. There is no associated job. Verbose mode does all calculations on the Search Head, so that all log messages are generated in one place, in a well defined order. To do this, it first streams the data from the Hadoop file system using an input stream. No MR job is started.

[jaredlaney]

@kschon - Thanks for the explanation. I was just trying to figure out where the compute happened and to understand the limitations, etc. If I have Billions of records and I run in verbose mode, I'm guessing the performance would be much worse than the MR job with fast mode. (search head processing vs. hadoop cluster)

[kschon_splunk]

Yes, much worse. You should only use verbose mode when you are trying to diagnose a problem.