topic Re: Hunk is taking too much time for processing hive ORC data. How do I improve the performance? in Monitoring Splunk

Hunk is taking too much time for processing hive ORC data. How do I improve the performance?

toabhishek16 — Thu, 25 Jun 2015 07:32:48 GMT

Hi Team,

I have set up hunk with Apache Hadoop 2.26 and my data is stored in Hive 0.13 table with ORC compression. Data size is around 2 TB.

When I am trying to execute any query through Hunk, it is taking too much time. Equivalent query in hive is taking only 80 Sec. I am executing the above on the same hive table.

Please help me to improve the performance of Hunk. How I can achieve fast data processing through Hunk?

Thanks
Abhishek

Re: Hunk is taking too much time for processing hive ORC data. How do I improve the performance?

rdagan_splunk — Thu, 25 Jun 2015 14:54:56 GMT

Can you share the Hunk query? Also please make sure you are in a smart search mode (not in Verbose mode)

Re: Hunk is taking too much time for processing hive ORC data. How do I improve the performance?

toabhishek16 — Mon, 28 Sep 2020 20:22:55 GMT

Hi rdagan_splunk,

I am using below queries in Hunk and Hive respectevely:

index=idxtmgorc cs_username="anyname" - taking too much time almost more then 30 minutes.

select cs_username from tmg_orc_table where cs_username='anyname - hive query taking only 80 sec on same data and same cluster.

give me some time, I will let you know about the mode.
Thanks & Regards
Abhishek Soni

Re: Hunk is taking too much time for processing hive ORC data. How do I improve the performance?

rdagan_splunk — Mon, 28 Sep 2020 20:23:30 GMT

Try this combo:
index=idxtmgorc cs_username="anyname" | stats count(cs_username) as username
Be in Smart Mode
In addition, Here is a link that shows you the Mode options:
http://docs.splunk.com/Documentation/Splunk/6.2.4/SearchTutorial/Aboutsearchactionsandmodes
and a link that shows you the Search commands:
http://docs.splunk.com/Documentation/Splunk/6.2.4/SearchTutorial/Usethesearchlanguage

Re: Hunk is taking too much time for processing hive ORC data. How do I improve the performance?

toabhishek16 — Mon, 29 Jun 2015 09:53:58 GMT

Hi rdagan_splunk,

I am in Smart mode as suggested by you.

I tried the queries suggested by you and observed when I am runnning query which involves stats command, it is working fast. but when I am trying to run generic queries like index=idxtmg, its is taking too much time for giving results.

please suggest me how I can improve the system's performance.

Re: Hunk is taking too much time for processing hive ORC data. How do I improve the performance?

rdagan_splunk — Mon, 29 Jun 2015 14:24:31 GMT

I am happy to see that you are able to improve the performance when running Hadoop MapReduce Jobs.
The rule is Hunk triggers an MR job if:
1. the search is not ran in verbose mode AND
2. the search contains any filtering predicates in the first search command
OR
3. the search contains any reporting commands

Re: Hunk is taking too much time for processing hive ORC data. How do I improve the performance?

toabhishek16 — Wed, 01 Jul 2015 07:04:00 GMT

thanks rdagan_splunk for clarifying the problem.

Re: Hunk is taking too much time for processing hive ORC data. How do I improve the performance?

burwell — Mon, 02 May 2016 02:10:36 GMT

Abhishek did you ever solve the performance issue? I have the same issue.