All Apps and Add-ons

Where is the bottleneck in Splunk Hadoop?



I'm running the search below:

index=windows sourcetype=windows (username=123456  event_id=4624) OR ("Account Name:  123456" event_id=4634)
| bin _time as mday span=1d
| eval logon=if(event_id==4624,_time,null())
| eval logoff=if(event_id==4634,_time,null())
| stats min(logon) as logon, max(logoff) as logoff, count(logon) as clogon, count(logoff) as clogoff  by mday
| where clogon>1 OR clogoff> 1
| convert ctime(logon) ctime(mday) ctime(logoff) 
| fields - clogon clogoff

The cut over is 14 days, and since I'm looking for results in January, February and early March all data is coming from hadoop.
The search is taking longer than 2/3 hours, it is trawling over 15*10^9 events for 1 month.
The same search for the last 13 days takes less than 1 minute in splunk.

I have looked at the search, application and container logs but did not see anything that looked obvious.

My understanding is that the map reduce job in hadoop only uses the time stamp, all the filtering happens in splunk is this correct?
Where is the bottleneck for this search to take much longer than when searching exclusively in splunk?

Thanks in advance for any pointers.

Tags (2)
0 Karma

Ultra Champion

The biggest factor I have seen is the hadoop queue - try to get an amplified one - it makes all the difference.

0 Karma



Could you elaborate on that?

0 Karma

Splunk Employee
Splunk Employee

This document can help debug these issues:
Are you running in Verbose mode?
Are you able to access the Hadoop logs to examine the performance of Hadoop itself?

0 Karma



Thanks for the response and the link.

It is not that I am seeing errors when the search runs, the issue is the time it takes to complete.
It could be an expectations issue, searches using hadoop do take longer, but I was not expecting the observed difference (for the same time interval and similar number of events).
Also, I was expecting some wiggle room for tuning and improving performance, which I haven't found yet.

I do have access to yarn, and have checked the application logs. All complete successfully...
Should I be checking anything in particular?


0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...