All Apps and Add-ons

Where is the bottleneck in Splunk Hadoop?

luisterra
Explorer

Hi,

I'm running the search below:

index=windows sourcetype=windows (username=123456  event_id=4624) OR ("Account Name:  123456" event_id=4634)
| bin _time as mday span=1d
| eval logon=if(event_id==4624,_time,null())
| eval logoff=if(event_id==4634,_time,null())
| stats min(logon) as logon, max(logoff) as logoff, count(logon) as clogon, count(logoff) as clogoff  by mday
| where clogon>1 OR clogoff> 1
| convert ctime(logon) ctime(mday) ctime(logoff) 
| fields - clogon clogoff

The cut over is 14 days, and since I'm looking for results in January, February and early March all data is coming from hadoop.
The search is taking longer than 2/3 hours, it is trawling over 15*10^9 events for 1 month.
The same search for the last 13 days takes less than 1 minute in splunk.

I have looked at the search, application and container logs but did not see anything that looked obvious.

My understanding is that the map reduce job in hadoop only uses the time stamp, all the filtering happens in splunk is this correct?
Where is the bottleneck for this search to take much longer than when searching exclusively in splunk?

Thanks in advance for any pointers.

Tags (2)
0 Karma

ddrillic
Ultra Champion

The biggest factor I have seen is the hadoop queue - try to get an amplified one - it makes all the difference.

0 Karma

luisterra
Explorer

Hi,

Could you elaborate on that?

0 Karma

rdagan_splunk
Splunk Employee
Splunk Employee

This document can help debug these issues: http://docs.splunk.com/Documentation/Splunk/latest/HadoopAnalytics/TroubleshootSplunkAnalyticsforHad...
Are you running in Verbose mode?
Are you able to access the Hadoop logs to examine the performance of Hadoop itself?

0 Karma

luisterra
Explorer

Hi,

Thanks for the response and the link.

It is not that I am seeing errors when the search runs, the issue is the time it takes to complete.
It could be an expectations issue, searches using hadoop do take longer, but I was not expecting the observed difference (for the same time interval and similar number of events).
Also, I was expecting some wiggle room for tuning and improving performance, which I haven't found yet.

I do have access to yarn, and have checked the application logs. All complete successfully...
Should I be checking anything in particular?

Thanks!

0 Karma
Get Updates on the Splunk Community!

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Using the Splunk Threat Research Team’s Latest Security Content

REGISTER HERE Tech Talk | Security Edition Did you know the Splunk Threat Research Team regularly releases ...