Splunk Dashboard for million of events

[nilaksh92]

Hi Everyone,

Please help me out to resolve below issue.

Data is coming in Index at every 30 seconds. I need to create real time aggregated dashboard (Lot of calculations) for that.

I am able to create dashboard with drill down functionality, but it is taking lot of time to display the final result.

What should I do in order to resolve this issue?

Thanks in advance.
Nikks.

[jkat54]

Convert the lookup to an auto-lookup in settings -> lookups -> auto-lookups so that the inputlookup and the join are not needed.

http://docs.splunk.com/Documentation/Splunk/6.6.1/Knowledge/DefineanautomaticlookupinSplunkWeb

Then change your dashboard search to just be index=xyz _index_earliest=-30m

The index_earliest will make the search only look for data that has been indexed in the last 30 minutes.

If you need even quicker results, you can further qualify the dashboard search such as index=xyz _index_earliest=-30m ttt=abc OR ttt=def OR ttt=xyz or you can reduce the fields it returns in the Map Reduce job index=xyz _index_earliest=-30m | fields ttt field1 field2.

I do not recommend a data model because you're getting all the data every 30 minutes. The data model would will take time & compute to rebuild every 30 minutes. Since ALL the data comes every 30 minutes, this would create a point in time where the data model is "stale" or out of date while it rebuilds. The DM approach would also consume unnecessary disk space, because the data from 30+ minutes ago isnt even needed, but it would be in the data model.

I also wouldnt recommend summary indexing because you have to trigger a summarizing search before the benefit is realized, and that would also take time, and consume additional disk space that is not needed.

[kamlesh_vaghela]

Hi nilaksh92,

As per my suggestion, you have to use datamodel. It will help you in performance as well as for calculation logic also.

http://docs.splunk.com/Documentation/SplunkCloud/latest/Knowledge/Aboutdatamodels

http://docs.splunk.com/Documentation/Splunk/6.6.0/PivotTutorial/Buildtutorialdatamodel

For searches, I will not suggest you use real-time searches in any dashboard but you can achieve the same thing by refreshing your search of panels in 30 sec interval.

So please let me know if you want to go with this approach.

Thanks
Kamlesh

[nilaksh92]

Hi Kamlesh,

Here Is my query

|inputlookup abc|join type=inner max=0 aaa[ search index=xyz | rename ttt as "aaa"]

How to optimize this kind of query?

Above query is taking data for full day.

Please help me out in this.

[kamlesh_vaghela]

Hi nilaksh92,

For the optimization, can you please let me know what are the fields you need from "lookup" and from "sub-search". My concern is about to know how we are using fields after the join. For example, for displaying table, chart or any calculation, etc. If we have the search for the chart, which uses aggregated data then we can use below search:

index=xyz | stats count by ttt | lookup abc as ttt | table abc field1_from_lookup field2_from_lookup

https://docs.splunk.com/Documentation/Splunk/6.6.0/SearchReference/Lookup#Usage

Can you please provide more information regarding our search?
And your data is coming in every 30 sec. Am I right??

Thanks
Kamlesh

[nilaksh92]

Hi Kamlesh

After this I am doing lot of calculation along with aggregation( Regex also). Then I am using the same in drill down panels.

Thanks
Nikks

[nilaksh92]

Hi Kamlesh,

Please guide me on this issue.

Thanks and Regards
Nikks

[kamlesh_vaghela]

Hey Hi Nikks,

Yes, sure I'll guide you...
I have discussed your issue with my colleagues. We can achieve better performance by twisting search and by doing some sort of performance tuning techniques in your search. As per your given information, your search has lots of aggregation & calculation. But we need to understand your requirement, sample data, and your existing search. so Can you please give me your search. so we can work on that

Thanks
Kamlesh

[nilaksh92]

Hi Kamlesh,

If you dont mind can you share your whatsapp number? So that I discuss in detail.

Thanks
Nikks

[kamlesh_vaghela]

Hi Nikks,
Sure.
Ping me on my registered email-id.

I'll send share my contact number in email. 🙂

Thanks
Kamlesh

[nilaksh92]

Hi Kamlesh

I am not able to see your registed id. Could please mention user id over here?

Thanks
Nikks

[kamlesh_vaghela]

Hi Nikks,

mail me on kamlesh@crestdatasys.com.

Thanks
Kamlesh

[nilaksh92]

Hi Kamlesh,

Thanks you so much. I have dropped a mail to you.

Thanks
Nikks

[adonio]

i think the question here is: " how i make my searches return results faster?" if that is the case, there are plenty of answers here in this portal. can you share the searches you are using today?
another point is "real time" what it the requirement? do you use real tome searches?
splunk has another ways such as summary index, accelerated reports and more to overcome this challenge
hope it helps

[nilaksh92]

I am getting data for full day (@d) and after every 30 seconds it is getting refreshed. So what should I do for this?

<search> 
<query> my search </query>
<earliest> @d </earliest>
<latest> now </latest>
<refresh> 30s </refresh>
</search>

[rjthibod]

You have to share more about the data in order to get a helpful answer.

What query or queries are you running? What fields are you looking for?

Basically, you will get the best answer if you can share the actual query. If you can't share the actual query, we need to know some somethings about the query, i.e., are you searching for specific fields or values, what transformations you are running, etc.