(Near) Realtime Search - Simple XML Dashboard

[madhav_dholakia]

Hello,

I am creating a Simple XML dashboard (with panels refreshing every 10 or 30 seconds), replicating a Live Telephony System Dashboard (which refreshes every 5 seconds).

A python script is fetching data from Telephony System using RestAPI every 10 seconds and pushes to Splunk using HEC.

Panles on Splunk Dashboard works ok most of the times, unless there are multiple live calls going on at a time or multiple users are accessing this dashboard. In later case, searches are taking long to complete (because they are in queue due to multiple users seeing the dashboard at the same time?).

What is the best way to handle this scenario?

Thank you.

[PickleRick]

Well, you're trying to force Splunk to do something it's not designed to do.

You can have real-time search with Splunk but the real-time searches are not a very good solution and there are very limited use cases when their use is reasonable. They have their limitations and they hog up resources (each real-time search blocks a single CPU _on every participating indexer_).

You can use a report with a minute schedule (but bear in mind that depending on the load, a search can be delayed or skipped altogether!) or create a dashboard with a relatively frequent refresh period. But all those walkarounds are fairly "heavy" for your environment since you're spawning a new search often (and spawning a search is a relatively complicated process).

Splunk Enterprise is not really a real-time monitoring solution (even though it has some functionality that does real-time stuff) so forcing it to do something like that might end in disappointment.

[madhav_dholakia]

thanks @PickleRick , that was helpful - for now I am settled with below approach and will monitor if this is causing any more issues:

I have created a saved search (Report) to run evey minute and then dashboard panels are using 

| loadjob

and refreshes every 15 seconds. 

[gcusello]

Hi @madhav_dholakia ,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the contributors 😉

[gcusello · Accepted solution]

Hi @madhav_dholakia,

if you have a real time dashboard continously used by many users you kill your system.

In this case, use a different approach:

create a report containing the information to display and then in the dashboard display the report using loadjob (https://docs.splunk.com/Documentation/SplunkCloud/9.1.2308/SearchReference/Loadjob).

this is an old post but the solution is still valid: https://community.splunk.com/t5/Dashboards-Visualizations/What-can-we-use-to-replace-loadjob-based-d...

Ciao.

Giuseppe

[madhav_dholakia]

thanks @gcusello - I will give it a try and will update result/query here.

Thank you.

[madhav_dholakia]

Hi @gcusello - I am not sure if I can schedule Splunk Report to run every 10 seconds? I added this cron expression in  Report Schedule but it says "Invalid Cron"

0/10 0 0 ? * * *

[gcusello]

Hi @madhav_dholakia,

you cannot schedule a cron every 10 seconds, but every 10 minutes using something like this:

*/10 * * * *

Ciao.

Giuseppe

[madhav_dholakia]

yes, but then it won't be a near realtime which is required. any other option to recreate this dashboard so that we can get data refreshed every 10 (or 15) seconds?

Thank you.

[gcusello]

Hi @madhav_dholakia,

yes, you can refresh the dashboard panel every 10 seconds, but, really your data changes every 10 seconds and you have all the data variations and they are mandatory to be displayed every 10 seconds?

Then, does your search give results in less than 10 seconds?

I hint to re-analyze your needs to define a more real requisite to implement.

Otherwise, continue to use real time searches giving much more resources (CPUs on Indexers and Search Heads) to your infrastructure: remember that a search takes a CPU ob SH and on IDX and release it when finishes, in your case never, so if your have 20 user that use the dashboard you have to add at least 20 CPUs to SH and to IDXs.

Ciao.

Giuseppe

[madhav_dholakia]

thsnks @gcusello - I got your point. why this needs to be almost realtime is, because it includes Live Calls Data so as soon as a call is landed, it should be reflected on this dashboard. 

for now, as you suggested, I have created a saved search (Report) to run evey minute and then dashboard panels are using 

| loadjob

and refreshes every 15 seconds. 

Alternately, if I create a Splunk User with only (and limited) access to this dashboard and then it can be used by anyone who wants to access this dashboard - in this case I would expect multiple search job requests won't be submitted hence this should not cause performance issues I am seeing currently - is this understading correct?

Thank you. 

[gcusello]

Hi @madhav_dholakia,

your process is ok, only one questions: if the report is refreshed every minute, whay do you refresh panel every 15 seconds? iy's unuseful!

let me know if I can help you more, or, please, accept one answer for the other people of Community.

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉

[madhav_dholakia]

thanks @gcusello - only reason to refresh panels every 15 seconds is to get the results from last executed report run in 15 seconds, rather than to wait for a minute to see the latest results on dashboard once report run is completed.

Thank you for your help on this.