I believe when I was originally shown Splunk I saw real time reporting for NMON charts, but because our instance is on one of our main application servers it was thought that running them real-time would potentially be counter productive. However, I have been asked to show realtime reporitng for NMON and can't get it to work. If I use any of the predefined nmon dashboards I get "Error in 'tstats' command: This command is not supported in a real-time search".
Please can someone confirm if Real-time nmon stats are possible through SPLUNK and provide some guidance as to how to configure.
I lately saw this topic (as it known flagged with Nmon Splunk App), but i wanted to give some more information.
Why most dashboards (eg. UI) use the "tsats" command, and why it is not compatible with real time searches
The Nmon App has been designed to deal with very massive server performance data, potentially with large deployments that means thousands of millions for events with large history and large number of servers.
To do so, it specially massively integrates Data model acceleration that allows incredibly powerful Splunk interfaces, as interfaces also need to offer a large panel of selections to fit the need for most of users, this requires using advanced searches and the use of the tsats command.
The "pivot" command which also implements data model searches and acceleration benefits (this is "the" data model search) can do relative searches AND real time searches, but is also much more limited than tsats or natural SPL language. (but is very simplified and easy to use)
** User Interfaces using real time searches **
A few of available interfaces are available in simplified interfaces that implements pivot command, and so is compatible with real time searches (which is selected by default, these interfaces are much more limited in functions)
These UI are accessible though the "RT" icon behind to metric in home pages:
In future release, i have in plan to create real time UI using pivot command for most metrics. (for future release)
*Custom dashboards for Real time searches or scheduled searches *
In Splunk, It is incredibly very easy to create your own searches, reports or dashboards for real time reporting, you can use pivot model interfaces that will help you building the search, you can use the real time search interface to get the sample request (click open in search) or use natural SPL.
To achieve custom real time reporting, let's say for example that you specifically want to monitor specific servers and specifics metrics, you can:
Create dashboards that uses pivot command in real time searches (as it has been exposed in this topic, real time searches have an important cost)
Create scheduled searches with short time range (every few minutes...) and create your own dashboards based on that searches.
Configure your panels with the panel auto refresh feature:
refresh.auto.interval Number 0
Specifies, in seconds, the refresh
interval. To disable panel refresh,
specify 0 (or a negative integer).
My personal preference goes clearly for the second solution, using this way you can create "quasi" real time dashboards (update every few minutes) that won't cost too much for the Splunk infrastructure, every people that will look at these dashboards won't require running users searches, and Splunk will use these jobs artifacts to serve the page.
It in deed has the price of scheduled searches, but in my opinion this allows building very large dashboards with numerous charts or stats for numerous people with low server costs compared to in line searches. (real time or relative)
It would not even be reasonable to think serving dashboards with numerous charts for users using inline searches without scheduling them..
Finally, your scheduled searches will use the acceleration of data models if you build them from given interfaces or pivot model interfaces, and will naturally benefit from it to reduce their cost or increase their potential.
Real time searches can have real interest for specific needs, therefore building dashboards for users in "real time" reporting does not requires this using Technics exposed above, my personal opinion again.
Splunk offers numbers of architecture possibility, having distributed configuration (Cluster indexers, sh cluster) will definitively increase the interest and power of this, the sh cluster for example will naturally share the scheduling of searches between available nodes, this is extremely powerful.
@john_howley : Could you please update your topic to associate it with the Application tag "NMON Performance Monitor for Unix and Linux Systems", this allows people to easily find Splunk answers associated with the App.
I just added the appropriate app tag
Hi guilmxm. I have tried to re-tag the topic, but it doesn't seem to work. Thanks for all your advice. As i mentioned on an earlier update I did manage to find one of the NMON charts that supported Real Time. I additionally build a custome rdashboard which also supports Real Time but because we're only on a Proof of Concept and the infrasturcture is sitting on out application master node we have decided not to use it in anger at the moment. I like your idea of scheduled search and auto=updating and will have a look into that.
Thank you all for your advice and answers on this topic. It is not 100% resolved in my mind, but I did find one NMON page that I could get to work in real-time mode partially, but enough to allow me to conclude that at least for our Proof of Concept exercise Real-time monitoring is not in-scope due to the restriction around both the splunk installation being on our master application server and limited scope for increasing concurrent search capability.
Please consider this question closed.
I am not completely sure what you are trying to do, but it seems the tstats command that are used in the nmon dashboards is not supported for realtime searches. One workaround is to go into the datamodels for the app and use pivot. You could select the information you need and use realtime in the timerange picker. That can then be saved to a dashboard. There may of course be other ways to get to what you want here, but as jtrucks says it would be nice with some more information.
Can you describe your installation a bit more? Also, please post the search query you use when the error happens.
The are four servers all running 6.2.2 with nmon add on. three of the servers only run splunk_forwarder and are configured to forward events onto the main server. the queries used are the standard NMON dashboards so comprise several queries, but here is one specific one that throws that error. Some are static (Max, Min, Avg) others display over time. The static queries run ok, but will not display real time the queries by time will not run
CPU_ALL(max) from datamodel=NMON_Data_CPU where (nodename = CPU.CPU_ALL) (CPU.hostname=) (CPU.frameID="") (CPU.OStype="") (CPU.hostname="")
No_Filter(CPU) groupby _time, "CPU.hostname" prestats=true span=1s | stats dedup_splitvals=t
CPU_ALL(max) by _time, "CPU.hostname" | fields * | fields * | fields _time,CPU.hostname,CPU.cpu_PCT | timechart
inline_customspan(type=CPU_ALL,earliest=rt-2m,latest=rtnow) useother=f limit=0 avg("CPU.cpu_PCT") AS cpu_PCT, avg("CPU.Idle_PCT") AS Idle_PCT, avg("CPU.Sys_PCT") AS Sys_PCT, avg("CPU.User_PCT") AS User_PCT, avg("CPU.Wait_PCT") AS Wait_PCT by "CPU.hostname"
does this provide enough information?
I agree with woodcock that it should be convertible, but this query is using the data model. Therefore my other comment about using pivot. I have not tested this enough to say anything about performance though. I guess indexed realtime would take some of the load off, if that slight lag is acceptable.
First of all, why are you using RealTime? Unless you specifically engineer your cluster for this, you will regret using RealTime because it is so resourcefully expensive that it will crater your cluster performance. In any case, any "tstats" command should easily be convertible to a slower-running "stats" command. What is your search string?