I tried to diagnose the search, and found that the following search:
index=jmx sourcetype=jmx mbean_domain="java.lang" mbean_property_type="OperatingSystem" OR "Runtime" host="*" jvmDescription="*" | stats max(jvmUptime) as uptime max(cpuTime) as cputime max(processors) as cpus by jvmDescription,_time | streamstats current=f global=f window=1 first(uptime) as next_uptime first(cputime) as next_cputime by jvmDescription | eval d_uptime=uptime-next_uptime | eval d_cputime=cputime-next_cputime
Returned the following kind of gaps:
_time jvmDescription uptime cputime cpus next_cputime next_uptime
11/13/12 6:38:19.850 PM qa2-zk-01-n1 1033382707
11/13/12 6:38:19.856 PM qa2-zk-01-n1 440610000000 2 1033382707
11/13/12 6:39:19.822 PM qa2-zk-01-n1 1033442680 440610000000
11/13/12 6:39:19.828 PM qa2-zk-01-n1 440640000000 2 1033442680
11/13/12 6:40:19.824 PM qa2-zk-01-n1 1033502685 440640000000
11/13/12 6:40:19.834 PM qa2-zk-01-n1 440690000000 2 1033502685
11/13/12 6:41:19.797 PM qa2-zk-01-n1 1033562654 440690000000
11/13/12 6:41:19.802 PM qa2-zk-01-n1 440730000000 2 1033562654
11/13/12 6:42:19.807 PM qa2-zk-01-n1 1033622669 440730000000
11/13/12 6:42:19.813 PM qa2-zk-01-n1 440760000000 2 1033622669
Is this because I am using the prependDate attribute on my formatter?
That is most likely the cause.
Try it without the prependDate setting , that optional property is there should you require a custom date format for you MBean output for whatever reason.
..or, take off the milliseconds pattern from the format string ("SSS")
If you do require to use the prependDate option , then you can have a seperate JMX config file for these specific instances.
That is most likely the cause.
Try it without the prependDate setting , that optional property is there should you require a custom date format for you MBean output for whatever reason.
..or, take off the milliseconds pattern from the format string ("SSS")
If you do require to use the prependDate option , then you can have a seperate JMX config file for these specific instances.
Thank you. I removed that attribute, restarted the Splunk universal forwarder on the server with the target JVM, and then the CPU Usage dashboard panel started to return results.