Advanced view using post processing is not displaying a graph

[jedinerd]

I have followed the documentation to create an advanced view that should utilize post processing to generate multiple graphs from a single search. My data source is the splunk jmx plugin so the data is already in 1 minute buckets and the data is already ready to graph. The following is my search that generates the graph i would like on my page. It works great and makes the graph i need.

index=jmx sourcetype="cassandra_jmx" | rex field=_raw "mbean=\"(?[a-zA-Z0-9.]*):" | search myMbean=org.apache.cassandra.db | eval coname = myMbean."-".columnfamily | search columnfamily=EventCounter | timechart span="5m" avg(RecentWriteLatencyMicros) by coname

I have taken 'index=jmx sourcetype="cassandra_jmx" | rex field=_raw "mbean=\"(?[a-zA-Z0-9.]*):" | search myMbean=org.apache.cassandra.db | eval coname = myMbean."-".columnfamily' and used that as my saved search and then used search columnfamily=EventCounter | timechart span="5m" avg(RecentWriteLatencyMicros) by coname as the search for generating the graph.

Currently, i have eliminated all of the syntax errors in my view and it loads, says that it is waiting for data, but displays nothing. Here is the xml that my view consists of :

<?xml version='1.0' encoding='utf-8'?>

mhogg cassandra test

Here is an example of the data that is returned from the combined search string with the timechart removed:

host=blah.blah.blah,jvmDescription="pdb",mbean="org.apache.cassandra.db:columnfamily=EventCounter,keyspace=rli,type=ColumnFamilyStores",LiveSSTableCount="5",LiveDiskSpaceUsed="457363579",RecentReadLatencyMicros="503.5",MemtableDataSize="33228",RecentWriteLatencyMicros="10.333333333333334"

Any help is greatly appreciated!

[Damien_Dallimor]

Great to see you're using Splunk for JMX 🙂
As a slight aside to your question, I see you are performing some search time transforms on the "mbean" field.
As you can see from the above raw data, by default, Splunk for JMX writes out the full canonical mbean name(domain:properties).

Splunk for JMX has a facility to plugin your own custom output formatter, so you could write out the mbean name in a more convenient format and alleviate the need for search time transforms.

Here is an example I created in quite literally a couple of minutes that outputs the MBean name in a more tokenized manner. As you can see, I've seperated out the mbean domain and properties components into their own fields.

host=blah.blah.blah,jvmDescription="pdb",mbean_domain="org.apache.cassandra.db",mbean_property_columnfamily="EventCounter",mbean_property_keyspace="rli",mbean_property_type="ColumnFamilyStores",RecentReadLatencyMicros="503.5"

View the pdf user guide for info on how to do this.

In the latest release of Splunk for JMX, version 1.3 , there is a custom formatter available that will achieve the above output format.

In your config XML file , just add this formatter declaration :

<formatter className="com.dtdsoftware.splunk.formatter.TokenizedMBeanNameFormatter" />

Damien