You might want to look at the Splunk Developer’s Guide and the associated Splunk Reference App, built by a Splunk dev team. It covers application development from getting data into Splunk Enterprise to producing custom visualizations to testing, packaging, and distributing your app. The example shows how to monitor document repositories, allowing you to see who has viewed, modified, deleted, or downloaded records.
The key element of the Splunk developer guidance is the code. The code repos are open, and you can look at the source code of the reference apps and the associated tests. In fact, you can see and replay the code in motion, as it was developed. Full documentation is available and combines design and implementation guidelines; this guidance is written by developers for developers.
If it doesn’t answer your current question you might want to bookmark it and check back occasionally. The team is committed to extending the guidance as the platform evolves to incorporate new features. An update is under development right now that extends the functionality, and takes advantage of the latest Splunk features.
If you prefer, a print copy is available from Amazon.com.
Here is an example using the splunkd_access.log
<dashboard>
<label>API Calls</label>
<row>
<panel>
<chart>
<search>
<query>index=_internal source=*splunkd_access.log | timechart count by method</query>
<earliest></earliest>
<latest></latest>
</search>
<option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
<option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
<option name="charting.axisTitleX.visibility">visible</option>
<option name="charting.axisTitleY.visibility">visible</option>
<option name="charting.axisTitleY2.visibility">visible</option>
<option name="charting.axisX.scale">linear</option>
<option name="charting.axisY.scale">linear</option>
<option name="charting.axisY2.enabled">0</option>
<option name="charting.axisY2.scale">inherit</option>
<option name="charting.chart">line</option>
<option name="charting.chart.bubbleMaximumSize">50</option>
<option name="charting.chart.bubbleMinimumSize">10</option>
<option name="charting.chart.bubbleSizeBy">area</option>
<option name="charting.chart.nullValueMode">gaps</option>
<option name="charting.chart.sliceCollapsingThreshold">0.01</option>
<option name="charting.chart.stackMode">default</option>
<option name="charting.chart.style">shiny</option>
<option name="charting.drilldown">all</option>
<option name="charting.layout.splitSeries">0</option>
<option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
<option name="charting.legend.placement">right</option>
</chart>
</panel>
</row>
</dashboard>
Some sample data with field extractions might get you somewhere faster.