Splunk provides great tools it its web interface for quickly creating an application, saving searches, adding dashboards, etc. What tools or best practices support development and maintenance of larger applications?
This summer I was tasked to create a "metrics application," which will expose summary data extracted from our log files. The list of metrics is 15 pages long.
Each individual view was easy to create, but it became difficult to keep them consistent, particularly when changes were requested.
Most of the views have a similar structure, but differ in the details: the form fields, the searches, the chart labels, etc. There seems to be no way to abstract the overall page layout away from the detailed contents of each page. The advanced XML views, in particular, expose every tiny detail of the page layout; any change must be carefully repeated in every advanced XML file.
I would welcome hints from others who have developed large Splunk applications.
Joel,
[Splunk Developer Guide - Building Splunk Solutions]1 contains many good and proven app building practices discussed in the context of real-world apps. Check it out!
Also, these days Simple XML (not Advanced XML) is a way to go.