Advanced XML views: foreach iteration over search results and including parameterized sub-view for each search result

[traugust]

Hi community,

I did da lot of research during the last weeks on how to implement a report for the following scenario:

• I have a dynamic list of subjects on which I want to run a quite complex sub-report (The report is an advanced XML view with 10+ charts and a custom layout, filling an A3 page for each subject when printed)
• The dynamic list of subjects (could be servers, users, or anything else ) is the result of a splunk query
• The sub-reports shall all be displayed in one View -- to be able to deliver it as a scheduled PDF

After three weeks I have the following research results:

• There is currently no foreach construct in Advanced XML
• There is currently no pattern to include a view in another view
• Subsequently, There is no 'standard' construct for parameterizing sub-views

What I have come up with after three weeks is the following approach to implement the complex report:

• As there is no foreach construct to iterate over search results, I do the search in advance and then use a regex replace pattern to generate the outer enclosing view XML, that runs the sub-reports on the particular search result items
• As there is no particular include construct for including a view as sub-view into the current view, I use the IFrameInclude module for including the sub-view into the main view
• The particular subject is passed to the sub-report via URL-encoded HTTP GET Parameters:<module name="IFrameInclude"> layoutPanel="messaging"><param name="src">/app/xxx/dev_single_subject_report?loggerId=myFoundLoggerId01</param></module> -- It seems that the XML display of code fragments gest somewhat mangled, if in doubt, look at the source code of this post to see what I meant
  
• The sub-view uses the URLLoader module of the SideviewUtils app to make the HTTP GET Parameters available to the sub-view context
• The sub-view runs charts on saved searches, which are then post-processed using SideviewUtils PostProcess module to filter the saved search result for the particular subject passed to the sub-view

This approach has the following disadvantages:

1. Due to the use of a beforehand search for the subject list of the outer view, ith dynamic is somewhat 'limited' to the manual updates of the view XML whenever the subject list changes.
2. Due to the use of IFrame includes the Scheduled PDF reporting is totally fragile: The emedded firefox does not correctly report when the Page has finished rendering -- it reports ist readiness whenever it has finished rendering a HTML Page containg 20+ IFrames . The Ste of the IFrames is not taken into account for this statsus and thus the page gets printed to PDF before the IFrames have finished rendering. To fix this problem I had to patch the Python code of pdfhandler.py to specify the option -printdelay 500 (To be able to use a 500 second printdelay, the javascript code of the cdlineprint firefox plugin must be patched, as it normally limits the value of this option to 120 seconds). After this patch, the PDF server unconditionally waits 5000 seconds for firefox to finish rendering the iframes and then starts to print the firefox page to PDF. To be able to use this for Scheduled PDF reports, several timeouts along the call chain of the scheduler need to be adjusted to take the long duration of PDF rendering into account. I'm not sure whether the next update might break this fragile chain, but that is the best approach I was able to find
3. The JSChart module sporadically fails to render the result of a post-processed saved search, which sporadiacally results into the SideviewUtils HTML module displaying the search result and the JSChart radial gauge next to it (rendering the same search result) failing to even show the result. Not very useful ina fully-automated scenario. This has been confirmed by Splunk to be a bug in JSChart [1]

I hope some splunk experts in the community have some Ideas on how to improve this approach or show a completely different approach

[1] :http://splunk-base.splunk.com/answers/46481/jschart-using-sideview-in-431

[traugust]

Could work, but I assume this to be alot of effort until it works seamlessly (I did not write a custom search command yet, so I am not sure how much work that would be).
My intention was to use 'all-stock' splunk functionality for my reporting, as I want to have power-user being able to copy the template and build their own reports on top of this. Thus shell-scripting and stuff sound a bit complex for me.

[sideview]

---

1) Multiplexer module.

A long time ago I created a prototype module called "Multiplexer" for use cases like yours.

Although it's still very much marked as a prototype module, it does actually ship in Sideview Utils 2.0.X, and I'd be interested in your details and requirements to see if it really can be a good solution for you.

How you use Multiplexer, is you give it a param called 'field', and you make sure that it only has one downstream module, or child. Note that although it has only a single child, that one child can have as many other children and descendants as you like.

How it works, is that it will inspect the values of the given field, across the current search result. And for every value in every row that it sees, it will clone that single sub-branch of modules. If the field param is "myField", then $myField$ can be used within the downstream param values, and for any given multiplexed child, it will refer to that row's value for the myField field. It's a little brainbending but it makes sense quickly.

The good news: It works great with HTML modules and a number of other simple modules. Also, it can be paged, meaning you can put a Multiplexer downstream from a Pager module, and then it will only 'multiplex' 10 at a time. And clicking page 2 will then 'multiplex' the 11th through 20th values.

The bad news: It does NOT work with the FlashChart module because the javascript-to-flash communication code gets tangled and throws errors when things get cloned. Also, I'm afraid I have not yet tested whether the JSChart module has any similar shortcomings.

---

2) IFrameInclude.
Since you're using Sideview Utils already, I recommend simply putting an <iframe> tag in an HTML module. IFrameInclude does some strange things which don't add much, and in fact often get in the way. However note that the HTML module will re-render it's HTML with every context push, so you'll be happiest with this approach if there's only one push and the overarching view is not very interactive.

---

3) The JSChart bug. Unless I'm mistaken it was me that initially reported this bug to Splunk, and I think i've already patched it if you're using the latest 2.0.X version of Sideview Utils . I will however double check that and update this answer.

---

4) dynamically creating views.
It's been done. Yes it's sad and unfortunate that there's no way of including a shared fragment of view xml. However some cool things have been done dynamically generating XML, adding, editing, deleting xml files, and then hitting the refresh endpoint.

---

5) print delay.
It's an interesting use case, but I think I can probably wire URLLoader into the PageStatus system, so that when URLLoader sees that hte parent page is another Splunk view on the same domain, it can register itself as another player in the game, and you would then be able to take that extra sleep out. I'll add an item to my queue to investigate and possibly implement in a future version of Utils.

[cphair]

Interesting question. Is the requirement to have a single view or simply a single PDF? If it's the latter, you might consider a custom search command to iterate over the results of the generating search and create each view separately. Then you could merge the resulting PDFs, either with a command line tool or in Python itself (I found some code samples with a quick search), and email it through the script, or maybe generate an alert with a link to the file location. I've never tried anything like this, so I don't know how feasible it is, but maybe it's helpful.