After upgrade to 5.x, the | rest command started working. ... View more

I now have paging and swapping charts integrated into the *nix app. It seemed like the best fit on the Memory by Host view. I modified the view to include two more charts on row 3, moving the Physical Memory by Host chart to row 4. The dropdown selector also works with the paging charts. By default it will show all hosts paging stats, and if you select a single host, it will reflect that one host. The XML for my charts is below: <module name="HiddenSearch" layoutPanel="panel_row3_col1" group="Average Paging Out in KB/s by Host" autoRun="False"> <param name="search">index="os" source="vmstat" host=$host$ | multikv fields pgPageOut | streamstats current=f global=f window=1 first(pgPageOut) as next_pPO by host | eval podiff = next_pPO-pgPageOut | eval podiffps = podiff/60 | timechart avg(podiffps) by host</param> <param name="groupLabel">Average Paging Out in KB/s by Host</param> <module name="HiddenFieldPicker"> <param name="strictMode">True</param> <module name="JobProgressIndicator"> <module name="EnablePreview"> <param name="enable">True</param> <param name="display">False</param> <module name="HiddenChartFormatter"> <param name="charting.secondaryAxisTitle.text">Avg Page Out KB/s</param> <param name="charting.chart">line</param> <param name="charting.chart.nullValueMode">connect</param> <module name="JSChart"> <param name="width">100%</param> <module name="ConvertToDrilldownSearch"> <module name="ViewRedirector"> <param name="viewTarget">flashtimeline</param> </module> </module> </module> <module name="ViewRedirectorLink"> <param name="viewTarget">flashtimeline</param> </module> </module> </module> </module> </module> </module> <module name="HiddenSearch" layoutPanel="panel_row3_col2" group="Average Swapping Out in KB/s by Host" autoRun="False"> <param name="search">index="os" source="vmstat" host=$host$ | multikv fields pgSwapOut | streamstats current=f global=f window=1 first(pgSwapOut) as next_pSO by host | eval sodiff = next_pSO-pgSwapOut | eval sodiffps = sodiff/60 | timechart avg(sodiffps) by host</param> <param name="groupLabel">Average Swapping Out in KB/s by Host</param> <module name="HiddenFieldPicker"> <param name="strictMode">True</param> <module name="JobProgressIndicator"> <module name="EnablePreview"> <param name="enable">True</param> <param name="display">False</param> <module name="HiddenChartFormatter"> <param name="charting.secondaryAxisTitle.text">Avg Swap Out KB/s</param> <param name="charting.chart">line</param> <param name="charting.chart.nullValueMode">connect</param> <module name="JSChart"> <param name="width">100%</param> <module name="ConvertToDrilldownSearch"> <module name="ViewRedirector"> <param name="viewTarget">flashtimeline</param> </module> </module> </module> <module name="ViewRedirectorLink"> <param name="viewTarget">flashtimeline</param> </module> </module> </module> </module> </module> </module> ... View more

Adding server specific hosts entries on the search head pool members seems to have corrected this issue. Per the documentation for configuring search head pooling, we changed our alert_actions.conf to point to the hostname of the VIP. This seemed to also cause the PDF server app to make a TCP call to the VIP, just to be routed back to a search head. This was inefficient and sporadic, since if the VIP did not route it back to the originating search head, the PDF generation would fail. Adding server specific hosts entries now allows us to keep the changes made to the alert_actions.conf. Initially I tried using the loopback IP, but since Splunk is not listening on that interface, I changed it to the server IP on eth0, which was successful. An example is as follows: search head pool member A /etc/hosts search head pool member B /etc/hosts Hope that helps anyone else with a similar environment. Don't know if Splunk 5.x would resolve this issue another way, since PDF generation is built in. ... View more

For some reason, the deployment server contained an alert_actions.conf with the server disabled. This has been corrected ... View more