We have a dashboard that I would like to use tstats to generate the data, and run a search ever 2 minutes using tscollect. The problem I have discovered is that since each search head in the pool saves tsidx files to it's own local tsidxstats directory, you only get a partial view of the data. Any suggestions? ... View more

I've written an external lookup script that makes a rest call to an API & returns data. The API destination requires going through a firewall, so we are only allowing our search head to make the call. When I use the lookup using tstats on the search head, the lookup executes very quickly. When I use it against searches that pull data from our indexers, the indexers appear to be running the script. This fails, however, due to the firewall not being open for the script to run. I have local=true set on the lookup command, and also used localop Search: | head 1 | localop | lookup local=true XXXX fieldA | table fieldA,lookupvalue From the search inspector: This search has completed and has returned 1 result by scanning 671 event in 1,141.566 seconds. Error message in the search inspector for every indexer: Script for lookup table 'XXXX' returned error code 1. Results may be incorrect. Any suggestions? My next step is to block replication of this to indexers. ... View more

After upgrading to Splunk 5.0, we started seeing the following error message in a popup when loading the web intelligence landing page: Splunk encountered the following unknown module: "GenericFTR" . The view may not load properly. It looks like the CSS is broken as well, as the app reverts to the Splunk color scheme, not the lighter toned web intelligence. Seeing this in the splunk web access logs: xx.xx.xx.xx - xxxx [08/Nov/2012:00:45:35.378 +0000] "GET /en-US/modules/@140868:123300/GenericFTR/GenericFTR.js HTTP/1.1" 404 1192 "https://*****/en-US/app/webintelligence/overview" useragent string any suggestions? ... View more

We are using Splunk to implement file integrity monitoring, but our security team has a requirement that I'm having trouble implementing. For monitoring file integrity, we are using fschange. For monitoring /var/log and other items we are using 'monitor'. We have SSL between Splunk forwarder & indexer, as well as event hashing on key log sources. The scenario, however, that I need to detect is this: Splunk monitors /var/log/messages Splunk uses fschange to perform FIM on /etc/* Bad actor accesses server, and edits /etc/hosts Bad actor accesses server, and removes log lines from /var/log/messages (assume file is already indexed by splunk) FSchange will notify that the file was edited - but how do we detect that log lines were removed from the log file? I've tried using send full event in FSchange, but that just re-indexes the entire file. ... View more

Currently, apps on our universal forwarders are controlled by the deployment server, and the forwarder RPM & deploymentClient.conf are installed by Puppet. Even with this setup, you can still put an app in the local forwarder's app directory, and the forwarder will run it. What can I do to only allow apps from the universal forwarder to run? ... View more

I'm running a transaction command against IP's in apache logs. If I'm running a real time search, will the transaction be updated as more access logs from an IP come in? ... View more

I have the following xml <module name="HiddenSearch" layoutPanel="panel_row2_col1" group="XXX" autoRun="True"> <param name="search">apache log search method=get OR method=post | eval responseStatus=case(match(status,"2\d\d"),"OK",match(status,"3\d\d"),"OK",match(status,"4\d\d"),"ERROR",match(status,"5\d\d"),"ERROR") </param> <param name="earliest">-30m@m</param> <module name="HiddenPostProcess" layoutPanel="panel_row2_col1_grp1"> <param name="search">timechart count by status</param> <module name="HiddenChartFormatter"> <param name="charting.chart">column</param> <param name="charting.chart.stackMode">stacked</param> <param name="charting.legend.placement">bottom</param> <module name="JobProgressIndicator"/> <module name="FlashChart"> <param name="width">100%</param> <param name="enableResize">True</param> <module name="ConvertToDrilldownSearch"> <module name="ViewRedirector"> <param name="viewTarget">flashtimeline</param> </module> </module> <module name="ViewRedirectorLink"> <param name="viewTarget">flashtimeline</param> </module> </module> </module> </module> <module name="HiddenPostProcess" layoutPanel="panel_row2_col1_grp2"> <param name="search">top responseStatus | where match(responseStatus,"ERROR") | gauge percent 0 5 10 100</param> <module name="HiddenChartFormatter"> <param name="charting.chart">fillerGauge</param> <param name="charting.chart.style">shiny</param> <param name="charting.chart.orientation">x</param> <param name="charting.chart.usePercentageRange">true</param> <param name="charting.chart.usePercentageValue">true</param> <module name="JobProgressIndicator"/> <module name="FlashChart"> <param name="width">100%</param> <param name="enableResize">False</param> <module name="ConvertToDrilldownSearch"> <module name="ViewRedirector"> <param name="viewTarget">flashtimeline</param> </module> </module> </module> <module name="ViewRedirectorLink"> <param name="viewTarget">flashtimeline</param> </module> </module> </module> </module> What happens is that the gauge from TOP is correctly displayed, but the timechart only gets 5 minutes of data. It should be displaying all 30 minutes of data. Any suggestions? ETA: If you change the timechart to chart count by _time then the chart will only display a 5 minute graph, vs a 30 minute graph with 5 minutes of data with timechart. ... View more