I have a search that should display a percentage of bad events compared to the good events over a time period. I wanted to migrate this search from a scheduled search to a real time search and for some reason that I can not figure out why the subsearch I use to calculate the total number of events returns a different amount (which varies quite a lot) when I convert it. Here is the search: sourcetype=xy source!="good_events" | stats count AS bad_hosts | eval all_hosts = [ search sourcetype="xy" source="good_events" | stats count AS good_hosts | rename good_hosts AS query ] | eval diff = round(bad_hosts/good_hosts*100, 2) Is it possible that the subsearch is run twice when the realtime search starts and backfills historical events? Or do you reckon the subsearch isn't necessary for this type of search ... that doesn't answer why switching to real time yields completely different results though. Thanks Chris ... View more

Hi We had a little discussion about splunk architectures and how roles should be designed. I was wondering if anyone has any hints/best practices to define roles. My approach so far was to build indexes that are then assigned to roles in authorize.conf using the 'srchIndexesAllowed'. I try not to use the 'srchFilter' parameter if I can avoid it. I think that 'srchFilter' will have performance impacts and I created a mess once using inheritance and different search filters. Regarding the capabilities I usually derive from the 'user' or 'power' role which are fine for 95% of the roles I have to define. Feedback is appreciated Chris ... View more

Is it somehow possible to trigger a scripted alert manually? We have some alerts that usually run once a day, sometimes there is a need to run the scripted alert more often ( to either manually check if a situation occured based on some information the users receive or to simply check whether the shell script is still working correctly). I know, that I can change the alert to run every minute in the manager and then change it back to what the intervall was. If I just click on the 'run' action in the manager only the search is run but the shell script (or email-action) is not triggered. Is there a way to do this? Chris ... View more

We have a couple of files, that are rotated by copying and then truncating the original file (so no new inode is created for the monitored file). We use lagadm on Solaris but I think there will be a lot of people using logrotate on Linux using the copytruncate option for apache logs. We have noticed, that for those files it happens that we get the following error message: 06-05-2012 09:46:01.761 +0200 ERROR TailingProcessor - Ignoring path due to: failed to compute crc for /path/to/my.log (method: 0, hint: Error 0) This error message occurs when the log rotation is in progress. If the error message is written by splunk the affected file will not be indexed from then on. Is there a recommended way how logfiles should be rotated? I guess, that for apache using piped logs is the solution (http://httpd.apache.org/docs/2.4/logs.html#piped). But what about other programs? Anyone? Thanks Chris ... View more

Is there a recommended/optimum size of all the indexes that one indexer can handle? I have not analysed this, but indexers that have 500gb of indexed data restart faster that those with 2tb. I am wondering if anyone has run into any problems because of this. Or if others have rules like "whenever our indexers manage more than xy gb we add another indexer". Thanks Chris ... View more

Is there a possibility to limit the filesize for csv files that are generated by the outputcsv command? ... View more

I just noticed, that I have to add the 'typer' command to a search in a dashboard so my cusotm event renderer is active for non admin Users. Is there some documentation that explains when and how eventtypes are recognized? Thanks Chris ... View more

Hi we get the following error message from time to time on our Indexers (Solaris 10 x86 64bit Splunk 4.2.5 or 4.3): ERROR TcpChannel - Fatal error on accept socket: Software caused connection abort If this occurs netstat shows that the port is still listening: ~:netstat -an | grep 99 .9997 *. 0 0 49152 0 LISTEN But i do not get a connection when telneting to the port, the forwarders can't connect to the indexer anymore and it has to be restarted. Is there anyone else with the same problem somewhere? Is there a fix for this? Thanks for helping Chris Edit: We have downgraded to 4.2.4 on one of our indexers, we haven't had the problem since then. But it has only been running for a couple of hours now. I will update this when more time has passed ... View more