Maybe NFSv4 has gotten some advanced black magic or something, but you usually can't really cache stat calls, because the files are changing, so the answers are always changing. The goal for splunkd is to keep up-to-date on every file in the monitor window, and NFS doesn't have any features like dnotify/inotify/fam/gamin whatever to acquire this information in a more aggreate or server-generated fashion. This I/O pattern is simply not one that's commonly followed by most apps and so there isn't an optimized set of system calls for it. ... View more

This issue is a search time problem, so it depends upon the search. Splunk has to return all the events in time order, so if there's a very large number of events during the same second being returned by a given search, splunk has to perform a sort of all those events based on subsecond. If the value of seconds differs, there is no large-scale memory intensive sort required, because they are already ordered as they are retreived. This behavior occurs per-indexer in distributed search so distributed search limits the problem by the number of nodes. ... View more

Yeah if you just want to bound, rather than get exact second, this is better. ... View more

In the example given, there would be no where clause. It simply becomes a constraint on the base search, which is vastly more performant. The base search command can and does perform numerical equality testing as well as numerical comparisons. ... View more

There's a clumsy hack to use another locale, if necessary. You can copy the splunk locale directory to another locale name to get english text from the splunk app with other settings for an alternate locale. Our user surveys suggested this was unacceptably confusing so we did not make it an easy action out of the box. ... View more

Not really sure what component is not liking its input. My first guess is the python mail sender is failing to handle the raw data being passed to it by splunk. Does this command work with other data? If you output that data to a csv file, is there anything strange in it? Is there more about the error, for example an indication of what component is emitting the error, or more about how the error is presented? Is it possible that those are "smart quotes" somehow? ... View more

This really needs to be a support case with http://www.splunk.com/support If this is setting up the functionality for the first time, I would recommend reviewing your alert conditions to see if they are getting met. If this is something which previously worked and has now stopped, I would recommend reviewing what has changed recently in your environment as well as calling into the support line. ... View more

I'm not sure what you mean by "deployed from the server". I'm surprised that changing the values in log.cfg directly has no effect upon restart. You might check to see if there are any warnings about setting not being honored. In the code, maxBackupIndex defaults to zero, so I suspect this is an environment problem. ... View more

This is pattern is possible using a subsearch, with kind of hack of a special field called 'search', mentioned over here: http://answers.splunk.com/questions/3471/using-a-subsearch-to-get-the-time-of-an-event ... View more

Sure, but Local System isn't a domain user. ... View more

Splunk does use the platform-provided tmpfile() function for some purposes. It's possible that tmpfile() may be configured in your platform to use /var/tmp. The tmpfile() call ensures that the file is deleted on close or program exit. This is a platform guarantee, not a Splunk code guarantee. Overall, it's quite unlikely that any pile-up is caused by the Splunk built-in operation. You may want to review any scripted inputs you have, or alert scripts. I would also recommend using system tools such as lsof, auditing, and fuser to evaluate which processes have these temp files open, if any, and investigating those processes. ... View more

splunkd has some failure modes I'm sure for searches above some size, even if just performance issues or ungainly handling, but several hundred kilobyte searches work just fine. The problem you're seeing is happening in splunkweb. I suspect it's some kind of quoting failure, rather than a length issue. If the exact string isn't sensitive, I'd love to reproduce and file the bug. A support ticket would be appropriate. As a lame workaround, manually adding the search to a savedsearches.conf file will likely bypass the problem. ... View more

splunk cmd has special behavior for running executables, (it looks in a particular directory) but python doesn't. Python myscript.py will look in the current directory, wherever that is. I recommend using the environment. For a batch file, %SPLUNK_HOME%\etc\apps\myapp\bin\foo.py ... View more

Splunk doesn't have a built-in way to launch a script in another user context. I think generally the operating system's security model isn't going to provide any possible way for the "local system" user to transparently launch a process as a domain user. Maybe it's possible by drastically relaxing a lot of security guarantees -- I'm no windows expert -- but I wouldn't recommend that. Here are some options: Run splunk as a domain user with the right permissions. (Problem, this can be problematic with an indexer, since now all its files are owned differently. Fixing this can be as simple as changing ownership on all the files, and changing the user that the splunk services run as, but backups are appropriate for such experiments.) Run your script externally to splunk, and write to log files that splunk tails. This has some nice properties of decoupling your code from splunk so it's easier to inspect, debug, etc. It also makes it much easier for you to always get complete data and never get duplicates. It has the additional burden of ensuring your code is running. Python's logger framework is a handy way to create the files. ... View more

It seems like the question here is: for events with a PercentProcessorTime field, what quantity are 80 or more? If wrong, maybe a rephrase of the question might clarify. Try something like: source="wmi:cputime" daysago=30 PercentProcessorTime=* | eval cpu_business=if(PercentProcessorTime>=80, "busy", "not_so_busy") | stats count by host, cpu_business ... View more

Depends how fast your logs are growing! ... View more

The documentation is incorrect. I'm not sure exactly what it was trying to convey, but it will get some edits. ... View more

It could be added. Diag can run whether or not splunk is running, so it's launched externally, but there's no reason we can't add launching it internally as well. I'd been imagining a flag to diag that uploads it directly to a case, but I'm not sure if that's really what's wanted here. ... View more

Of course, if you're using a community app, and you understand the app, but are having a general Splunk problem, feel free to use all the normal Splunk channels: chat, tickets, answers, etc., just be sure to focus on the Splunk issue. ... View more

I know the commercial side of DRBD built an example configuration to try to do some co-selling with us. I think we had too many projects on our end and didn't pursue so far, but if you have a commercial relationship with that enterprise, I bet they could dig it up. I haven't used drbd since 2001 so I'm out of date. Things to be aware of: Splunk can shut down slowly if it's being fed by light forwarders with large files. If you want to avoid missplit events etc, it's best to let this finish. Splunk however should always have a searchable index regardless of how it shuts down. Splunk doesn't ensure indexes are locked against other splunk indexes, because that's not really a supported model (you can finagle it in some cases). Multiple splunks writing to the same hot buckets will be v. v. bad, while multiple splunks rolling the same buckets will be bad. Be sure your favourite cluster manager is capable of avoiding this case very well. ... View more