Clustering seeks to regain its configured level of redundancy once outages occur. Therefore avoiding additional resource usage during failures is not realistic. Instead, plan to keep your capacity to a level where you can handle your indexing AND search workload during outage with reduced capacity with the additional load of reduplicating data. Perhaps ballpark 20% contention from reduplicating. At very small cluster sizes where the redundancy factors cannot be satisfied during outages, this specific aspect of concern is not relevant. ... View more

Thanks for the updated answer. I forget when we introduced this, but it was probably before 5.x. ... View more

This answer was indeed correct when provided in 2011 for Splunk 4.x. ... View more

We should have very little care about the distribution; we try to depend only on libc. Please try: splunk cmd $(which env) | grep -i splunk and ls -l /opt/splunkforwarder/lib/*pcre* and splunk cmd $(which ldd) /opt/splunkforwarder/bin/splunkd ... View more

sedcmd should be effective, but modifies _raw. I can't really tell if the problem is the value of _raw or the value of other fields. sedcmd happens in the regexreplacement phase of data handling, and indexed extractions happen before this. ... View more

To add to this, we will treat this as a regression and resolve. Our releases team is doing some long-needed work to modernize package creation, and unfortunately this is an oversight. ... View more

I question the use of unbounded values here, though. If your system can't operate inside a thousand or so simultaneous sockets/requests then it's just going to run out of system memory, most likely, and fall over. ... View more

For clarity, dumbo me, what stanza are these in? ... View more

This warning is correct but pointless for a hot bucket. If Splunk incorrectly attempts to parse hot buckets for the time endpoints in some cases, then numbers which indicate the oldest and newest times of the bucket are not available to be parsed. In other words, for hot buckets, attempting to parse the names will produce this warning. My best information (grain of salt here) suggests that we are not currently aware of the reasons why Splunk would parse hot buckets, and we have not been able to produce the problem via any means in Splunk 6.1. It might be advisable to manually investigate these buckets and see if there's anything unusual about them (missing, truncated files, etc.) that would lead to a problem needing further investigation by Splunk. However it could be that this is just a messaging problem was fixed by changes not specifically targetted at this message between early 5.0.x and 6.1. More information is welcome. ... View more

Splunk does not reindex anything, at least by design or intent. Avoiding reindexing should never require followTail. If you have a scenario where it does, that's a separate question/answer. ... View more

There is one exception; limits.conf settings are only used on the local system, so if you need to alter limits.conf settings they must be done on the indexing nodes. ... View more

Sadly, there is no Sparc hardware on the market with good single thread performance. Sparc core design stagnated in the 2000s, and while clock speeds have raised, per-clock peformance on Sparc cpus is an order of magnitude inferior to modern AMD and Intel chips. Since it is common scenario to run CPU-bound searches on Splunk that will be single-threaded in their criticial paths, and users will expect prompt answers, there is no reason to choose Sparc for splunk indexing or search tasks. ... View more

I should mention that time-limited licenses (TERM) are currently rare outside of pre-sales scenarios. Most customers have perpetual licenses; this is the default scenario. ... View more

In Splunk 6.0+, the btree/fishbucket files have a size ceiling that is maintained. If the fishbucket files grow over a configurable ceiling, they are moved from $SPLUNK_HOME/var/lib/splunk/fishbucket/splunk_private_db to $SPLUNK_HOME/var/lib/splunk/fishbucket/splunk_private_db/save. We then populate a new, empty btree upon request -- entries we actually use are copied from the 'save' version. Ultimately this means that your size will be bounded to 2x the ceiling. If you need to resolve a current problem where the file is very large (let's say 10GB), we will copy your current btree/fishbucket data to 'save', so the space will not be immediately improved. In this case you can resolve your space concerns in the following way: Run your system with 6.0+, and the 'save' subdirectory should be observed with the large file still in it Wait for the system to check every file in the monitored locations, you could validate that it is caught up with an alltime realtime search for that host, or you could just wait a day or so. Stop Splunk Delete, or move out the 'save' subdirectory Start Splunk At this point your disk usage for btree/fishbucket to be constrained to 2x the limit. In 6.0.x we use the maxTotalDataSizeMB value for the [fishbucket] index to configure this limit. After 6.0.x+ (next major release) there will be a dedicated configuration in limits.conf for this purpose. ... View more

As Our Wolverine intimates, there are tradeoffs between specificity and not. I tried to provide enough information to give mitigation information without giving steps to cause trouble. I'll relay this question along internally. It's possible I may be chided for saying more than is desirable. We'll see. ... View more

The vulnerability is a privilege escalation via a search mechanism. You can mitigate the problem by making it impossible for users to run searches on your forwarders, for example by not providing access to the management port, or by avoiding having accounts on those forwarders that anyone has the credentials for, or by not providing the search capability to any users who do exist on those systems, or a combination of any of the above. Of of course as jbacking points out, by running UFs where executing search is not permitted at all. ... View more

This code is not special to AIX at all, it will be pretty much identical for all UNIX types. For an input like [monitor:///inputs/*/1/2/36/file[0-9]{3}.log] We have to set up a watch on /inputs, because everything beyond this point is a pattern match. We do use pcre partial-match testing, so that if we reach a directory that PCRE can tell us will never match, no matter how much additional text is added, we can skip it. Thus, for example if we find a dir such as /inputs/q/2 we should be able to skip over this, because no matter how much additional text is added, the 2 will never match the 1 in the regex. However, I'm a little unclear about the case of /inputs/q/1/2/3 I think we try to force this to fail to match here by adding a slash after the directory name, but I'm not certain. We might descend into this directory. I would recommend testing locally in a simple setup. Yann's answer to specify only the exact dirs you want observed will certainly work. As for your attempted workaround, I think it's a little sketchy to ask Splunk to monitor /. However your regex which gets built out as ^/input[^/]*/[a-zA-Z0-9/]+file[0-9]+.log$ is very permissive. It allows any sequence of dir names that contain only ascii alphanumerics, followed a numbered filename. This regex should allow tailing to look at every single file in the hierarchy. ... View more

You'd sure think the code buttton in the editor here would work. Sad that it doesn't. ... View more

Snark alert! The + on the [class] is unnecessary. ... View more

The reason setting the executable setUID works (which I don't recommend!) is that Linux sanitizes the environment when jumping through a setuid gate, specifically dropping LD_LIBRARY_PATH and other linker controls, so that a user cannot execute arbitrary code as root trivially. Thus this indirectly requests the action the app should have taken in the first place, to strip the library path when running a system binary. ... View more

Is there a reason you wouldn't rather do unset LD_LIBRARY_PATH instead? ... View more

Right now, as an edge-case feature in the first place there's not a lot of pressure to rebuild a totally different replacement. If you think we're dumb or that reliable filtering of old data is important to you, then please do tell sales, support, etc. The normal channel is to file an enhancement request with Splunk Support. ... View more

That's the relatively easy-to-understand problem, and in some environments you can assure yourself that you're not ever going to be adding new filenames, or if you are that you don't care about missing the beginning of the first time the filename appears. However there are additional edge cases that are impossible to handle with the intersection of the concept of followTail and the overall design of Splunk Tailing. One of the two has to be changed pretty fundamentally to make it reliable. continued... ... View more