Hi, I'm struggling with an issue involving my old nemesis, inputs.conf rules :-). In this case, we have a catch-all rule on our apache servers' inputs.conf at the bottom that looks like [monitor:///var/weblogs/] crcSalt = <SOURCE> whitelist = .../(access|error)\.log$ index = apache which works fine. There are multiple directories for different applications (Apache virtual hosts) under /var/weblogs like /var/weblogs/foo/access.log and so on. What I need to do is tell Splunk to make an exception for applications that start with "presell". As a normal regular expression this would look like "^/var/weblogs/presell.*/(access|error).log$". I believe it used to be that monitor: lines inputs.conf only understood their special wildcards and no regular expression stuff but now it seems like there are some rules that allow some RE's to be understood if it's mixed with wildcards in the same segment (weird!). In theory I would then expect [monitor:///var/weblogs/presell*/*(access|error).log] crcSalt = <SOURCE> index = presell to work. It does not. Events are still captured but go to the default index (i.e. "apache"). That extra '*' in front of the grouping expression is in accordance with the new rules that say that it would be recognized as an RE since there is a wildcard in the same segment. I would have also expected [monitor:///var/weblogs/presell*/access.log] crcSalt = <SOURCE> index = presell to work. It does not. Also thought [monitor:///var/weblogs/presell*/*.log] crcSalt = <SOURCE> index = presell would give me mostly what I want (I don't really want to capture anything that might crop up there with a .log suffix...). Still sends these events to the 'apache' index. I know I can't say [monitor:///var/weblogs/presell*/] crcSalt = <SOURCE> index = presell whitelist = .../(access|error)\.log$ because the implicit whitelist from the monitor: line conflicts with the explicit whitelist (and it doesn't work anyway). The only things I've found that work are to explicitly list the directories and/or files. That is, either [monitor:///var/weblogs/presellAppA/access.log] crcSalt = <SOURCE> index = presell [monitor:///var/weblogs/presellAppA/error.log] crcSalt = <SOURCE> index = presell [monitor:///var/weblogs/presellAppB/access.log] crcSalt = <SOURCE> index = presell [monitor:///var/weblogs/presellAppB/error.log] crcSalt = <SOURCE> index = presell OR [monitor://var/weblogs/presellAppA/] crcSalt = <SOURCE> index = presell whitelist = .../(access|error)\.log$ [monitor:///var/weblogs/presellAppB/] crcSalt = <SOURCE> index = presell whitelist = .../(access|error)\.log$ both of which are undesirable because it means I still have to enumerate all applications that start with "presell", meaning that if a new one cropped up tomorrow, it would not be handled the way I want. That is, if I could truly match on "presell*". Normally if I stare at stuff like this long enough I see something obvious that I'm doing wrong, but so far I've been unable to figure out why this isn't working the way I'd expect. I had been using a 6.4.3 universal forwarder here (sending to 6.5.0 forwarders and indexers) and then moved to 6.5.0 universal forwarders. As expected, the different versions work the same. Any idea what I'm doing wrong here? Thanks Mark ... View more

Hello! We've been using Splunk here for quite some time (back since the 4.x days) but only recently have we had a need to use the "sendemail" command. The server can send email just fine when scheduling an alert and can do it just fine from the Linux command line. However, when we try to use the sendemail command by copying examples in the doc, we always get command="sendemail", [('system library', 'fopen', 'No such file or directory'), ('BIO routines', 'BIO_new_file', 'no such file'), ('x509 certificate routines', 'X509_load_cert_crl_file', 'system lib')] while sending mail to: me@mycompany.com when I look in python.log I see pretty much the same non-informative information: 2016-10-19 16:33:35,866 -0400 INFO sendemail:1067 - sendemail pdfgen_available = 1 2016-10-19 16:33:35,892 -0400 INFO sendemail:1195 - sendemail:mail effectiveTime=None 2016-10-19 16:33:40,594 -0400 INFO sendemail:1215 - Generated PDF for email 2016-10-19 16:33:40,658 -0400 ERROR sendemail:131 - Sending email. subject="Here is an email notification 2", results_link="None", recipients="[u'me@mycompany.com']", server="internalmailhub.com" 2016-10-19 16:33:40,659 -0400 ERROR sendemail:417 - [('system library', 'fopen', 'No such file or directory'), ('BIO routines', 'BIO_new_file', 'no such file'), ('x509 certificate routines', 'X509_load_cert_crl_file', 'system lib')] while sending mail to: me@mycompany.com So this seems like some kind of a python error with sendemail.py but it's not informative enough to clue me in to what might be wrong. Note that I've tried both using the "server" argument to sendemail and without it as both email types would work from this server (i.e. sending directly to a mail hub or letting it trickle through the internal mail system which would also punt the message to our internal mail hub). Other useful information - all servers are SLES 11.x - default Linux mail configuration on our servers is for postfix. I believe it includes its own flavor of the "sendmail" command that is a work-alike to commands that want to call sendmail (i.e. "sendmail -t ..."). Don't know if that's a factor here - servers we've tested this on are all search heads and we get consistent responses on each - search heads are running 6.5.0 It probably goes without saying that if I try to just run the sendemail.py python script just to see if there's something obvious it can't find, it fails $ python sendemail.py Traceback (most recent call last): File "sendemail.py", line 1, in <module> import re, time, splunk.Intersplunk, splunk.mining.dcutils as dcu ImportError: No module named splunk.Intersplunk but I assume that's expected as it's not expecting to be run from an external python and rather from inside Splunk. Thanks for your help! -Mark ... View more

Hi. I'm wondering if I'm missing something here, but it seems like I can't manage a server that has more than one Splunk instance on it using the Distributed Management Console (Splunk 6.4+). I noticed that I needed to have a different serverName set via server.conf in order to be able to add it as a search peer on my DMC. So same hostname, but different instance name and, of course, different management port. When I go to the DMC Setup page to turn on monitoring on my second instance and then hit Apply, I get You have some unresolved errors that need to be fixed before you can proceed. Check the problems column and expand for more detail. and then under the specific hosts (both entries with the same hostname but different instance names show this): Duplicate instance name. Ensure each instance has a unique instance (host) name. I don't see a way around this and it's kind of a bummer to not be able to see some of my deployment servers' status in the DMC. It's frustrating that I've done what was needed to make Splunk accept the second instance as a search peer, but the DMC wants more than that. Thanks ... View more

Maybe I'm mistaken, but it seems as if there's some issue with the latest v2.2.9 Search Activity app , specifically with the tar file. I noticed that when I tried to update it via the link in the Splunk UI, it spins for a while then gives me a 500 error . When I downloaded the tar file itself and try to extract it as an ordinary user on a 64-bit Linux server, it gives me a permission denied on every file (and yes, I'm doing this in a directory where I have permissions and on a filesystem that is not full). When I attempt to look through what was extracted, I see a few files and directories, but then some look like > ./search_activity/static: ls: cannot > access > ./search_activity/static/appIconAlt.png: > Permission denied ls: cannot access > ./search_activity/static/appLogo.png: > Permission denied ls: cannot access > ./search_activity/static/appLogo_2x.png: > Permission denied ls: cannot access > ./search_activity/static/appIconAlt_2x.png: > Permission denied ls: cannot access > ./search_activity/static/appIcon.png: > Permission denied ls: cannot access > ./search_activity/static/appIcon_2x.png: > Permission denied total 0 > -????????? ? ? ? ? ? appIcon_2x.png > -????????? ? ? ? ? ? appIconAlt_2x.png > -????????? ? ? ? ? ? appIconAlt.png > -????????? ? ? ? ? ? appIcon.png > -????????? ? ? ? ? ? appLogo_2x.png > -????????? ? ? ? ? ? appLogo.png I'm guessing that there's an issue with the way this tar file was created as I've never seen an issue with any tar file like this one before. thanks ... View more

Hi. We're in the process of redoing a lot of our existing Splunk infrastructure. Currently we have a license master that co-resides in the same instance as an indexer. We also have a deployment server that runs as a separate instance on a separate indexer server. (We learned early on that deployment servers don't play nice with streams of events coming in on the same port. We now have a few hundred deployment clients.) In the new infrastructure, we want to move those 2 functions off to dedicated servers. Or at least semi-dedicated. Or at least not search heads or indexers. However, since we're also going to convert to search head clustering we'll need to run a "deploy server". We're also going to use this opportunity to setup the distributed management console. This means we have 4 functions Deployment server Deployer for clustered search heads License Master Distributed management console and we have 2 (new) servers to do it on -- 2 servers that exist for no purpose other than to provide these 4 functions. My problem is, I'm not exactly sure how best to combine them. As I'm pretty sure that the deployment server won't work well with any of the other 3 functions, it seems reasonable to make a single instance with just that. It would be kind of nice to avoid having to have multiple instances on a single server as that's a minor pain to manage (multiple $SPLUNK_HOME's, etc) and I thought it might be possible to run the deployment server on one host and then all of the remaining 3 (deploy server, license master and DMC) in a single instance on the other host. I don't know if that's reasonable though. Absolute worst case, I can run 2 instances on each of the 2 servers and provide one instance per function, but I'd rather not if I don't have to. Does anyone have any experience and/or opinions about how best to combine these functions across 2 hosts with or without multiple instances on them? Thanks! ... View more

I'm trying to build some new search heads and I'm looking at how much disk space I should really need for indexes ($SPLUNK_DB) on a search head, particularly if I'm configuring that search head to forward any events its told to index to the indexers as is the recommended best practice. I see that there is some space used on my current indexers and that a certain minimum amount is probably necessary. Could I potentially scale back indexes that exist on the search heads for one reason or another (sos, os) so that they're very small since any events that are supposed to go there will be forwarded to indexers anyway where they'll be more space allocated? I was thinking about 10GB of local space for indexes on the search heads, but if I don't even have to use that much, I'd rather not, especially when I can use it for dispatch. Thanks ... View more

Hello. Our site does not currently use or have a particular need for indexer clustering. We are, however, about to shut down the datacenter that holds our indexers and move that functionality to a different datacenter a couple thousand miles away. The indexers themselves are not moving. We have a fairly steady stream of events flowing into those indexers currently. We are faced with the problem of how to get existing indexed events to the new datacenter without dropping events coming from forwarders. We've got 100's of GB of data in the indexers now so a directly copy/rsync would take a very long time and we can't shut the indexers down while that happens as events from forwarders would be dropped. I realize we can copy the cold buckets directly, but even that will take a long time to do directly. I was wondering if using indexer clustering (multisite) might be a way to get Splunk to do the replication for me. If I could setup a cluster that replicated all the data in the datacenter that's going away to the new datacenter, wait until all the data is replicated, then remove the old datacenter's indexers from the cluster and shut them down, is that a viable strategy? Thanks ... View more