Hi. Recently I ran btool to see just what stanzas were being honored in various inputs.conf files. My command was: ./splunk cmd btool inputs list The response "looked" fine, but I later realized that config stanzas from certain inputs.conf files were omitted from the response. Ultimately, I determined that any config files in folders that I did not have permission to read, were ignored by btool. So, my questions... 1) Is this the way it is INTENDED to work? 2) If so, shouldn't btool at least send back a message notifying me that the results are incomplete??? This cost me a lot of time and head scratchin... thx, mfeeny1 ... View more

Hello, We are running 4.2.5-113966. Scanning splunkd.log, I see we are getting LOTS of errors (~2600 yesterday) of the form (always a pair of errors): 02-24-2012 08:01:54.076 -0600 ERROR StreamedSearch - preparing to stream search, failed to mkdir /usr2/splunkshare1/splunk-ui/var/run/splunk/dispatch/remote_<name of the other Splunk Search Head>-splnkcpo_subsearch_subsearch_scheduler_ nobody SplunkDeploymentMonitor_RE0gc291cmNldHlwZXMgdG9vIGxpdHRsZSBkYXRh_at_1330092000_8faa392bb73975e f_1330092112.2_1330092112.3<name of THIS Splunk Search Head>: File name too long 02-24-2012 08:01:54.077 -0600 ERROR SearchResults - Unable to open output file: path=/usr2/splunkshare1/splunk-ui/var/run/splunk/dispatch/remote_<name of the other Splunk Search Head>-splnkcpo_subsearch_subsearch_scheduler_ nobody SplunkDeploymentMonitor_RE0gc291cmNldHlwZXMgdG9vIGxpdHRsZSBkYXRh_at_1330092000_8faa392bb73975ef_1330092112 .2_1330092112.3<name of THIS Splunk Search Head>-splnkcpo/info.csv.tmp error=File name too long It would seem that there is some mechanism / algorithm that is generating these long file names -- longer than Linux can handle. Any ideas?? Thx! ... View more

Hi. We are seeing duplicate logfile entries in our Search results with certain logfiles. It is happening in a directory that is pointed to with a symbolic link (i.e., "/hosting/logs/myapp/" is a symbolic link for "/hosting/logs/app123/"). Our monitor stanza looks like this... [monitor:///hosting/logs] crcSalt = SOURCE blacklist = blah blah blah whitelist = blah blah blah recursive = true NOTE: In the actual inputs.conf, there are angle brackets surrounding SOURCE, but adding them in this text input box seems to invoke some markdown directive, so I removed them. For certain logfiles, we see the same logfile entry twice... One entry for /hosting/logs/app123/abc.log, and one for /hosting/logs/myapp/abc.log My questions are... Is the crcSalt statement forcing /hosting/logs/app123/abc.log to look like a different file than /hosting/logs/myapp/abc.log, causing it to be indexed twice? If so, what is/are suggested workaround(s)? And finally... it doesn't happen for ALL logfiles in /hosting/logs/app123/, only some of them. Any ideas why this is? Thx, mfeeny1 ... View more

I am somewhat new to tags as a "Knowledge Management" tool, and I am reviewing the tags configured on my SPLUNK search head, and totally confused by the following situation... From the GUI, I go to Manage -> Tags -> List by tag name. I see a particular tag (let's call it tag1) that is listed twice, with different field-value pairs. In one row, it includes two field-value pairs, and in the other it includes about 60 field-value pairs. In both definitions, the owner and the app are the same (admin, search). Is this viable? If so, if I were logged in as admin, and I used this tag in a search, which set of field-value pairs would it use? Thanx for any clarification... ... View more

Due to some inconsistencies in how some of our servers use Symbolic Links (symlinks), we need to understand how Splunk would handle the following situation... /usr2/wlp_logs is a symlink pointing to /hosting/logs. If we have the following 2 monitor stanzas in inputs.conf... [monitor:///usr2/wlp_logs/blah/myapp.log] [monitor:///hosting/logs/blah/myapp.log] ...would Splunk recognize that these are 2 references to the same file, and index it only once? Or, would myapp.log get indexed twice? Thx for any clarification / enlightenment. mfeeny1 ... View more

Hi... Pls forgive me, I have not used Regular Expressions in quite a while, and the following one really threw me. I'm looking for some clarification (enlightenment, maybe??? 🙂 ... I found the following in inputs.conf on a certain Universal Forwarder: whitelist = r.log$|t.log$|s_log$ I would have THOUGHT that only the following - very specific - log names would match: r.log, t.log, s_log But, I'm told that anything ENDING in these strings will match, e.g.: avatar.log, long-long-stringt.log, whatever-you-wants_log Is the second correct? If so, it confuses me that I wouldn't have to precede each regex with something like "dot star" to tell it that any number of characters can precede the "r". Any clarification will be greatly appreciated!! mfeeny1 ... View more

Hello. I am writing a script to install the Universal Forwarder on a set of Windows servers. The script will check for adequate disk space before proceeding with the install. What should I use as the minimum disk space required? Thx, mfeeny1 ... View more

Hi. I'm using a Deployment Server to push configurations to a Splunk Universal Forwarder that is running Windows. For the most part, it is working just fine, thanx. 😉 But... In my deploymentclient.conf, I have a line that says... phoneHomeIntervalInSecs = 120 And, when I issue the below command on the Deployment Client, it echoes that above setting exactly. splunk cmd btool deploymentclient list However, a packet capture shows that the Deployment Client is opening up a TCP connection with the Deployment Server, on the management port, every 60 seconds. Any ideas why??? Thx, Michael ... View more

Hi. I have a Deployment Server, and I am using it to configure a Deployment Client, which is a Windows Universal Forwarder. I want to use SSL for both Forwarder-Indexer communications, as well as Deployment Client - Deployment Server communications, using "our" Certs (not Splunk's). The initial state is that Forwarder-Indexer communication, as well as Deployment Client - Deployment Server communication, is working fine, but not using our Certs. My first step is to configure the Forwarder-Indexer communication to use our Certs. So, I modify outputs.conf in the appropriate deployment-apps directory on the Deployment Server, with the needed SSL parameters, and reload the app. It works like a charm - the UF is now sending data to the Indexers, using SSL, with our Certs. So now I want to make the equivalent change for the Deployment Client - Deployment Server communication. This time, I modify server.conf in the appropriate deployment-apps directory on the Deployment Server, with the needed SSL parameters, and reload the app. This does NOT work. splunkd.log, on both Deployment Client and Deployment Server, report errors (shown below), and the two endpoints no longer communicate, and a "./splunk list deploy-clients" command on the Deployment Server no longer displays the target machine as one of its Deployment Clients. As best as I can tell, I am using the same "forwarder.pem", "cacert.pem", and cert passwords in the two .conf files (outputs and server), but things just aren't working for the Deployment Client - Deployment Server communications. Any enlightenment, or at least tips on troubleshooting this, would be GREATLY appreciated! Here are the splunkd.log errors... On the Deployment Client 09-21-2011 20:38:02.971 -0700 ERROR SSLCommon - Can't read key file C:\Program Files\SplunkUniversalForwarder\etc\apps\\*app-name*\local\forwarder.pem errno=101077092 error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt. 09-21-2011 20:38:02.971 -0700 ERROR ServerConfig - Couldn't initialize SSL Context for HTTPClient in ServerConfig 09-21-2011 20:38:03.994 -0700 ERROR SSLCommon - Can't read key file C:\Program Files\SplunkUniversalForwarder\etc\apps\\*app-name*\local\forwarder.pem errno=101077092 error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt. 09-21-2011 20:38:03.994 -0700 ERROR HTTPServer - SSL context could not be created - error in cert or password is wrong 09-21-2011 20:38:03.994 -0700 ERROR HTTPServer - SSL will not be enabled On the Deployment Server 09-21-2011 20:36:13.548 -0700 ERROR TcpInputFd - SSL Error = error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http request 09-21-2011 20:36:13.548 -0700 ERROR TcpInputFd - ACCEPT_RESULT=-1 VERIFY_RESULT=0 09-21-2011 20:36:13.548 -0700 ERROR TcpInputFd - SSL Error for fd from HOST:*hostname*, IP:*ip-addr*, PORT:50252 And, here are the .conf files... outputs.conf [tcpout] defaultGroup = splunkssl-autolb-group disabled = false [tcpout-server://*hostname*:9998] sslCertPath=C:\Program Files\SplunkUniversalForwarder\etc\apps\\*app-name*\local\forwarder.pem sslPassword=*password* sslRootCAPath=C:\Program Files\SplunkUniversalForwarder\etc\apps\\*app-name*\local\cacert.pem sslVerifyServerCert=true sslCommonNameToCheck=*full_hostname* altCommonNameToCheck=*short_hostname* there is more, but not SSL-related server.conf [sslConfig] enableSplunkdSSL = true sslKeysfile = forwarder.pem sslKeysfilePassword = *password* caCertFile = cacert.pem caPath = C:\Program Files\SplunkUniversalForwarder\etc\apps\\*app-name*\local ... View more

Hi. I've been struggling with this for a more days than I'd care to admit. I'm HOPING someone can advise... (EnableBegging=True 🙂 GOAL: Install Universal Forwarder on Windows as a Deployment Client, via the CLI ("msiexec.exe /i ..."), such that deploymentclient.conf is NOT in "etc\system\local". WHY: With deploymentclient.conf in "etc\system\local", it is impossible for the Deployment Server to change or override that configuration, in the event that, at some later date, we want to change the Deployment Server. In that scenario, we would have to touch every Deployment Client manually. RESULTS FROM TESTING: If I include the DEPLOYMENT_SERVER flag in the CLI msiexec command, a deploymentclient.conf file is created, and placed in "etc\system\local". This actually "works", in that the DeploymentClient finds the Deployment Server, gets its config files, and begins eating log files and forwarding them to the Indexers - just like clockwork!. But, as explained above, it has the undesirable effect of placing deploymentclient.conf in a place where no DS-deployed apps can override it. Conversely, if I OMIT the DEPLOYMENT_SERVER flag from the CLI msiexec command, NO deploymentclient.conf file is created in "etc\system\local" -- or anywhere else, for that matter. So far, so good (I thought!). So, I next manually place deploymentclient.conf where I want it (in an "apps" directory), where it could be overridden at some later date. The problem is: I CAN'T GET THE DEPLOYMENT CLIENT TO FIND AND/OR USE deploymentclient.conf. I have done a "splunk restart", and I have Started the SplunkForwarder service (many, many times), but the Deployment Client never tries to contact the Deployment Server (packet captures confirm that it never sends a single packet to the DS). It seems that, once the Universal Forwarder is installed WITHOUT the DEPLOYMENT_SERVER flag, it will never learn how to contact the Deployment Server, and will never become a Deployment Client. As further indication of this last statement, I tried one other test. I installed the Universal Forwarder, WITHOUT the DEPLOYMENTSERVER flag, and then I placed deploymentclient.conf in "etc\system\local". STILL, after issuing splunkd restarts, and restarting the service, the Universal Forwarder does not try to contact the Deployment Server, even though deploymentclient.conf is in the very place where it lands when things "work" (when I do use the DEPLOYMENT_SERVER flag). And... One final, additional piece of evidence... When I install WITHOUT the DEPLOYMENTSERVER flag, splunkd.log contains the below message, that may be the key to explaining this behavior: WARN DeploymentClient - Property targetUri not found. DeploymentClient is disabled. So, it seems that, if no Deployment Server is defined at install time, then Deployment Client functionality is disabled. So... Either what I'm trying to do is not possible, OR... I need to learn how to "enable" DeploymentClient after the install. Thanx for listening - I know it took a while! I will entertain all suggestions, questions or explanations. mfeeny1 ... View more