Hi, I must confess I'm still not understanding how wildcards work in inputs.conf. I've got a clustered application, with five instances on one server. The instances are named live-1,live-2,live-3,staging-1,staging-2. They're all located in /opt/INSTANCE_NAME I'm trying to monitor all the live instances [monitor:///opt/foo/live-*/logs/] index = foo_live sourcetype = log4j crcSalt = <SOURCE> blacklist= (\.(gz|bz2|z|zip)$) The problem with this stanza is that everything under /opt/foo will be listed by 'splunk list monitor'. More than 16000 files... Including everything in /opt/foo/staging-[12] and /opt/foo/whatever. I don't understand how that's possible, since none of those paths include the "live-" part, but anyway... Problem number 2 is that nothing will actually be forwarded to the indexer by this. The documentation seems pretty straigh-forward on this, so I really don't understand why it isn't working. If I list every directory as individual stanzas, the forwarding will work as expected. However, I would really need a generic solution to match all future environments as well. (Several applications, several instances.) ... View more

Hi, Is there any way of creating indexes on several indexers centrally? For a fairly small indexer-farm, it isn't much hassle to do it manually, but for several servers, it might be easier and more resistant to human error to do it centrally. ... View more

Hi, I'm having a weird problem with recognizing timestamps. The actual timestamp looks like this: [2012-04-11 11:24:11+03:00] However, since the event itself contains the compilation time of the kernel (uname -a), Splunk will identify the date based on that, but will pick up the time from the timestamp. So the time of day changes, but the date is constantly 2012-01-20. I've tried these settings in props.conf: [my_sourcetype] TIME_PREFIX = ^ MAX_TIMESTAMP_LOOKAHEAD = 28 or [my_sourcetype] TIME_PREFIX = \[ MAX_TIMESTAMP_LOOKAHEAD = 26 But the result is still the same. ... View more

Hi, is there a way of ignoring the time zone in the searches? Currently, Splunk will reinterpret the difference in time zones. Because of daylight savings time, Splunk will offset the results by one hour from the actual timestamp. So, I would like to know what actually happened at say 11:23 and not 11:23 minus an hour. ... View more

Hi, We're about to move our Splunk infrastructure from an environment on 4.1 to 4.3. I'm wondering if I can just copy the directories in etc/users to the new search head, in order to have retain all scheduled searches and alerts. Indeed, I probably need to copy stuff under etc/apps as well. Any pointers are appreciated. ... View more

Hi, I'm trying to set authenticate the forwarders using SSL certificates. If using certificates signed by a single root CA, this causes no problems. However, when using an intermediate CA, I can't seem to get it working no matter what. I'm using Splunk 4.2.3. I'm basing my efforts on this: http://www.splunk.com/wiki/Community:Splunk2Splunk_SSL_3rdPartyCA Q1) Which certificates need to be stored on the forwarder and indexer? I'm guessing both. Q2) Where should these certificates be located? Does order matter? Currently, I'm putting them both in cacert.pem. I've tried both putting the root CA first and the intermediate CA first. If I put the root CA first, the indexer will complain that the CA is untrusted. If I put the intermediate CA first, the indexer will report "routines:SSL3_GET_CLIENT_CERTIFICATE:no certificate returned" Q3) On both the forwarder and the indexer, I've done this: "cat cert.pem key.pem cacert.pem >server.pem". (Based on the document linked to above.) Is this correct? forwarder's system/local/outputs.conf [tcpout] defaultGroup = default-autolb-group disabled = false [tcpout-server://indexer1.company.com:9996] sslCertPath = /opt/splunkforwarder/etc/auth/server.pem sslPassword = wouldntyouliketoknow sslRootCAPath = /opt/splunkforwarder/etc/auth/cacert.pem [tcpout:default-autolb-group] autoLB = true disabled = false server = indexer1.company.com:9996,indexer2.company.com:9996 [tcpout-server://indexer2.company.com:9996] sslCertPath = /opt/splunkforwarder/etc/auth/server.pem sslPassword = wouldntyouliketoknow sslRootCAPath = /opt/splunkforwarder/etc/auth/cacert.pem indexer's system/local/inputs.conf: [default] host = indexer1.company.com [splunktcp-ssl:9996] [SSL] password = betyoudliketoknow requireClientCert = true rootCA = /opt/splunk/etc/auth/cacert.pem serverCert = /opt/splunk/etc/auth/server.pem Any tips would be greatly appreciated. ... View more

Hi, I'm just setting up a deployment server and created a simple app to test it. The app was installed fine on my universal forwarder. Undeploying the app also seemed to work fine. Basically, I deleted the directory from etc/deployment-apps and did a 'splunk reload deploy-server' on the deployment server. The forwarder's log shows it indeed picked up the instruction and the directory is removed from the forwarder. However, the forwarder keeps sending the information to the indexer and 'splunk list monitor' still shows the monitored file. The forwarder's log also contains tens of lines like this: 10-11-2011 14:41:54.735 +0300 ERROR IniFile - Cannot open ini file for parsing: No such file or directory 10-11-2011 14:41:54.735 +0300 ERROR ConfObjectManagerDB - Cannot initialize: /opt/splunkforwarder/etc/apps/Bar/metadata/default.meta 10-11-2011 14:41:54.735 +0300 ERROR IniFile - Cannot open ini file for parsing: No such file or directory 10-11-2011 14:41:54.735 +0300 ERROR ConfObjectManagerDB - Cannot initialize: /opt/splunkforwarder/etc/apps/Bar/metadata/local.meta Is this normal behaviour? ... View more

Hi, I'm trying to extract the name of the tomcat instance based on the path of the source. I've been successful by specifying the sourcetype in props.conf: [app_foo.log] EXTRACT-tomcat_instance = /opt/tomcat/(?<tomcat_instance>[^/]+)/logs/.* in source The above works, but I would like to match all the logs in tomcat directories, since there are several sourcetypes and I'd rather not repeat the same regex several times. So I tried the following: [source::/opt/tomcat/[^/]+/logs/.*] EXTRACT-tomcat_test = /opt/tomcat/(?<tomcat_instance>[^/]+)/logs/.* in source However, this does not seem to work. I've tested the regex with the commands rex and regex and it works there. Any pointers would be appreciated. ... View more