I was looking for a way to capture hosting events (maintenance, planned downtimes, unplanned downtimes, etc) and log them in splunk. I could just index a file on a server and append well formatted text to and have splunk ingest it.... Or, I could have a simple web view within splunk, fill in the data, submit it and have splunk index it. The latter seems a bit nicer. ... View more

When accessing Splunk DB Connect, the start page for the app just keeps refreshing continuously. There are db connections configured and they do work, the bridge does appear to be running, I can run queries. If I check the bridge status from the menu in run query, it does a similar behavior, just says loading and keeps refreshing. It's not possible to edit database inputs as a result of this, portions of those pages keep refreshing. Any help much appreciated. Here are the particulars of the environment. Thanks -Doug dbx 1.2.2 splunk enterprise 6.3.0 /usr/lib/jvm/jre-1.7.0-openjdk.x86_64 mysql-connector-java-5.1.18-bin.jar ps output for bridge: root 27507 27005 1 18:58 ? 00:00:02 /usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java -cp /opt/splunk/etc/apps/dbx/bin/lib/hsqldb.jar:/opt/splunk/etc/apps/dbx/bin/lib/jtds-1.2.6.jar:/opt/splunk/etc/apps/dbx/bin/lib/commons-pool-1.5.6.jar:/opt/splunk/etc/apps/dbx/bin/lib/log4j-1.2.15.jar:/opt/splunk/etc/apps/dbx/bin/lib/stringtemplate-3.2.1.jar:/opt/splunk/etc/apps/dbx/bin/lib/dbx.jar:/opt/splunk/etc/apps/dbx/bin/lib/antlr-2.7.7.jar:/opt/splunk/etc/apps/dbx/bin/lib/xstream-1.4.1.jar:/opt/splunk/etc/apps/dbx/bin/lib/h2-1.3.162.jar:/opt/splunk/etc/apps/dbx/bin/lib/jdbm-2.2.jar:/opt/splunk/etc/apps/dbx/bin/lib/commons-logging-1.0.4.jar:/opt/splunk/etc/apps/dbx/bin/lib/mysql-connector-java-5.1.18-bin.jar:/opt/splunk/etc/apps/dbx/bin/lib/postgresql-9.0-801.jdbc3.jar:/opt/splunk/etc/apps/dbx/bin/lib/sqlite-jdbc-3.7.2.jar -Xmx512m -Dfile.encoding=UTF-8 -server -Duser.language=en -Duser.region= -Dsplunk.app.ctx=dbx com.splunk.bridge.JavaBridgeServer 27005 ... View more

I'm trying to detect when a server goes from an error state to operational on our load balancers for an email alert. The first part of the query looks for the last 'operational' message and the second (joined) part of the search looks for a non-operational message. The problem I'm having is t2 (time from second query) doesn't seem to get evaled. I don't see it in the results and can't do the final search on it. earliest=-30m@m | eval r=_raw|eval t1=_time | stats first(r) as fr1 by lb_server |search fr1="*operational*"| search fr1 | sort lb_server | join lb_server [search host="12.130.11.2" latest=-60m@m | eval t2=_time | eval r=_raw | stats first(r) as fr2 by lb_server| search fr2!="*operational*"] | search t1 > t2 Hopefully I'm doing something wrong that's easy to fix, or there is another better, stronger, faster way do what I'm after. Thanks -Doug Here's what the data looks like: May 22 08:29:22 1.2.3.2 May 22 8:29:19 Primary NOTICE AlteonOS <slb>: Services are available for IP4 Virtual Server 3:1.2.4.142 May 22 08:29:22 1.2.3.2 May 22 8:29:19 Primary NOTICE AlteonOS <slb>: real server 1.2.3.41 operational May 22 08:29:22 1.2.3.2 May 22 8:29:19 Primary NOTICE AlteonOS <slb>: real service 1.2.3.41:80 operational May 22 08:19:52 1.2.3.2 May 22 8:19:49 Primary NOTICE AlteonOS <slb>: No services are available for IP4 Virtual Server 3:1.2.4.142 May 22 08:19:52 1.2.3.2 May 22 8:19:49 Primary ALERT AlteonOS <slb>: cannot contact real server 1.2.3.41 May 22 08:19:52 1.2.3.2 May 22 8:19:49 Primary ALERT AlteonOS <slb>: script 1 healthcheck failed on real server 1.2.3.41 May 22 07:11:50 1.2.3.2 May 22 7:11:47 Primary NOTICE AlteonOS <slb>: Services are available for IP4 Virtual Server 3:1.2.4.142 May 22 07:11:50 1.2.3.2 May 22 7:11:47 Primary NOTICE AlteonOS <slb>: real server 1.2.3.41 operational May 22 07:11:50 1.2.3.2 May 22 7:11:47 Primary NOTICE AlteonOS <slb>: real service 1.2.3.41:80 operational May 22 06:23:49 1.2.3.2 May 22 6:23:47 Primary NOTICE AlteonOS <slb>: No services are available for IP4 Virtual Server 3:1.2.4.142 May 22 06:23:49 1.2.3.2 May 22 6:23:47 Primary ALERT AlteonOS <slb>: cannot contact real server 1.2.3.42 May 22 06:23:49 1.2.3.2 May 22 6:23:47 Primary ALERT AlteonOS <slb>: script 1 healthcheck failed on real server 1.2.3.42 May 22 06:23:33 1.2.3.2 May 22 6:23:31 Primary ALERT AlteonOS <slb>: cannot contact real server 1.2.3.41 May 22 06:23:33 1.2.3.2 May 22 6:23:31 Primary ALERT AlteonOS <slb>: script 1 healthcheck failed on real server 1.2.3.41 May 21 17:21:39 1.2.3.2 May 21 17:21:33 Primary NOTICE AlteonOS <slb>: Services are available for IP4 Virtual Server 5:1.2.4.139 May 21 17:21:39 1.2.3.2 May 21 17:21:33 Primary NOTICE AlteonOS <slb>: real server 1.2.3.40 operational May 21 17:21:39 1.2.3.2 May 21 17:21:33 Primary NOTICE AlteonOS <slb>: real service 1.2.3.40:80 operational May 21 17:20:39 1.2.3.2 May 21 17:20:33 Primary NOTICE AlteonOS <slb>: No services are available for IP4 Virtual Server 5:1.2.4.139 May 21 17:20:39 1.2.3.2 May 21 17:20:33 Primary ALERT AlteonOS <slb>: script 1 healthcheck failed on real server 1.2.3.40 May 21 17:20:24 1.2.3.2 May 21 17:20:18 Primary ALERT AlteonOS <slb>: Script 1 failed on real(9): expect OK, received HTTP/1.1 302 Found May 21 17:15:59 1.2.3.2 May 21 17:15:54 Primary NOTICE AlteonOS <slb>: real server 1.2.3.41 operational May 21 17:15:59 1.2.3.2 May 21 17:15:54 Primary NOTICE AlteonOS <slb>: real service 1.2.3.41:80 operational May 21 17:15:59 1.2.3.2 May 21 17:15:54 Primary NOTICE AlteonOS <slb>: Services are available for IP4 Virtual Server 3:1.2.4.142 May 21 17:15:59 1.2.3.2 May 21 17:15:54 Primary NOTICE AlteonOS <slb>: real server 1.2.3.42 operational May 21 17:15:59 1.2.3.2 May 21 17:15:54 Primary NOTICE AlteonOS <slb>: real service 1.2.3.42:80 operational May 21 17:15:26 1.2.3.2 May 21 17:15:20 Primary NOTICE AlteonOS <slb>: No services are available for IP4 Virtual Server 3:1.2.4.142 May 21 17:15:26 1.2.3.2 May 21 17:15:20 Primary ALERT AlteonOS <slb>: cannot contact real server 1.2.3.42 May 21 17:15:26 1.2.3.2 May 21 17:15:20 Primary ALERT AlteonOS <slb>: script 1 healthcheck failed on real server 1.2.3.42 May 21 17:15:25 1.2.3.2 May 21 17:15:19 Primary ALERT AlteonOS <slb>: cannot contact real server 1.2.3.41 May 21 17:15:25 1.2.3.2 May 21 17:15:19 Primary ALERT AlteonOS <slb>: script 1 healthcheck failed on real server 1.2.3.41 May 21 17:14:12 1.2.3.2 May 21 17:14:07 Primary NOTICE AlteonOS <slb>: real server 1.2.3.42 operational May 21 17:14:12 1.2.3.2 May 21 17:14:07 Primary NOTICE AlteonOS <slb>: real service 1.2.3.42:80 operational May 21 17:14:12 1.2.3.2 May 21 17:14:07 Primary NOTICE AlteonOS <slb>: Services are available for IP4 Virtual Server 3:1.2.4.142 May 21 17:14:12 1.2.3.2 May 21 17:14:07 Primary NOTICE AlteonOS <slb>: real server 1.2.3.41 operational May 21 17:14:12 1.2.3.2 May 21 17:14:07 Primary NOTICE AlteonOS <slb>: real service 1.2.3.41:80 operational May 21 17:13:41 1.2.3.2 May 21 17:13:35 Primary NOTICE AlteonOS <slb>: No services are available for IP4 Virtual Server 3:1.2.4.142 May 21 17:13:41 1.2.3.2 May 21 17:13:35 Primary ALERT AlteonOS <slb>: cannot contact real server 1.2.3.42 May 21 17:13:41 1.2.3.2 May 21 17:13:35 Primary ALERT AlteonOS <slb>: script 1 healthcheck failed on real server 1.2.3.42 May 21 17:13:40 1.2.3.2 May 21 17:13:34 Primary ALERT AlteonOS <slb>: cannot contact real server 1.2.3.41 May 21 17:13:40 1.2.3.2 May 21 17:13:34 Primary ALERT AlteonOS <slb>: script 1 healthcheck failed on real server 1.2.3.41 May 21 16:01:22 1.2.3.2 May 21 16:01:16 Primary NOTICE AlteonOS <slb>: Services are available for IP4 Virtual Server 5:1.2.4.139 May 21 16:01:22 1.2.3.2 May 21 16:01:16 Primary NOTICE AlteonOS <slb>: real server 1.2.3.40 operational May 21 16:01:22 1.2.3.2 May 21 16:01:16 Primary NOTICE AlteonOS <slb>: real service 1.2.3.40:80 operational May 21 15:21:35 1.2.3.2 May 21 15:21:30 Primary NOTICE AlteonOS <slb>: No services are available for IP4 Virtual Server 5:1.2.4.139 May 21 15:21:35 1.2.3.2 May 21 15:21:30 Primary ALERT AlteonOS <slb>: cannot contact real server 1.2.3.40 May 21 15:21:35 1.2.3.2 May 21 15:21:30 Primary ALERT AlteonOS <slb>: script 1 healthcheck failed on real server 1.2.3.40 May 21 15:21:20 1.2.3.2 May 21 15:21:15 Primary ALERT AlteonOS <slb>: Script 1 failed on real(9): expect OK, received HTTP/1.1 302 Found ... View more

I have a series of servers that run apache that serve up the same url via post 99% of the time and in high volume. Indexing them individually would each up way too much of the indexing volume so currently they're excluded. Using awk I can process the file at log rotation time and produce aggregates like this: 28/Sep/2011:11:40 count=20393 avgsize=32535 avgtime=150 maxtime=710 For a five minute interval per server. I'd like the information more real time then waiting until the end of the week. Is there any way to do this completely within spunk (without indexing every access log entry)? Is there another way that I can cron something to run periodically to a log file that then spunk eats? Thanks -Doug ... View more

I have a lwf sending apache logs (/var/log/httpd/access.log) to an indexer and they're being sourcetyped as 'unknown'. On the forwarder in props.conf I have: [source::/var/log/httpd/access.log(.\d+)?] sourcetype = apache_access Assuming this is the correct place, what am I doing wrong. I think I might be able to also do a transform but thought I'd stick with this for now. Thanks ... View more