An app of ours spits such a huge volume of data when our Devs increase its debug level to Trace that it essentially results on the Splunk Heavy forwarders sitting on those servers to stall and becoming non operational (we implemented a script to auto restart them when we don't have indexed data on those servers for more than 10m) As far as I know it's only possible to send events to nullQueuing, ie, after properties like line breaking (which is costly but vital for us for those logs) have already been computed. Any way we can setup props / transforms / anything to first completely ignore the TRACE lines and only then applying the props to the logs? ... View more

Why is this monitor whitelist not working ? [monitor:///opt/logs/] whitelist = (connectors/connectors\-\d\-boot|app1/app1\-\d\-boot|app2/app2(-stdout|-boot))\.log$ for these log files, connectors/connectors-1-boot.log app1/app1-1-boot.log app2/app2-boot.log app2/app2-stdout.log whereas a wilcarded input is? [monitor:///opt/logs/.../*-boot.log] [monitor:///opt/logs/app2/app2-stdout.log] If I split the problematic one into several ones it will pickup the corresponding files, eg [monitor:///opt/logs/app2/] whitelist = app2(-stdout|-boot)\.log$ Disclaimer, is it clashing with this other perfectly working monitor of mine ? If it is, and the wildcard in the file input gets translated into an implicit whitelist anyway, why the difference ? [monitor:///opt/logs/] whitelist = (connectors/connectors\-.*|app2/app2(-server|-service))\.log$ blacklist = (boot|\.zip) ... View more

I'm having trouble recognizing the timestamp for a logs with this structure, (field timestamp appears = none in Splunk fields column) DEBUG [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-3] 09 Dec 2014 22:00:03,760 (?:?) - Performing a clean update ... DEBUG [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-3] 09 Dec 2014 22:00:03,761 (?:?) - Delete Query: delete bla_bla INFO [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-3] 09 Dec 2014 22:00:03,797 (?:?) - Records processed - total: 9, inserted: 9, skipped: 0 DEBUG [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-3] 09 Dec 2014 22:00:03,808 (?:?) - My props.conf [spring_output] NO_BINARY_CHECK=1 SHOULD_LINEMERGE = true TIME_FORMAT = %d %b %Y %H:%M:%S,%3N # Breaking events on lines like: "DEBUG [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker:3] 09 Dec 2014 22:00:03,949" TIME_PREFIX = ^\s*([A-Z])\w+ \[.*]\s+ MAX_TIMESTAMP_LOOKAHEAD=1 TZ=EST BREAK_ONLY_BEFORE=^FATAL|^DEBUG|^ERROR|^INFO|^TRACE|^WARN|^ WARN|^ INFO|^CRIT|^ CRIT MAX_EVENTS = 99999 TRUNCATE = 0 ... View more

Hi, what would be the SPL to return a table ordered by most common oid, server ? 2014-02-11 17:09:23,855 ERROR SNMP get returned empty value: SWITCH-UAT-07 1.3.6.1.4.1.9.9.117.1.1.2.1.3.104000023 2014-02-11 16:06:01,655 ERROR SNMP get returned empty value: server-prd-01 1.3.6.1.4.1.232.6.2.6.8.1.6.1.9 2014-02-11 14:48:26,846 ERROR SNMP get returned empty value: SWITCH-US1 1.3.6.1.2.1.31.1.1.1.11.5 ... View more

Congrats on this great major release ! Are there any plans to introduce horizon charts(similar to sparklines but not quite the same) (also here) visualization? There are some cases where the number of series is just to big to fit in a multiline chart without turning it into a train wreck (and splitting them into several charts forces the Dashboard to become bigger and bigger). It would be quite a welcome addition. ... View more

How is it possible that an eval expression which its components total weight is 100, breach that expected value? For a given period, I'm getting a result of 101, while expression components weights are (it totals to 100) 100*(0.08 + 0.03 + 0.12 + 0.16 + 0.06 + 0.18 + 0.06 + 0.06 + 0.25) and there is not even one component value which exceeds 1. | eval TimeoutScore=100*( 0.08*(bw1sdpSdfPccCsApdex) + 0.03*bw1sdpSdfPccMnCsApdex + 0.12*bw1sdpSdfPccPsApdex + 0.16*(0.5*bw1sdpSdfPpsCsApdex +0.5*bw2sdpSdfPpsCsApdex) + 0.06*(0.5*bw1sdpSdfPpsMnCsApdex +0.5*bw2sdpSdfPpsMnCsApdex) + 0.18*(0.5*bw1sdpSdfPpsPsApdex + 0.5*bw2sdpSdfPpsPsApdex) + 0.06*(0.5*bw1sdpSdfPpsVisaCsApdex + 0.5*bw2sdpSdfPpsVisaCsApdex) + 0.06*bw1careSdfMnApdex + 0.25*(0.5*bw1sdpSdfSlrApdex + 0.5*bw2sdpSdfSlrApdex)) | stats sum(TimeoutScore) as TimeoutScore sum(bw1sdpSdfPccCsApdex) sum(bw1sdpSdfPccMnCsApdex) sum(bw1sdpSdfPccPsApdex) sum(bw1sdpSdfPpsCsApdex) sum(bw2sdpSdfPpsCsApdex) sum(bw1sdpSdfPpsMnCsApdex) sum(bw2sdpSdfPpsMnCsApdex) sum(bw1sdpSdfPpsPsApdex) sum(bw2sdpSdfPpsPsApdex) sum(bw1sdpSdfPpsVisaCsApdex) sum(bw2sdpSdfPpsVisaCsApdex) sum(bw1careSdfMnApdex) sum(bw1sdpSdfSlrApdex) sum(bw2sdpSdfSlrApdex) TimeoutScore sum(bw1sdpSdfPccCsApdex) sum(bw1sdpSdfPccMnCsApdex) sum(bw1sdpSdfPccPsApdex) sum(bw1sdpSdfPpsCsApdex) sum(bw2sdpSdfPpsCsApdex) sum(bw1sdpSdfPpsMnCsApdex) sum(bw2sdpSdfPpsMnCsApdex) sum(bw1sdpSdfPpsPsApdex) sum(bw2sdpSdfPpsPsApdex) sum(bw1sdpSdfPpsVisaCsApdex) sum(bw2sdpSdfPpsVisaCsApdex) sum(bw1careSdfMnApdex) sum(bw1sdpSdfSlrApdex) sum(bw2sdpSdfSlrApdex) 101 1 0.854839 0.972727 1 1 1 1 0.975410 1 1 1 0.983871 1 1 EDIT: Thanks to kristian.kolb suggestion, this issue was solved replacing | eval TimeoutScore=... by | eval TimeoutScore=exact(...) ... View more

I've got a custom source A and B, which I need to compute a weighted average over, each source has only 2 collums: datetime and its own score (one per day, both sources times coincide). EDIT: EtuServiceApdex.csv headers: _time,EtuServiceApdex EtuOpApdex.csv headers: _time,EtuOpApdex index=resources_reporting (source="*/output/ora_queries/client07/EtuOpApdex.csv" OR source="*/output/ora_queries/client07/EtuServiceApdex.csv") returns something like, 28/02/2013 00:00:00.000 2013-02-28T00:00:00.000+00:00,64.583333 host=vsplunk sourcetype=csv-47 source=/opt/splunk/etc/apps/remote_scripts/output/ora_queries/client07/EtuServiceApdex.csv 28/02/2013 00:00:00.000 2013-02-28T00:00:00.000+00:00,50 host=vsplunk sourcetype=csv-49 source=/opt/splunk/etc/apps/remote_scripts/output/ora_queries/client07/EtuOpApdex.csv 27/02/2013 00:00:00.000 2013-02-27T00:00:00.000+00:00,77.083333 host=vsplunk sourcetype=csv-47 source=/opt/splunk/etc/apps/remote_scripts/output/ora_queries/client07/EtuServiceApdex.csv 27/02/2013 00:00:00.000 2013-02-27T00:00:00.000+00:00,54.166667 host=vsplunk sourcetype=csv-49 source=/opt/splunk/etc/apps/remote_scripts/output/ora_queries/client07/EtuOpApdex.csv and both index=resources_reporting (source="*/output/ora_queries/mt/MT_EtuOpApdex.csv" OR source="*/output/ora_queries/client07/EtuServiceApdex.csv") | timechart avg(EtuOpApdex) or | timechart avg(EtuServiceApdex) return the expected graph over time, however, If I add the following I lose the _time properties | eval TotalApdexScore=(0.66*EtuServiceApdex + 0.33*EtuOpApdex) and | timechart avg(TotalApdexScore) returns a _time series with null values over TotalApdexScore. Eg: _time avg(TotalApdexScore) 1 01/01/2013 00:00:00.000 2 02/01/2013 00:00:00.000 3 03/01/2013 00:00:00.000 How should I change the expression to get the intended weighted average over time ? ... View more