So everything is working as it should be; scheduled searches are all run by the dedicated search head and other search heads don't run any. If you login to search head that IS running scheduler you should also see the correct scheduled time for your searches. I think this is feature not a bug in your config and everything seems to be the way you wanted. After all those normal search heads are not going to run any of scheduled searches so in that sense NONE is the correct answer for them. ... View more

How does your data look? If you already have "du" -field in it I don't see any need for "multikv" -commands in your search. Also in your original example you were counting number of events by opp(?) but then you have changed it to be hourly average of du -field. Was this done on purpose? Does your search/charting work for today and week-ago if you run them separately? ... View more

Do you mean something like ... | stats sum(duration) as browsing_time by user, date_year, date_month, date_mday ... View more

Curl should take care of encoding for you. There is an option "--data-urlencode" that should do the trick. You don't need to do anything, except escape it for your shell so it doesn't get altered before curl gets it 🙂 If you think you have problem with your shell messing with the data, you can try putting it into file and passing to curl with @filename option for --data-urlencode (see curl man-page). Also this this tutorial can be useful. If tempted to encode your string manually (not recommended), here is the spec what to do. ... View more

Just guessing but could it be those "\n"s ? You have put sed to line-by-line -mode, so "$" is now end-of-line and I doubt if need that extra newline in your sed command. SEDCMD-httpheader = s/(?mg)^POST.*$|^User-Agent.*$//g ... View more

Here is a short presentation "Real Time in Splunk 4.1" explaining how it works. I would say real-time will do the job with a bit less total load on your system as extra disk read is avoided. But big difference is real-time load is spread evenly, events are processed as they come in, while scheduled search does all 5min at one go (= can have some "spikes" on your system load). ... View more

Sounds a bit strange. Real-time searches aren't that much different from normal searches and Splunk is taking care that only 1 search head in a pool is running each scheduled search. 1) Search heads don't distribute jobs to another search heads but their search peers (aka indexers). If your search heads are also indexers then I suppose it's normal you see some activity on both systems. 2) You can disable all scheduled searches on search head. I assume this would also disable real-time searches. See "how does search head pooling work with scheduled searches?" This might act as work-a-round for you problem. BTW: Are you sure you aren't sending the same data to both your indexers? Ie. how did you verified both alerts were triggered from the same SINGLE event? ... View more

Something to do with bloom filters maybe? ... View more

It's not really "never happened before" but have you considered alert throttling? I'm not sure how long time period you can define (and will it kill the performance) but in theory something like this* could do almost what you want. http://splunk-base.splunk.com/answers/52688/alerting-on-unique-field-occurrence ... View more

Are you sure you have followed instructions in the answer above and not just bumped your SPLUNKD_CONNECTION_TIMEOUT to 300? 300 seconds is just an example (that should be fine for most cases) but in your logs there seems to be some very long response times >300000ms. ... View more

Are you sure you have loaded test.browser.html from web server, not using your local file:// -URL? Using local files can not work because single origin policy, ie. javascript can only talk to host where it was loaded from. See "Dependencies and cross domain communication" at http://dev.splunk.com/view/splunk-javascript-sdk-getting-started/SP-CAAAEFM ... View more

As it says in docs, if/when your index reaches maxTotalDataSize (that is total size of hot, warm and cold data) Splunk will delete data starting from the oldest. The catch is, your index may never reach this size as there are other controls that will freeze your data (ie delete it) before the total size goes up to maxTotalDataSize. ... View more

Have you checked if there is packet loss between forwarder and indexer? Run "netstat -s | grep retransmited" on forwarder. It will give you absolute value so take couple samples when forwarder is sending data and see if value increases. If you have more than 1 forwarder, are others working better or do they all have similar performance problems? ... View more

Yep, it's just a longer list of bug fixes you're missing. My understanding is x.y.z -releases have only bug fixes. New features are introduced in x.y -releases. ... View more

Reading the release notes of 4.3.3 it seems it is just a bug-fix release http://docs.splunk.com/Documentation/Splunk/latest/ReleaseNotes/4.3.3 If you don't have any of these issues, then there is not much you will gain with upgrade. But it might be a good idea to upgrade your indexers anyway to avoid some of known bugs in future. I wouldn't worry too much about forwarders as there wasn't (m)any forwarder specific fixes included in 4.3.3. ... View more

Looks like I waited a bit too long and got second place in this race 🙂 ... View more

I think the problem is you try to assign "human readable time" into _time while it should be seconds from epoch. Good news is, you already have it from strptime(), ie you don't need those "convert ..." cmds unless you want log_time and log_date for some other purposes. Maybe this would work better ... | rex field=source "^(?P<path>[^\_]+)\/(?P<log_type>[^\/1]+)\_(?P<ip>[^\_]+)\_(?P<datetime>[^\_]+\_[^\_]+)" | eval _time=strptime(datetime, "%d/%m/%Y_%H:%M:%S") | timechart span=5min count by log_type If you want use only log_time (ie sum all dates into one) then bucket -cmd could help you. Simply dropping date from above example will make Splunk assume it's current date. ... View more

... | stats dc(IP) as visitor by app | sort visitor ... View more

Sounds like Splunk alert throttling? Extract TestClass ID -field, if you haven't done that already. ... | rex field=_raw "TestClass with ID (?<TestClassID>\d*)" And now you can set throttling on TestClassID like shown in here http://docs.splunk.com/Documentation/Splunk/latest/User/SchedulingSavedSearches#Define_alerts_that_trigger_in_real-time_whenever_a_result_matches (scroll down to "Set up throttling for a per-result alert") ... View more