Why are alerts not working after upgrade to Splunk 6.5.0?

[alewkowicz]

Hi,

All of our alerts are not working after the upgrade to Splunk 6.5.0
In the scheduler.log I have this error :

ERROR SavedSplunker - vector::_M_range_check: __n (which is 0) >= this->size() (which is 0)

Anyone else have this issue ?

Thanks !

[christopherr_sp]

The issue with sending alerts in Splunk Enterprise 6.5.0 and 6.5.1, will be fixed in Splunk Enterprise 6.5.2 targeted for release by the end of January 2017.

SPL-131375
SavedSplunker ERROR message in scheduler.log needs more context ERROR SavedSplunker - vector::_M_range_check: __n (which is 0) >= this->size() (which is 0)

[andrea_o]

We too have the same problem, and we cannot wait until the end of January.

If the error is caused by a defective configuration file, can you please share a script (or the instructions) to detect which file is defective??????

[christopherr_sp]

The error is not caused by a defective .conf file. As soon as the Splunk Enterprise 6.5.2 is released, I will let you know via this Splunk Answers posting.

[andrea_o]

Meanwhile I've traced back the problem to an old (and forgotten) saved search.
This search referenced a now-dismissed lookup csv and its presence was totally fine under splunk 6.3, but caused havoc in 6.5.

[gmachacek]

I went through and cleaned up old alerts in saved searches and looks for funky characters, but couldn't get it to come up.

I ended up spinning up a new server, and installed splunk 6.4.5 on it
I pointed it at the existing splunk license master, and then added all the 6.5.1 indexers to it.
From the original search head I copied the /opt/splunk/etc/apps/search folder over.
I had a back up of that folder before I upgraded to 6.5.1, not sure if that would have caused issues if i had not had the old files.

It complains and says searching won't work (since indexers were on 6.5.1 and the search head is 6.4.5) but I have all my Production Alert/Reports working again. So I can at least get by until this patch.

[twinspop]

My problem with 6.5.1 scheduler wasn't invalid characters. It was repeated fields. Something like |stats last(FIELD1) as FIELD1 last(FIELD1) as FIELD1. I removed the repeaters and the scheduler immediately started working. The error was found in splunkd.log.

[christopherr_sp]

Splunk Enterprise 6.5.2 was released on 25 January 2016. This should fix your issue with
Alerts. The download link is below.

https://www.splunk.com/en_us/download/splunk-enterprise.htmlhttps://www.splunk.com/en_us/download/sp...

[EhpcLicenses]

Upgraded from 6.5.1 to 6.5.2 today. The issue did not appear in the new version. Thanks Christopher!

[gmachacek]

Same issue as well.
At least point us to how we can manually check/fix please.

[christopherr_sp]

You can manually check whether you have the issue in the file: SPLUNK_HOME/var/log/splunk/scheduler.log. Search for the string SavedSplunker and will see multiple
instances of the following:

SavedSplunker ERROR message in scheduler.log needs more context ERROR SavedSplunker - vector::M_range_check: _n (which is 0) >= this->size() (which is 0)

[alewkowicz · Accepted solution]

We have found a solution : the issue was the \n character (maybe a change with the SPL in the v6.5 ) in some of our alerts.

Please find below the answer of splunk support on this :

"We have a few related sounding known issues like this (listed below).

Your one actually isn't documented externally yet though.
Internal reference (which you can us when talking to support/accounts team is SPL-129846). It is a regression bug, and is due to be fixed in 6.5.1.

http://docs.splunk.com/Documentation/Splunk/6.5.0/ReleaseNotes/KnownIssues

SPL-34347 = wmi input default fields - with value including newlines doesn't search properly becasue of \r\n issue

SPL-74209, SPL-74167 = Persistent queues are not created on Windows for stanzas that contain unusual characters (such as < and >).
Workaround: Specify the persistentQueue explicitly in the input definition.

SPL-78179 = REST /saved/searches App names with special characters have invalid links. "

[kristianaasen]

Just upgraded to 6.5.1 and the problem is still there. Opening a supoort case.

[ultima]

Did you get a response back from Splunk ? We also have this error. Running version 6.5.1

[kristianaasen]

Supplying support with extra info as we speak.
I'll keep you posted.

If you open a case, please refer to: Case: 428672

[twinspop]

There was a scheduled search that had repeated fields in it. It was in splunkd.log. After fixing the search, searches immediately began firing again.

[twinspop]

Concur! This is no bueno

[sbrice]

I had the same problem with this alert on my search head "sourcetype=splunkd action=login status=failure" I monitor bad login events and trigger an email to splunk admins. However, after the 6.5 upgrade, I noticed alerts from this sourcetype were not working. I had to re-enable the monitor for "splunkd.log" Now my alerts are triggering.. Make sure you monitors are still in place.. From the Command line on your forwarders try "./bin/splunk list monitor" This will provide a list of monitors in place. Not sure why the splunkd.log dropped off, but now its being forwarded to the indexer fine!

[pietertruter]

Im having the same issues after upgrading to 6.5. Splunkd is definatley monitored and searchable from my indexers. No scheduled searches are running.

[dalesheils]

I now have this issue in Norway. After upgrade to 6.5 triggered alerts fail.

[burwell]

Can you say more? Given an example? Do you mean you don't see the alert in the list of triggered alerts?

Thanks.