Alerting

Why are alerts not working after upgrade to Splunk 6.5.0?

Explorer

Hi,

All of our alerts are not working after the upgrade to Splunk 6.5.0
In the scheduler.log I have this error :

ERROR SavedSplunker - vector::_M_range_check: __n (which is 0) >= this->size() (which is 0)

Anyone else have this issue ?

Thanks !

1 Solution

Explorer

We have found a solution : the issue was the \n character (maybe a change with the SPL in the v6.5 ) in some of our alerts.

Please find below the answer of splunk support on this :

"We have a few related sounding known issues like this (listed below).

Your one actually isn't documented externally yet though.
Internal reference (which you can us when talking to support/accounts team is SPL-129846). It is a regression bug, and is due to be fixed in 6.5.1.

http://docs.splunk.com/Documentation/Splunk/6.5.0/ReleaseNotes/KnownIssues

SPL-34347 = wmi input default fields - with value including newlines doesn't search properly becasue of \r\n issue

SPL-74209, SPL-74167 = Persistent queues are not created on Windows for stanzas that contain unusual characters (such as < and >).
Workaround: Specify the persistentQueue explicitly in the input definition.

SPL-78179 = REST /saved/searches App names with special characters have invalid links. "

View solution in original post

Splunk Employee
Splunk Employee

The issue with sending alerts in Splunk Enterprise 6.5.0 and 6.5.1, will be fixed in Splunk Enterprise 6.5.2 targeted for release by the end of January 2017.

SPL-131375
SavedSplunker ERROR message in scheduler.log needs more context ERROR SavedSplunker - vector::_M_range_check: __n (which is 0) >= this->size() (which is 0)

Explorer

We too have the same problem, and we cannot wait until the end of January.

If the error is caused by a defective configuration file, can you please share a script (or the instructions) to detect which file is defective??????

0 Karma

Splunk Employee
Splunk Employee

The error is not caused by a defective .conf file. As soon as the Splunk Enterprise 6.5.2 is released, I will let you know via this Splunk Answers posting.

0 Karma

Explorer

Meanwhile I've traced back the problem to an old (and forgotten) saved search.
This search referenced a now-dismissed lookup csv and its presence was totally fine under splunk 6.3, but caused havoc in 6.5.

0 Karma

New Member

I went through and cleaned up old alerts in saved searches and looks for funky characters, but couldn't get it to come up.

I ended up spinning up a new server, and installed splunk 6.4.5 on it
I pointed it at the existing splunk license master, and then added all the 6.5.1 indexers to it.
From the original search head I copied the /opt/splunk/etc/apps/search folder over.
I had a back up of that folder before I upgraded to 6.5.1, not sure if that would have caused issues if i had not had the old files.

It complains and says searching won't work (since indexers were on 6.5.1 and the search head is 6.4.5) but I have all my Production Alert/Reports working again. So I can at least get by until this patch.

0 Karma

Influencer

My problem with 6.5.1 scheduler wasn't invalid characters. It was repeated fields. Something like |stats last(FIELD1) as FIELD1 last(FIELD1) as FIELD1. I removed the repeaters and the scheduler immediately started working. The error was found in splunkd.log.

0 Karma

Splunk Employee
Splunk Employee

Splunk Enterprise 6.5.2 was released on 25 January 2016. This should fix your issue with
Alerts. The download link is below.

https://www.splunk.com/en_us/download/splunk-enterprise.htmlhttps://www.splunk.com/en_us/download/sp...

Engager

Upgraded from 6.5.1 to 6.5.2 today. The issue did not appear in the new version. Thanks Christopher!

0 Karma

New Member

Same issue as well.
At least point us to how we can manually check/fix please.

0 Karma

Splunk Employee
Splunk Employee

You can manually check whether you have the issue in the file: SPLUNK_HOME/var/log/splunk/scheduler.log. Search for the string SavedSplunker and will see multiple
instances of the following:

SavedSplunker ERROR message in scheduler.log needs more context ERROR SavedSplunker - vector::M_range_check: _n (which is 0) >= this->size() (which is 0)

0 Karma

Explorer

We have found a solution : the issue was the \n character (maybe a change with the SPL in the v6.5 ) in some of our alerts.

Please find below the answer of splunk support on this :

"We have a few related sounding known issues like this (listed below).

Your one actually isn't documented externally yet though.
Internal reference (which you can us when talking to support/accounts team is SPL-129846). It is a regression bug, and is due to be fixed in 6.5.1.

http://docs.splunk.com/Documentation/Splunk/6.5.0/ReleaseNotes/KnownIssues

SPL-34347 = wmi input default fields - with value including newlines doesn't search properly becasue of \r\n issue

SPL-74209, SPL-74167 = Persistent queues are not created on Windows for stanzas that contain unusual characters (such as < and >).
Workaround: Specify the persistentQueue explicitly in the input definition.

SPL-78179 = REST /saved/searches App names with special characters have invalid links. "

View solution in original post

New Member

Just upgraded to 6.5.1 and the problem is still there. Opening a supoort case.

0 Karma

Explorer

Did you get a response back from Splunk ? We also have this error. Running version 6.5.1

0 Karma

New Member

Supplying support with extra info as we speak.
I'll keep you posted.

If you open a case, please refer to: Case: 428672

0 Karma

Influencer

There was a scheduled search that had repeated fields in it. It was in splunkd.log. After fixing the search, searches immediately began firing again.

0 Karma

Influencer

Concur! This is no bueno

0 Karma

Explorer

I had the same problem with this alert on my search head "sourcetype=splunkd action=login status=failure" I monitor bad login events and trigger an email to splunk admins. However, after the 6.5 upgrade, I noticed alerts from this sourcetype were not working. I had to re-enable the monitor for "splunkd.log" Now my alerts are triggering.. Make sure you monitors are still in place.. From the Command line on your forwarders try "./bin/splunk list monitor" This will provide a list of monitors in place. Not sure why the splunkd.log dropped off, but now its being forwarded to the indexer fine!

0 Karma

New Member

Im having the same issues after upgrading to 6.5. Splunkd is definatley monitored and searchable from my indexers. No scheduled searches are running.

0 Karma

New Member

I now have this issue in Norway. After upgrade to 6.5 triggered alerts fail.

0 Karma

SplunkTrust
SplunkTrust

Can you say more? Given an example? Do you mean you don't see the alert in the list of triggered alerts?

Thanks.

0 Karma