Splunk Enterprise

Getting backup job failed messages

ansif
Motivator

My license master (which is used as dmc,deployment server,Cluster Master,Deployer) is throwing a lot of messages like below:

Scheduled backup job failed.The next scheduled backup job will run in an hour.
12/5/2017, 3:46:39 PM

Backup job "Default Scheduled Backup" has failed. Error: expected string or buffer
12/5/2017, 3:46:35 PM

alt text

Help me to identify and resolve this issue.

Tags (1)

twollenslegel_s
Splunk Employee
Splunk Employee

This is a known issue for ITSI 3.0.X
2017-07-03 ITOA-8115 ITSI app throws Errors during installation ITSI components on License server

http://docs.splunk.com/Documentation/ITSI/3.0.1/ReleaseNotes/Knownissues

aditya0473
Engager

I had the same issue. Mine was a dedicated license server.

I update inputs.conf in SA-ITOA/local/ dir with below settings.

[itsi_backup_restore://itsi_backup_restore]
disabled = 1

[itsi_scheduled_backup_caller://itsi_scheduled_backup]
disabled = 1

This solved the issue. As this is a license server there is no need of backup here.
Hope this helps.

NOTE: If license server is same as search head do not update these settings.

ansif
Motivator

Thanks @aditya0473 ,do you know where exactly this backup needed? Indexer? Search head?

0 Karma

aditya0473
Engager

Where ever ITSI is setup. Mostly on search heads. These settings are by default enabled, so where ITSI is up and running it shouldn't throw any error.

0 Karma

nickhills
Ultra Champion

I presume you are running ITSI?

I think I would be tempted to raise a ticket with Splunk for that issue, as it seems it should be configured out of the box
https://docs.splunk.com/Documentation/ITSI/3.0.0/Configure/BackupandRestoreITSIconfig

However, the first thing I would check is that the KV store is running ok.

If my comment helps, please give it a thumbs up!
0 Karma

ansif
Motivator

Hi nickhillscpl,

As per the document http://docs.splunk.com/Documentation/ITSI/3.0.0/Configure/InstallSplunkITServiceIntelligence
I have installed only A-ITSI-Licensechecker, SA-ITOA, and SA-UserAccess on License master.

Please find the below kvstore status:

[root@xxxxxxxxxxxxx ~]# /opt/splunk/bin/splunk show kvstore-status

 This member:
                                     date : Wed Dec  6 04:22:09 2017
                                  dateSec : 1512534129.847
                                 disabled : 0
                                    guid : xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxx
                     oplogEndTimestamp : Wed Dec  6 04:21:10 2017
                     oplogEndTimestampSec : 1512534070
                      oplogStartTimestamp : Sun Dec  3 23:40:04 2017
                     oplogStartTimestampSec : 1512344404
                                     port : xxxx
                               replicaSet : xxxxxxx-xxxx-xxxx-xxxxxxxxxxx
                        replicationStatus : KV store captain
                               standalone : 1
                                   status : ready

 KV store members:
        127.0.0.1:xxxx
                            configVersion : 1
                             electionDate : Wed Nov 15 13:49:35 2017
                          electionDateSec : 1510753775
                              hostAndPort : 127.0.0.1:xxxx
                               optimeDate : Wed Dec  6 04:21:10 2017
                            optimeDateSec : 1512534070
                        replicationStatus : KV store captain
                                   uptime : 1780355

Does License Master requires to run kvstore and backup jobs?

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...