topic Re: Getting backup job failed messages in Splunk Enterprise

Getting backup job failed messages

ansif — Tue, 05 Dec 2017 10:43:19 GMT

My license master (which is used as dmc,deployment server,Cluster Master,Deployer) is throwing a lot of messages like below:

Scheduled backup job failed.The next scheduled backup job will run in an hour.
12/5/2017, 3:46:39 PM

Backup job "Default Scheduled Backup" has failed. Error: expected string or buffer
12/5/2017, 3:46:35 PM

Help me to identify and resolve this issue.

Re: Getting backup job failed messages

nickhills — Tue, 05 Dec 2017 13:35:20 GMT

I presume you are running ITSI?

I think I would be tempted to raise a ticket with Splunk for that issue, as it seems it should be configured out of the box
https://docs.splunk.com/Documentation/ITSI/3.0.0/Configure/BackupandRestoreITSIconfig

However, the first thing I would check is that the KV store is running ok.

Re: Getting backup job failed messages

ansif — Wed, 06 Dec 2017 04:25:46 GMT

Hi nickhillscpl,

As per the document http://docs.splunk.com/Documentation/ITSI/3.0.0/Configure/InstallSplunkITServiceIntelligence
I have installed only A-ITSI-Licensechecker, SA-ITOA, and SA-UserAccess on License master.

Please find the below kvstore status:

[root@xxxxxxxxxxxxx ~]# /opt/splunk/bin/splunk show kvstore-status

 This member:
                                     date : Wed Dec  6 04:22:09 2017
                                  dateSec : 1512534129.847
                                 disabled : 0
                                    guid : xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxx
                     oplogEndTimestamp : Wed Dec  6 04:21:10 2017
                     oplogEndTimestampSec : 1512534070
                      oplogStartTimestamp : Sun Dec  3 23:40:04 2017
                     oplogStartTimestampSec : 1512344404
                                     port : xxxx
                               replicaSet : xxxxxxx-xxxx-xxxx-xxxxxxxxxxx
                        replicationStatus : KV store captain
                               standalone : 1
                                   status : ready

 KV store members:
        127.0.0.1:xxxx
                            configVersion : 1
                             electionDate : Wed Nov 15 13:49:35 2017
                          electionDateSec : 1510753775
                              hostAndPort : 127.0.0.1:xxxx
                               optimeDate : Wed Dec  6 04:21:10 2017
                            optimeDateSec : 1512534070
                        replicationStatus : KV store captain
                                   uptime : 1780355

Does License Master requires to run kvstore and backup jobs?

Re: Getting backup job failed messages

aditya0473 — Tue, 29 Sep 2020 17:46:38 GMT

I had the same issue. Mine was a dedicated license server.

I update inputs.conf in SA-ITOA/local/ dir with below settings.

[itsi_backup_restore://itsi_backup_restore]
disabled = 1

[itsi_scheduled_backup_caller://itsi_scheduled_backup]
disabled = 1

This solved the issue. As this is a license server there is no need of backup here.
Hope this helps.

NOTE: If license server is same as search head do not update these settings.

Re: Getting backup job failed messages

ansif — Tue, 23 Jan 2018 04:05:22 GMT

Thanks @aditya0473 ,do you know where exactly this backup needed? Indexer? Search head?

Re: Getting backup job failed messages

aditya0473 — Tue, 23 Jan 2018 16:23:37 GMT

Where ever ITSI is setup. Mostly on search heads. These settings are by default enabled, so where ITSI is up and running it shouldn't throw any error.

Re: Getting backup job failed messages

twollenslegel_s — Thu, 15 Feb 2018 22:29:27 GMT

This is a known issue for ITSI 3.0.X
2017-07-03 ITOA-8115 ITSI app throws Errors during installation ITSI components on License server

http://docs.splunk.com/Documentation/ITSI/3.0.1/ReleaseNotes/Knownissues