Splunk Enterprise

Getting backup job failed messages

ansif
Motivator

My license master (which is used as dmc,deployment server,Cluster Master,Deployer) is throwing a lot of messages like below:

Scheduled backup job failed.The next scheduled backup job will run in an hour.
12/5/2017, 3:46:39 PM

Backup job "Default Scheduled Backup" has failed. Error: expected string or buffer
12/5/2017, 3:46:35 PM

alt text

Help me to identify and resolve this issue.

Tags (1)

twollenslegel_s
Splunk Employee
Splunk Employee

This is a known issue for ITSI 3.0.X
2017-07-03 ITOA-8115 ITSI app throws Errors during installation ITSI components on License server

http://docs.splunk.com/Documentation/ITSI/3.0.1/ReleaseNotes/Knownissues

aditya0473
Engager

I had the same issue. Mine was a dedicated license server.

I update inputs.conf in SA-ITOA/local/ dir with below settings.

[itsi_backup_restore://itsi_backup_restore]
disabled = 1

[itsi_scheduled_backup_caller://itsi_scheduled_backup]
disabled = 1

This solved the issue. As this is a license server there is no need of backup here.
Hope this helps.

NOTE: If license server is same as search head do not update these settings.

ansif
Motivator

Thanks @aditya0473 ,do you know where exactly this backup needed? Indexer? Search head?

0 Karma

aditya0473
Engager

Where ever ITSI is setup. Mostly on search heads. These settings are by default enabled, so where ITSI is up and running it shouldn't throw any error.

0 Karma

nickhills
Ultra Champion

I presume you are running ITSI?

I think I would be tempted to raise a ticket with Splunk for that issue, as it seems it should be configured out of the box
https://docs.splunk.com/Documentation/ITSI/3.0.0/Configure/BackupandRestoreITSIconfig

However, the first thing I would check is that the KV store is running ok.

If my comment helps, please give it a thumbs up!
0 Karma

ansif
Motivator

Hi nickhillscpl,

As per the document http://docs.splunk.com/Documentation/ITSI/3.0.0/Configure/InstallSplunkITServiceIntelligence
I have installed only A-ITSI-Licensechecker, SA-ITOA, and SA-UserAccess on License master.

Please find the below kvstore status:

[root@xxxxxxxxxxxxx ~]# /opt/splunk/bin/splunk show kvstore-status

 This member:
                                     date : Wed Dec  6 04:22:09 2017
                                  dateSec : 1512534129.847
                                 disabled : 0
                                    guid : xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxx
                     oplogEndTimestamp : Wed Dec  6 04:21:10 2017
                     oplogEndTimestampSec : 1512534070
                      oplogStartTimestamp : Sun Dec  3 23:40:04 2017
                     oplogStartTimestampSec : 1512344404
                                     port : xxxx
                               replicaSet : xxxxxxx-xxxx-xxxx-xxxxxxxxxxx
                        replicationStatus : KV store captain
                               standalone : 1
                                   status : ready

 KV store members:
        127.0.0.1:xxxx
                            configVersion : 1
                             electionDate : Wed Nov 15 13:49:35 2017
                          electionDateSec : 1510753775
                              hostAndPort : 127.0.0.1:xxxx
                               optimeDate : Wed Dec  6 04:21:10 2017
                            optimeDateSec : 1512534070
                        replicationStatus : KV store captain
                                   uptime : 1780355

Does License Master requires to run kvstore and backup jobs?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...