Version Control for Splunk - "Restore has failed to complete successfully"

[dshatto]

@Garethan

When trying to test the app I get the following when trying to restore a dashboard 

"Restore has failed to complete successfully in app search, object of type dashboard, with name <object name> "

I have confirmed that Im' using the Object URI/Name (not label).

In looking at the backup.log it is giving two warnings and an error:

WARNING i="splunkVCBackup" error/fatal messages in git stderroutput please review. stderrout="b'\n*** Please tell me who you are.\n\nRun\n\n git config --global user.email "you@example.com"\n git config --global user.name "Your Name"\n\nto set your account\'s default identity.\nOmit --global to set the identity only in this repository.\n\nfatal: empty ident name (for <splunk@<server.xx.xxx>) not allowed\nTo git@gitlab.com-<repo>/sharedteams/<project>/<repo>.git\n * [new tag] 2020-08-03_0746 -> 2020-08-03_0746\n'"

ERROR i="splunkVCBackup" git failure occurred during runtime, not updating the epoch value. This failure may require investigation, please refer to the WARNING messages in the logs

WARNING i="splunkVCBackup" wiping the git directory, dir=/opt/<repo dir>/vcbackup to allow re-cloning on next run of the script.  vcbackup is the temp directory under the main repo.

[gjanders]

Thanks for the direct email, looks like I don't get notified when someone mentions my application anymore 😞

I suspect the git command on this Linux server is refusing to do a checkout because the git configuration doesn't exist.

Something like:

git config --global user.email "you@example.com"

git config --global user.name "Your Name"

Run as the user running Splunk should fix it, although you might want to use a real email/name.

In the manual on github (or README.md in the app) I have mentioned under the "Can I use this application on Windows?":

And set the git global configuration git config --global user.name "John Doe" git config --global user.email johndoe@example.com

That's what I had to do on Windows to make the git clone/git checkout command work as expected...

[dshatto]

Thanks for the response Gareth.

I checked the git config on the Linux box logged in as the Splunk user.  The user.email and user.name are currently set.

[gjanders]

Try setting debugMode to true and run it again and the logs should advise which command threw that error 

[dshatto]

When running requests.get() on the url output_mode=json is showing a 404

[gjanders]

If you can trim the https://<yourservername> part out, I'd just need to see the endpoint it is trying to hit.

Most likely the application name which defaults to SplunkVersionControl has been changed and a 404 will occur.

I have a setting called "remoteAppName" which is supposed to be the name of the app on the remote server, but I might have some code with the app name hardcoded somewhere.

Which server is throwing the 404? And what action is taken (or which log file) are you seeing it in?

Or I might have some context with more log entries..

[dshatto]

Here are the lines from splunkversioncontrol_rest_restore.log

020-08-20 13:48:03,821 root INFO Received remote request checking username and role related to the token on url=https:///services/authentication/current-context?output_mode=json
2020-08-20 13:48:03,830 root INFO username= roles=['admin']
2020-08-20 13:48:03,860 root INFO Sleeping for 30 seconds to wait for audit logs
2020-08-20 13:48:33,890 root INFO Sleep completed
2020-08-20 13:48:35,683 root INFO i="splunkVCRestore" Splunk Version Control Restore run with arguments="{'debugMode': 'true', 'destURL': 'https://58:8089', 'destUsername': 'splunkvcgituser', 'disabled': False, 'eai:acl': None, 'gitRepoURL': 'git@gitlab.com-pt9splunk:ciorg/sharedteams/ptn/pt9splunk.git', 'gitTempDir': '/opt/pt9splunk/vcrestore', 'host': 'GMC0449', 'host_resolved': 'xxxx', 'index': 'main', 'interval': '', 'python.version': None, 'name': 'splunkversioncontrol_restore://splunkVCRestore'}"
2020-08-20 13:48:35,687 root DEBUG Begin OS process run of cd /opt/xxxx/vcrestore; git checkout master; git pull
2020-08-20 13:48:37,124 root DEBUG OS process exited with zero code, for command cd /opt/xxxx/vcrestore; git checkout master; git pull
2020-08-20 13:48:37,125 root INFO i="splunkVCRestore" Successfully ran the git pull for URL=xxxx/sharedteams/xxx/xxxx.git from directory dir=/opt/xxxx/vcrestore
2020-08-20 13:48:37,125 root DEBUG i="splunkVCRestore" The restore list is [{'app': 'search', 'type': 'savedsearch', 'name': 'dds_git_test', 'tag': '2020-08-20_1345', 'scope': 'all', 'time': 1597952793, 'restoreAsUser': '', 'user': 'xxxx'}]
2020-08-20 13:48:37,125 root DEBUG i="splunkVCRestore" Running requests.post() on url=https://xxxx.58:8089/servicesNS/-/SplunkVersionControl/search/jobs with user=xxxx query="| savedsearch "SplunkVersionControl CheckAdmin" ldapFilter="*xxxx", usernameFilter="xxxx"" proxies_length=0
2020-08-20 13:48:37,303 root INFO i="splunkVCRestore" user=dougs has requested the object with name=dds_git_test of type=savedsearch to be restored from tag=2020-08-20_1345 and scope=all, restoreAsUser=, this was requested at time=1597952793 in app context of app=search
2020-08-20 13:48:37,304 root DEBUG Begin OS process run of cd /opt/xxxx/vcrestore; git checkout 2020-08-20_1345
2020-08-20 13:48:37,315 root DEBUG OS process exited with zero code, for command cd /opt/pt9splunk/vcrestore; git checkout 2020-08-20_1345
2020-08-20 13:48:37,316 root INFO i="splunkVCRestore" Successfully ran the git checkout for URL=xxxx/sharedteams/ptn/xxxx.git from directory dir=/opt/xxxxx/vcrestore
2020-08-20 13:48:37,316 root DEBUG i="splunkVCRestore" Known app list is
2020-08-20 13:48:37,316 root INFO i="splunkVCRestore" user=xxxx, attempting to restore name=dds_git_test in app=search of type=savedsearches in scope=all, restoreAsUser=, adminLevel=False
2020-08-20 13:48:37,316 root DEBUG i="splunkVCRestore" Running requests.get() on url=https://xxxx.58:8089/servicesNS/-/search/saved/searches/dds_git_test?output_mode=json with user=xxxx in app=search proxies_length=0
2020-08-20 13:48:37,378 root DEBUG i="splunkVCRestore" URL=https://xxxx.58:8089/servicesNS/-/search/saved/searches/dds_git_test?output_mode=json is throwing a 404, assuming new object creation
2020-08-20 13:48:37,378 root DEBUG i="splunkVCRestore" user=xxxx, name=dds_git_test, found typeFile=/opt/xxxx/vcrestore/search/user/savedsearches to restore from
2020-08-20 13:48:37,399 root INFO i="splunkVCRestore" user=xxxx, name=dds_git_test not found at scope=user in file=/opt/xxxx/vcrestore/search/user/savedsearches
2020-08-20 13:48:37,400 root DEBUG i="splunkVCRestore" user=xxxx, name=dds_git_test, found typeFile=/opt/xxxx/vcrestore/search/app/savedsearches to restore from
2020-08-20 13:48:37,407 root INFO i="splunkVCRestore" user=xxxx, name=dds_git_test not found at app level scope in typeFile=/opt/xxxx/vcrestore/search/app/savedsearches
2020-08-20 13:48:37,407 root DEBUG i="splunkVCRestore" user=xxxx, name=dds_git_test, found typeFile=/opt/xxxx/vcrestore/search/global/savedsearches to restore from
2020-08-20 13:48:37,409 root INFO i="splunkVCRestore" user=xxxx, name=dds_git_test not found at scope=global in typeFile=/opt/xxxx/vcrestore/search/global/savedsearches
2020-08-20 13:48:37,409 root WARNING i="splunkVCRestore" user=xxxx attempted to restore name=dds_git_test, type=savedsearches, restoreAsUser=, adminLevel=False however the object was not found, the restore was unsuccessful. Perhaps check the restore date, scope & capitilisation before trying again?
2020-08-20 13:48:37,409 root INFO i="splunkVCRestore" Done
2020-08-20 13:48:37,409 root WARNING Restore has failed to complete successfully in app=search, object of type=savedsearch, with name=dds_git_test was not restored from tag=2020-08-20_1345, scope=all with restoreAsUser= and requested by username=xxxx

The restore is being requested from a member of the SHC and is the server that is throwing the 404 error.  The application and the git repo are on the deployment server.  The SplunkVCRestore input is pointing to .58 which is one of the SHC members.

I thought that maybe it was because I wasn't trying to do the restore from .58 so I logged into that server but it still fails on the restore.

[gjanders]

So did the search app have a savedsearch named dds_git_test in that tag?

The code is saying it cannot find the saved search with that name in the backup...

[dshatto]

It is recording the name in the splunkversioncontrol_backup.log with the correct type (saved searches), app (search) and owner (dougs)

I am getting this info in the same log file: (NOTE: Items in bold are redacted)

2020-08-24 08:32:43,243 root DEBUG Begin OS process run of cd /opt/pt9splunk/vcbackup; git add -A; git commit -am "Updated by Splunk Version Control backup job splunkVCBackup"; git tag 2020-08-24_0832; git push origin master --tags
2020-08-24 08:32:46,050 root DEBUG OS process exited with zero code, for command cd /opt/{GIT repo}/vcbackup; git add -A; git commit -am "Updated by Splunk Version Control backup job splunkVCBackup"; git tag 2020-08-24_0832; git push origin master --tags
2020-08-24 08:32:46,051 root WARNING i="splunkVCBackup" error/fatal messages in git stderroutput please review. stderrout="b'\n*** Please tell me who you are.\n\nRun\n\n git config --global user.email "you@example.com"\n git config --global user.name "Your Name"\n\nto set your account\'s default identity.\nOmit --global to set the identity only in this repository.\n\nfatal: empty ident name (for <splunk@{deployer}>) not allowed\nTo git@gitlab.com-{repo}\n * [new tag] 2020-08-24_0832 -> 2020-08-24_0832\n'"
2020-08-24 08:32:46,051 root DEBUG i="splunkVCBackup" Running requests.post() on url=https://{SH}/servicesNS/-/local/search/jobs with user=splunkvcgituser query="| makeresults | eval tag="2020-08-24_0832" | fields - _time | outputlookup append=t splunkversioncontrol_taglist" proxies_length=0
2020-08-24 08:32:46,246 root INFO i="splunkVCBackup" messages from query="| makeresults | eval tag="2020-08-24_0832" | fields - _time | outputlookup append=t splunkversioncontrol_taglist" were messages="[{'type': 'INFO', 'text': "Results written to file '/opt/splunk/etc/apps/local/lookups/splunkversioncontrol_taglist.csv' on serverName='{SH}'."}]"
2020-08-24 08:32:46,246 root ERROR i="splunkVCBackup" git failure occurred during runtime, not updating the epoch value. This failure may require investigation, please refer to the WARNING messages in the logs
2020-08-24 08:32:46,246 root WARNING i="splunkVCBackup" wiping the git directory, dir=/opt/{GIT repo}/vcbackup to allow re-cloning on next run of the script
2020-08-24 08:32:46,349 root INFO i="splunkVCBackup" Done

I have confirmed the .gitconfig has the user.email and user.name but it continues to give a "Warning" in the log file.

[gjanders]

So if you login to the server where these logs are and run the below as the same username that is running Splunk:

cd /opt/{GIT repo}/vcbackup; git add -A; git commit -am "Updated by Splunk Version Control backup job splunkVCBackup"; git tag 2020-08-24_0832; git push origin master --tags

Note that you might have to edit a file or similar to make the git add/commit work, does that throw an error?

Because all the code does is run a shell that runs the above...so it should be identical to what you see as a command line user (assuming you are running as the same username that runs splunk)

[dshatto]

When I first cd to the git repo there is no vcbackup folder.

If I recall correctly from previous conversations you indicated to just let the Data Input created the temp directory.  So I went into the backup data input and changed the temp directory to vcbackup1 and confirmed that it created the directory, which it did.

I then did cd to that directory and ran the commands you suggested.  I had to use a different time for git tag because there was already one with that date/time.  It didn't throw any errors  on those commands.

It is still giving the Warning about the git config user.email and the error about the epoch value.

[gjanders]

I think something like this will work:

Go to the bin directory of the SplunkVersionControl app then:

run:

splunk cmd python

In the python window try:

from splunkversioncontrol_utility import runOSProcess, get_password
import logging
logger = logging.getLogger()
(output, stderrout, res) = runOSProcess("cd ~; echo `pwd` `whoami`; git config --list --global", logger, shell=True)

print(output)

print(stderrout)

I'll run some testing a bit later to confirm that should work but I believe that will print the git config

[gjanders]

With that runOSProcess you can execute any shell command so you could test a :

cd <your git directory>; echo `pwd`; touch file; git add file; git commit -m test; git push origin master

Or similar and see what happens 🙂

[dshatto]

Spoiler

error: failed to push some refs to 'git@gitlab.com-<repo>/<repo>/<repo>/.git'
hint: Updates were rejected because the remote contains work that you do
hint: not have locally. This is usually caused by another repository pushing
hint: to the same ref. You may want to first merge the remote changes (e.g.,
hint: 'git pull') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.

[gjanders]

Perhaps run a 
git pull

In that directory first 

[dshatto]

The git pull worked fine and the commands ran fine after the pull as well.

One thing I missed was your response earlier about the python commands to print the .gitconfig but it's not working

Here is the config

user.email=<my_email>
user.name=<my_userid>
push.default=simple

[gjanders]

Something doesn't seem right here, so the git commands work fine?

Just to confirm the git commands are been run on the server that is running the actual backup/restore from git? Which might not be the one requesting the restore of the object...

[dshatto]

Correct, the Git commands were run on the deployment server which is connected to the Git repo

[gjanders]

So something is happening that is not obvious at all 😞

Is there an SSH key or something in use?

When you run this:

Go to the bin directory of the SplunkVersionControl app then:

run:

splunk cmd python

In the python window try:

from splunkversioncontrol_utility import runOSProcess, get_password
import logging
logger = logging.getLogger()
(output, stderrout, res) = runOSProcess("cd ~; echo `pwd` `whoami`; git config --list --global", logger, shell=True)

print(output)

print(stderrout)

Which step or steps are failing? Clearly when splunk runs the commands it is getting a different result to when you run the commands in the shell...

[dshatto]

Ok, let's break this down a bit.

We are using an ssh key to connect to the repository.

I have gone to opt/splunk/etc/apps/splunkversioncontrol/bin and ran splunk cmd python

From there I successfully ran the commands you listed below.  Here are the results.

>>> (output,stderrout, res) = runOSProcess("cd ~;echo 'pwd' 'whoami';git config --list --global",logger,shell=True)
>>> print(output)
pwd whoami
user.email=<my email address>
user.name=dshatto
push.default=simple

>>> print(stderrout)

I don't recall exactly where it was failing but I suspect that it was that I separating part of the command

[gjanders]

Ok, so perhaps try:

splunk cmd splunkd print-modinput-config splunkversioncontrol_backup splunkversioncontrol_backup://replaceme | splunk cmd python simpleinput.py | splunk cmd python /opt/splunk/etc/apps/SplunkVersionControl/bin/splunkversioncontrol_backup.py

Where "replaceme" is the name of your input name, and simpleinput.py is from:

https://community.splunk.com/t5/Getting-Data-In/How-do-you-test-a-modular-input-on-the-command-line-...

I borrowed the code and updated the password with my local one....all the above will do is run the code within your current shell which might be the missing link...the above will test backup but restore is a very similar process (replace the backup with restore in the python file & input name)