Version Control for Splunk - "Restore has failed to complete successfully"

[dshatto]

@Garethan

When trying to test the app I get the following when trying to restore a dashboard 

"Restore has failed to complete successfully in app search, object of type dashboard, with name <object name> "

I have confirmed that Im' using the Object URI/Name (not label).

In looking at the backup.log it is giving two warnings and an error:

WARNING i="splunkVCBackup" error/fatal messages in git stderroutput please review. stderrout="b'\n*** Please tell me who you are.\n\nRun\n\n git config --global user.email "you@example.com"\n git config --global user.name "Your Name"\n\nto set your account\'s default identity.\nOmit --global to set the identity only in this repository.\n\nfatal: empty ident name (for <splunk@<server.xx.xxx>) not allowed\nTo git@gitlab.com-<repo>/sharedteams/<project>/<repo>.git\n * [new tag] 2020-08-03_0746 -> 2020-08-03_0746\n'"

ERROR i="splunkVCBackup" git failure occurred during runtime, not updating the epoch value. This failure may require investigation, please refer to the WARNING messages in the logs

WARNING i="splunkVCBackup" wiping the git directory, dir=/opt/<repo dir>/vcbackup to allow re-cloning on next run of the script.  vcbackup is the temp directory under the main repo.

[dshatto]

Backup results:

Traceback (most recent call last):
File "/opt/splunk/etc/apps/SplunkVersionControl/bin/splunkversioncontrol_backup.py", line 294, in <module>
vc.run_script()
File "/opt/splunk/etc/apps/SplunkVersionControl/bin/splunkversioncontrol_backup_class.py", line 931, in run_script
config = self.get_config()
File "/opt/splunk/etc/apps/SplunkVersionControl/bin/splunkversioncontrol_backup_class.py", line 120, in get_config
raise Exception("Error getting Splunk configuration via STDIN: %s" % str(e))
Exception: Error getting Splunk configuration via STDIN: list index out of range

Restore results:

Traceback (most recent call last):
File "/opt/splunk/etc/apps/SplunkVersionControl/bin/splunkversioncontrol_restore.py", line 279, in <module>
vc.run_script()
File "/opt/splunk/etc/apps/SplunkVersionControl/bin/splunkversioncontrol_restore_class.py", line 852, in run_script
config = self.get_config()
File "/opt/splunk/etc/apps/SplunkVersionControl/bin/splunkversioncontrol_restore_class.py", line 111, in get_config
raise Exception("Error getting Splunk configuration via STDIN: %s" % str(e))
Exception: Error getting Splunk configuration via STDIN: list index out of range

[gjanders]

Ok sure, so let's start a bit smaller:

splunk cmd splunkd print-modinput-config splunkversioncontrol_backup splunkversioncontrol_backup://replaceme

Does that print you current config?

splunk cmd splunkd print-modinput-config splunkversioncontrol_backup splunkversioncontrol_backup://replaceme | splunk cmd python simpleinput.py

Does that keep the current config? (it adds in a session token if it's working, where simpleinput.py is from that other answers post updated with your username/password)

If the above two work then the final | to the backup script should work, if not then it will fail 🙂

[dshatto]

If you're asking if the first command prints the git config the answer would be no.

Results for first command:

<?xml version="1.0" encoding="UTF-8"?>
<input>
<server_host>xxx0449</server_host>
<server_uri>https://127.0.0.1:8089</server_uri>
<session_key>PfNoaFwCGcToSSVQ5kZArE8o9_Xo_2fHjkoVVcd3mrdcIQrgDZzwIqKadLrac1sSjjTGYlGV9pTaeqo4zYbr3OQSCZlllY^gv1Se_POL0QxRdCmxnG3MlpKqOo</session_key>
<checkpoint_dir>/opt/splunk/var/lib/splunk/modinputs/splunkversioncontrol_backup</checkpoint_dir>
<configuration/>
</input>

[dshatto]

Update:

Ok so I ran the two commands here are the results for each.

First:

[splunk@xxx0449 bin]$ splunk cmd splunkd print-modinput-config splunkversioncontrol_backup splunkversioncontrol_backup://replaceme
<?xml version="1.0" encoding="UTF-8"?>
<input>
<server_host>xxx0449</server_host>
<server_uri>https://127.0.0.1:8089</server_uri>
<session_key>IpQDTnImlHjbWXWbD3CZBS6QNk5PlYNETW^zCK9CkkUXJoXRx6a22CahehtQysSoYo6BUOJUac0orTGXmvjXZutToHVRKqqkjr_NRf5nklFfQ8PJWXAzZdC</session_key>
<checkpoint_dir>/opt/splunk/var/lib/splunk/modinputs/splunkversioncontrol_backup</checkpoint_dir>
<configuration/>
</input>

Second:

[splunk@xxx0449 bin]$ splunk cmd splunkd print-modinput-config splunkversioncontrol_backup splunkversioncontrol_backup://replaceme | splunk cmd python simpleinput.py
<?xml version="1.0" encoding="UTF-8"?>
<input>
<server_host>xxx0449</server_host>
<server_uri>https://127.0.0.1:8089</server_uri>
<session_key>NoK09aPhdzZ7OS9aJEYXfQS4hWqeZ0j6ZBFZPMUNA4bltpBbkYHTjS7HelV9qj0geqGKLI7lubiZpyYZkmPLk5L3DLvpQ3wndAupY94gDklr9C</session_key>
<checkpoint_dir>/opt/splunk/var/lib/splunk/modinputs/splunkversioncontrol_backup</checkpoint_dir>
<configuration/>
</input>

[gjanders]

Just to confirm you are changing replaceme to match the actual name in the data inputs -> splunk version control backup?

The configuration section should have something like:

<configuration>
<stanza name="splunkversioncontrol_backup://namegoeshere" app="SplunkVersionControl">

With a list under configuration...

[dshatto]

Sorry, guess I didn't catch that.  I have rerun both commands changing it to the input name:

[splunk@xx0449 bin]$ splunk cmd splunkd print-modinput-config splunkversioncontrol_backup splunkversioncontrol_backup://splunkVCbackup
<?xml version="1.0" encoding="UTF-8"?>
<input>
<server_host>xx0449</server_host>
<server_uri>https://127.0.0.1:8089</server_uri>
<session_key>fBmnf6SSgtAQGAA1WVVHkAr86CeqKYEKfFWCOm5NQn5pCcIsufbseTxVdziEGkogm7IGgsWHyApIOzOlIxyPe77LPV86K1FFAK8rxvkRWIB_A^hfqSr</session_key>
<checkpoint_dir>/opt/splunk/var/lib/splunk/modinputs/splunkversioncontrol_backup</checkpoint_dir>
<configuration/>
</input>
[splunk@xx0449 bin]$ splunk cmd splunkd print-modinput-config splunkversioncontrol_backup splunkversioncontrol_backup://splunkVCbackup | splunk cmd python simpleinput.py
<?xml version="1.0" encoding="UTF-8"?>
<input>
<server_host>xx0449</server_host>
<server_uri>https://127.0.0.1:8089</server_uri>
<session_key>loPzTBXmABI9UqlhOT2_x1LhepL9_W8vSZQy_YkM^^5e_qhDuxCRJQDr73Piwx1CVaSdTEi6eu91AKSWRJk_hCstzfsP1sOkl7fMNt^RsqtAmaGR2F</session_key>
<checkpoint_dir>/opt/splunk/var/lib/splunk/modinputs/splunkversioncontrol_backup</checkpoint_dir>
<configuration/>
</input>

[gjanders]

<configuration/>

Is a problem, if you are on <yoursplunkinstance>/en-GB/manager/search/data/inputs/splunkversioncontrol_backup

Does the splunkVCbackup appear as the name?

If so you should see some information under <configuration/> from that command...

[dshatto]

The data input is actually named splunkVCBackup not splunkVCbackup.

Here are the results after rerunning the commands:

First:

<input>
<server_host>xx0449</server_host>
<server_uri>https://127.0.0.1:8089</server_uri>
<session_key>uzIbJ1FWmEyWVag7Dt7BVI2RMXAXZO2KRsek7hcu3ntKNjMIy307D1DBWgEnyiNxVM0stfwHLrSqEwE3P3fadwj8b8aGzjlyZP8xF81K^AnPj9maThWbCF</session_key>
<checkpoint_dir>/opt/splunk/var/lib/splunk/modinputs/splunkversioncontrol_backup</checkpoint_dir>
<configuration>
<stanza name="splunkversioncontrol_backup://splunkVCBackup" app="search">
<param name="debugMode">true</param>
<param name="gitRepoURL">git@gitlab.com-pt9splunk:ciorg/sharedteams/ptn/pt9splunk.git</param>
<param name="gitTempDir">/opt/pt9splunk/vcbackup1</param>
<param name="host">xxx0449</param>
<param name="includeEntities"></param>
<param name="index">main</param>
<param name="interval">300</param>
<param name="noDisabled">false</param>
<param name="noPrivate">false</param>
<param name="python.version">python3</param>
<param name="remoteAppName">local</param>
<param name="srcPassword">c0mp4ss10n!</param>
<param name="srcURL">https://xx.xx.xx.58:8089</param>
<param name="srcUsername">splunkvcgituser</param>
</stanza>
</configuration>
</input>

Second:

<input>
<server_host>xxx0449</server_host>
<server_uri>https://127.0.0.1:8089</server_uri>
<session_key>9rrMOzI4oyNcG156YdUK5t68WTRHEVbaMFj6VlkxhPNBKAHM6U^DXue24icQ3kFkxo8d1ghTXXtFUkfNBEA6pYXYKZhiRLlql3Whb7hSikK0RlS</session_key>
<checkpoint_dir>/opt/splunk/var/lib/splunk/modinputs/splunkversioncontrol_backup</checkpoint_dir>
<configuration/>
</input>

[gjanders]

splunk cmd splunkd print-modinput-config splunkversioncontrol_backup splunkversioncontrol_backup://splunkVCBackup | splunk cmd python /opt/splunk/etc/apps/SplunkVersionControl/bin/splunkversioncontrol_backup.py

Will probably fail but:

splunk cmd splunkd print-modinput-config splunkversioncontrol_backup splunkversioncontrol_backup://splunkVCBackup | splunk cmd python simpleinput.py | splunk cmd python /opt/splunk/etc/apps/SplunkVersionControl/bin/splunkversioncontrol_backup.py

Should have worked, I'm unsure why you are not seeing the <configuration> section after adding the simpleinput.py into the mix, it worked on my env.

Let me re-test again

[dshatto]

Any update?

[gjanders]

Sorry I tested and they worked fine for me!

So just to confirm, is the git repository getting commits related to backups?

If not is the splunkversioncontrol_backup log on the server running the backups still throwing the error about the name not been configured?

Finally, does the previous python command still not show the <configuration> section as it should if run on the right server...

[dshatto]

I just confirmed that the Gitlab remote repository is getting commits, or at least it got some last week.

When I run the last python script from the bin directory on the deployment server I now get this error:

InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings