I ended up changing domains back to my original domain where I could see Splunk in Programs and Features. Before I uninstalled it, I backed up etc, var/lib, and bin/script. However in Windows, some folders are locked. Be sure to check the size of the backup copies with the originals. I had to manually find which folders were missing files. Once the folders were backed up, I uninstalled Splunk, and changed domains back.
Before I installed Splunk on D drive I gave Domain Admins (a small group) full control of the entire drive. I setup a Splunk account in AD with membership in Domain Admins. I wanted to avoid the issue of only the installer being able to see the program and being able to update it. I used the Splunk account to log in to the server, install Splunk and restore the backup files.
Like before with the copy, folders are locked. The Windows copy process took 3 times to get most of the data. I had to manually find which folders were missing files and add files back. I started up Splunk and logged in and success…almost. I could see logs coming in from the agents, search, verified my licenses and apps. Everything checked out, but I had two errors:
Can not enable audit.db. Save checkpointstr: unable to open checkpointfile=’…\wineventlog\application’ for write: Access Denied
Received event for unconfigured/disabled/delted index=’ _audit’ with source=’source::audittrail’ host=host::hostname sourcetype=’sourcetype::audittrail’ (1 missing)
There were two indexes that were disabled. One enabled fine, the other _audit did not.
Looking at the splunkd.log, I noticed errors dealing with folder discrepancies hot_v1_0 and hot_v1_##. I looked in my backup folder var\lib\Splunk\audit\db and only saw hot_v1_##. In the same folder on D drive there were two folders. I copied the hot_v1_0 for backup and then deleted it. Error 1 cleared but error 2 was still present. I restarted Splunk (from within Splunk) and both errors were cleared. Splunk is back and fully restored.
Also, Splunk now shows up Program and Features for all Domain Admins. I am guessing it because I gave Domain Admins full access prior doing the install. But I am not sure.
... View more