Error while installing Splunk universal forwarder: Migration information is being logged

[Poojitha]

Hi Team,

I am installing splunk universal forwarder using ansible : When I am trying to start splunk and accept license, I am getting below error :

```
fatal: [Server-a]: FAILED! => {"changed": true, "cmd": ["/opt/splunkforwarder/bin/splunk", "start", "--accept-license", "--answer-yes", "--no-prompt"], "delta": "0:00:00.130544", "end": "2022-04-16 17:17:20.807732", "msg": "non-zero return code", "rc": 1, "start": "2022-04-16 17:17:20.677188", "stderr": "\n-- Migration information is being logged to '/opt/splunkforwarder/var/log/splunk/migration.log.2022-04-16.17-17-20' --\nERROR while running renew-certs migration.", "stderr_lines": ["", "-- Migration information is being logged to '/opt/splunkforwarder/var/log/splunk/migration.log.2022-04-16.17-17-20' --", "ERROR while running renew-certs migration."], "stdout": "\nThis appears to be an upgrade of Splunk.\n--------------------------------------------------------------------------------)\n\nSplunk has detected an older version of Splunk installed on this machine. To\nfinish upgrading to the new version, Splunk's installer will automatically\nupdate and alter your current configuration files. Deprecated configuration\nfiles will be renamed with a .deprecated extension.\n\nYou can choose to preview the changes that will be made to your configuration\nfiles before proceeding with the migration and upgrade:\n\nIf you want to migrate and upgrade without previewing the changes that will be\nmade to your existing configuration files, choose 'y'.\nIf you want to see what changes will be made before you proceed with the\nupgrade, choose 'n'.\n\n\nPerform migration and upgrade without previewing configuration changes? [y/n] y\n\nMigrating to:\nVERSION=8.2.4\nBUILD=87e2dda940d1\nPRODUCT=splunk\nPLATFORM=Linux-x86_64\n\n\nERROR: In order to migrate, Splunkd must not be running.", "stdout_lines": ["", "This appears to be an upgrade of Splunk.", "--------------------------------------------------------------------------------)", "", "Splunk has detected an older version of Splunk installed on this machine. To", "finish upgrading to the new version, Splunk's installer will automatically", "update and alter your current configuration files. Deprecated configuration", "files will be renamed with a .deprecated extension.", "", "You can choose to preview the changes that will be made to your configuration", "files before proceeding with the migration and upgrade:", "", "If you want to migrate and upgrade without previewing the changes that will be", "made to your existing configuration files, choose 'y'.", "If you want to see what changes will be made before you proceed with the", "upgrade, choose 'n'.", "", "", "Perform migration and upgrade without previewing configuration changes? [y/n] y", "", "Migrating to:", "VERSION=8.2.4", "BUILD=87e2dda940d1", "PRODUCT=splunk", "PLATFORM=Linux-x86_64", "", "", "ERROR: In order to migrate, Splunkd must not be running."]}
```

This error not happens everytime. First time when I run the script it doesnot throw this error and runs successfully. If I run second time on same host, it shows this error . Can someone help me to understand this please ???
```
command I am using : /opt/splunkforwarder/bin/splunk start --accept-license --answer-yes

Thanks in Advance,
Poojitha 

[eswara9]

-> stop the Splunk and upgrade again 
-> any server certificate expiry also gives this error

[tscroggins]

@Poojitha 

If you're installing the RPM, the preinstall scriptlet does this:

if [ -x "$SPLUNK_HOME/bin/splunk" ] ; then
    echo "This looks like an upgrade of an existing Splunk Server. Attempting to stop the installed Splunk Server..."
    $SPLUNK_HOME/bin/splunk stop
fi

The installer doesn't confirm whether the stop was successful.

Does your Ansible playbook include a step to stop Splunk before you attempt to upgrade in place? After stopping Splunk, you should confirm 1) Splunk stopped successfully and 2) $SPLUNK_HOME/var/run/splunk/splunkd.pid was removed. You can both confirm Splunk is stopped and automatically remove the .pid file if present by running '$SPLUNK_HOME/bin/splunk status'.