docker container for Splunk exits with below error on restart (it runs fine as long as I keep it up). I was designing Splunk indexer cluster with one master and 2 indexer containers, master container is starting fine, however 2 indexer nodes are failing, while restarting with below error. all three containers accept traffic from outside on different ports 8000, 8001, 8002
TASK [splunk_common : Start Splunk via cli] ************************************
fatal: [localhost]: FAILED! => {"changed": false, "cmd": ["/opt/splunk/bin/splunk", "start", "--accept-license", "--answer-yes", "--no-prompt"], "delta": "0:05:20.859094
", "end": "2020-04-18 09:15:03.654801", "msg": "non-zero return code", "rc": 1, "start": "2020-04-18 09:09:42.795707", "stderr": "\n\nBypassing local license checks since this
instance is configured with a remote license master.", "stderr_lines": ["", "", "Bypassing local license checks since this instance is configured with a remote license master
."], "stdout": "splunkd 268 was not running.\nStopping splunk helpers...\n\nDone.\nStopped helpers.\nRemoving stale pid file... done.\n\nSplunk> Winning the War on Error\n\nCh
ecking prerequisites...\n\tChecking http port [8000]: open\n\tChecking mgmt port [8089]: open\n\tChecking appserver port [127.0.0.1:8065]: open\n\tChecking kvstore port [8191]
: open\n\tChecking configuration... Done.\n\tChecking critical directories...\tDone\n\tChecking indexes...\n\t\tValidated: _audit _internal _introspection _telemetry _thefishb
ucket history main summary\n\tDone\n\tChecking filesystem compatibility... Done\n\tChecking conf files for problems...\n\tDone\n\tChecking default conf files for edits...\n\t
Validating installed files against hashes from '/opt/splunk/splunk-7.3.0-657388c7a488-linux-2.6-x86_64-manifest'\n\tAll installed files intact.\n\tDone\n\tChecking replication
_port port [8050]: open\nAll preliminary checks passed.\n\nStarting splunk server daemon (splunkd)... \nDone\n\n\nWaiting for web server at http://127.0.0.1:8000 to be availa
ble............................................................................................................................................................................
................................................................................................................................\n\nWARNING: web interface does not seem to be
available!", "stdout_lines": ["splunkd 268 was not running.", "Stopping splunk helpers...", "", "Done.", "Stopped helpers.", "Removing stale pid file... done.", "", "Splunk> W
inning the War on Error", "", "Checking prerequisites...", "\tChecking http port [8000]: open", "\tChecking mgmt port [8089]: open", "\tChecking appserver port [127.0.0.1:8065
]: open", "\tChecking kvstore port [8191]: open", "\tChecking configuration... Done.", "\tChecking critical directories...\tDone", "\tChecking indexes...", "\t\tValidated: _au
dit _internal _introspection _telemetry _thefishbucket history main summary", "\tDone", "\tChecking filesystem compatibility... Done", "\tChecking conf files for problems..."
, "\tDone", "\tChecking default conf files for edits...", "\tValidating installed files against hashes from '/opt/splunk/splunk-7.3.0-657388c7a488-linux-2.6-x86_64-manifest'",
"\tAll installed files intact.", "\tDone", "\tChecking replication_port port [8050]: open", "All preliminary checks passed.", "", "Starting splunk server daemon (splunkd)...
", "Done", "", "", "Waiting for web server at http://127.0.0.1:8000 to be available...........................................................................................
...............................................................................................................................................................................
..................................", "", "WARNING: web interface does not seem to be available!"]}
PLAY RECAP *********************************************************************
localhost : ok=18 changed=1 unreachable=0 failed=1 skipped=16 rescued=0 ignored=0
... View more