docker container for splunk exits with below error on restart (it runs fine as long as I keep it up)

jagdish0886 — Thu, 11 Jun 2020 00:34:26 GMT

docker container for Splunk exits with below error on restart (it runs fine as long as I keep it up).
I was designing Splunk indexer cluster with one master and 2 indexer containers, master container is starting fine, however 2 indexer nodes are failing, while restarting with below error. all three containers accept traffic from outside on different ports 8000, 8001, 8002

TASK [splunk_common : Start Splunk via cli] ************************************
 fatal: [localhost]: FAILED! => {"changed": false, "cmd": ["/opt/splunk/bin/splunk", "start", "--accept-license", "--answer-yes", "--no-prompt"], "delta": "0:05:20.859094
 ", "end": "2020-04-18 09:15:03.654801", "msg": "non-zero return code", "rc": 1, "start": "2020-04-18 09:09:42.795707", "stderr": "\n\nBypassing local license checks since this
  instance is configured with a remote license master.", "stderr_lines": ["", "", "Bypassing local license checks since this instance is configured with a remote license master
 ."], "stdout": "splunkd 268 was not running.\nStopping splunk helpers...\n\nDone.\nStopped helpers.\nRemoving stale pid file... done.\n\nSplunk> Winning the War on Error\n\nCh
 ecking prerequisites...\n\tChecking http port [8000]: open\n\tChecking mgmt port [8089]: open\n\tChecking appserver port [127.0.0.1:8065]: open\n\tChecking kvstore port [8191]
 : open\n\tChecking configuration... Done.\n\tChecking critical directories...\tDone\n\tChecking indexes...\n\t\tValidated: _audit _internal _introspection _telemetry _thefishb
 ucket history main summary\n\tDone\n\tChecking filesystem compatibility...  Done\n\tChecking conf files for problems...\n\tDone\n\tChecking default conf files for edits...\n\t
 Validating installed files against hashes from '/opt/splunk/splunk-7.3.0-657388c7a488-linux-2.6-x86_64-manifest'\n\tAll installed files intact.\n\tDone\n\tChecking replication
 _port port [8050]: open\nAll preliminary checks passed.\n\nStarting splunk server daemon (splunkd)...  \nDone\n\n\nWaiting for web server at http://127.0.0.1:8000 to be availa
 ble............................................................................................................................................................................
 ................................................................................................................................\n\nWARNING: web interface does not seem to be 
 available!", "stdout_lines": ["splunkd 268 was not running.", "Stopping splunk helpers...", "", "Done.", "Stopped helpers.", "Removing stale pid file... done.", "", "Splunk> W
 inning the War on Error", "", "Checking prerequisites...", "\tChecking http port [8000]: open", "\tChecking mgmt port [8089]: open", "\tChecking appserver port [127.0.0.1:8065
 ]: open", "\tChecking kvstore port [8191]: open", "\tChecking configuration... Done.", "\tChecking critical directories...\tDone", "\tChecking indexes...", "\t\tValidated: _au
 dit _internal _introspection _telemetry _thefishbucket history main summary", "\tDone", "\tChecking filesystem compatibility...  Done", "\tChecking conf files for problems..."
 , "\tDone", "\tChecking default conf files for edits...", "\tValidating installed files against hashes from '/opt/splunk/splunk-7.3.0-657388c7a488-linux-2.6-x86_64-manifest'",
  "\tAll installed files intact.", "\tDone", "\tChecking replication_port port [8050]: open", "All preliminary checks passed.", "", "Starting splunk server daemon (splunkd)... 
  ", "Done", "", "", "Waiting for web server at http://127.0.0.1:8000 to be available...........................................................................................
 ...............................................................................................................................................................................
 ..................................", "", "WARNING: web interface does not seem to be available!"]}
 PLAY RECAP *********************************************************************
 localhost                  : ok=18   changed=1    unreachable=0    failed=1    skipped=16   rescued=0    ignored=0

Re: docker container for splunk exits with below error on restart (it runs fine as long as I keep it up)

jagdish0886 — Sun, 19 Apr 2020 01:05:18 GMT

I could fix this by keeping wait time between each splunk, but I am not sure, why it needs to wait for another container to start before first starts

docker@splunk-demo-host:~$ cat start.sh

!/bin/bash

docker start 0f1f55c255ee
sleep 150
docker start 646fb5b9cecb
sleep 150
docker start bc4e86af1277
sleep 150
docker@splunk-demo-host:~$

Re: docker container for splunk exits with below error on restart (it runs fine as long as I keep it up)

MuS — Wed, 22 Apr 2020 03:49:02 GMT

You can check index=_internal sourcetype=splunkd for error messages for the instances.

cheers, MuS

topic Re: docker container for splunk exits with below error on restart (it runs fine as long as I keep it up) in Deployment Architecture

docker container for splunk exits with below error on restart (it runs fine as long as I keep it up)

Re: docker container for splunk exits with below error on restart (it runs fine as long as I keep it up)

!/bin/bash

Re: docker container for splunk exits with below error on restart (it runs fine as long as I keep it up)