Deployment Architecture

docker container for splunk exits with below error on restart (it runs fine as long as I keep it up)

jagdish0886
Explorer

docker container for Splunk exits with below error on restart (it runs fine as long as I keep it up).
I was designing Splunk indexer cluster with one master and 2 indexer containers, master container is starting fine, however 2 indexer nodes are failing, while restarting with below error. all three containers accept traffic from outside on different ports 8000, 8001, 8002

TASK [splunk_common : Start Splunk via cli] ************************************
 fatal: [localhost]: FAILED! => {"changed": false, "cmd": ["/opt/splunk/bin/splunk", "start", "--accept-license", "--answer-yes", "--no-prompt"], "delta": "0:05:20.859094
 ", "end": "2020-04-18 09:15:03.654801", "msg": "non-zero return code", "rc": 1, "start": "2020-04-18 09:09:42.795707", "stderr": "\n\nBypassing local license checks since this
  instance is configured with a remote license master.", "stderr_lines": ["", "", "Bypassing local license checks since this instance is configured with a remote license master
 ."], "stdout": "splunkd 268 was not running.\nStopping splunk helpers...\n\nDone.\nStopped helpers.\nRemoving stale pid file... done.\n\nSplunk> Winning the War on Error\n\nCh
 ecking prerequisites...\n\tChecking http port [8000]: open\n\tChecking mgmt port [8089]: open\n\tChecking appserver port [127.0.0.1:8065]: open\n\tChecking kvstore port [8191]
 : open\n\tChecking configuration... Done.\n\tChecking critical directories...\tDone\n\tChecking indexes...\n\t\tValidated: _audit _internal _introspection _telemetry _thefishb
 ucket history main summary\n\tDone\n\tChecking filesystem compatibility...  Done\n\tChecking conf files for problems...\n\tDone\n\tChecking default conf files for edits...\n\t
 Validating installed files against hashes from '/opt/splunk/splunk-7.3.0-657388c7a488-linux-2.6-x86_64-manifest'\n\tAll installed files intact.\n\tDone\n\tChecking replication
 _port port [8050]: open\nAll preliminary checks passed.\n\nStarting splunk server daemon (splunkd)...  \nDone\n\n\nWaiting for web server at http://127.0.0.1:8000 to be availa
 ble............................................................................................................................................................................
 ................................................................................................................................\n\nWARNING: web interface does not seem to be 
 available!", "stdout_lines": ["splunkd 268 was not running.", "Stopping splunk helpers...", "", "Done.", "Stopped helpers.", "Removing stale pid file... done.", "", "Splunk> W
 inning the War on Error", "", "Checking prerequisites...", "\tChecking http port [8000]: open", "\tChecking mgmt port [8089]: open", "\tChecking appserver port [127.0.0.1:8065
 ]: open", "\tChecking kvstore port [8191]: open", "\tChecking configuration... Done.", "\tChecking critical directories...\tDone", "\tChecking indexes...", "\t\tValidated: _au
 dit _internal _introspection _telemetry _thefishbucket history main summary", "\tDone", "\tChecking filesystem compatibility...  Done", "\tChecking conf files for problems..."
 , "\tDone", "\tChecking default conf files for edits...", "\tValidating installed files against hashes from '/opt/splunk/splunk-7.3.0-657388c7a488-linux-2.6-x86_64-manifest'",
  "\tAll installed files intact.", "\tDone", "\tChecking replication_port port [8050]: open", "All preliminary checks passed.", "", "Starting splunk server daemon (splunkd)... 
  ", "Done", "", "", "Waiting for web server at http://127.0.0.1:8000 to be available...........................................................................................
 ...............................................................................................................................................................................
 ..................................", "", "WARNING: web interface does not seem to be available!"]}
 PLAY RECAP *********************************************************************
 localhost                  : ok=18   changed=1    unreachable=0    failed=1    skipped=16   rescued=0    ignored=0
Labels (2)
0 Karma
1 Solution

jagdish0886
Explorer

I could fix this by keeping wait time between each splunk, but I am not sure, why it needs to wait for another container to start before first starts

docker@splunk-demo-host:~$ cat start.sh

!/bin/bash

docker start 0f1f55c255ee
sleep 150
docker start 646fb5b9cecb
sleep 150
docker start bc4e86af1277
sleep 150
docker@splunk-demo-host:~$

View solution in original post

0 Karma

jagdish0886
Explorer

I could fix this by keeping wait time between each splunk, but I am not sure, why it needs to wait for another container to start before first starts

docker@splunk-demo-host:~$ cat start.sh

!/bin/bash

docker start 0f1f55c255ee
sleep 150
docker start 646fb5b9cecb
sleep 150
docker start bc4e86af1277
sleep 150
docker@splunk-demo-host:~$

0 Karma

MuS
SplunkTrust
SplunkTrust

You can check index=_internal sourcetype=splunkd for error messages for the instances.

cheers, MuS

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...