Deployment Architecture

docker container for splunk exits with below error on restart (it runs fine as long as I keep it up)

jagdish0886
Explorer

docker container for Splunk exits with below error on restart (it runs fine as long as I keep it up).
I was designing Splunk indexer cluster with one master and 2 indexer containers, master container is starting fine, however 2 indexer nodes are failing, while restarting with below error. all three containers accept traffic from outside on different ports 8000, 8001, 8002

TASK [splunk_common : Start Splunk via cli] ************************************
 fatal: [localhost]: FAILED! => {"changed": false, "cmd": ["/opt/splunk/bin/splunk", "start", "--accept-license", "--answer-yes", "--no-prompt"], "delta": "0:05:20.859094
 ", "end": "2020-04-18 09:15:03.654801", "msg": "non-zero return code", "rc": 1, "start": "2020-04-18 09:09:42.795707", "stderr": "\n\nBypassing local license checks since this
  instance is configured with a remote license master.", "stderr_lines": ["", "", "Bypassing local license checks since this instance is configured with a remote license master
 ."], "stdout": "splunkd 268 was not running.\nStopping splunk helpers...\n\nDone.\nStopped helpers.\nRemoving stale pid file... done.\n\nSplunk> Winning the War on Error\n\nCh
 ecking prerequisites...\n\tChecking http port [8000]: open\n\tChecking mgmt port [8089]: open\n\tChecking appserver port [127.0.0.1:8065]: open\n\tChecking kvstore port [8191]
 : open\n\tChecking configuration... Done.\n\tChecking critical directories...\tDone\n\tChecking indexes...\n\t\tValidated: _audit _internal _introspection _telemetry _thefishb
 ucket history main summary\n\tDone\n\tChecking filesystem compatibility...  Done\n\tChecking conf files for problems...\n\tDone\n\tChecking default conf files for edits...\n\t
 Validating installed files against hashes from '/opt/splunk/splunk-7.3.0-657388c7a488-linux-2.6-x86_64-manifest'\n\tAll installed files intact.\n\tDone\n\tChecking replication
 _port port [8050]: open\nAll preliminary checks passed.\n\nStarting splunk server daemon (splunkd)...  \nDone\n\n\nWaiting for web server at http://127.0.0.1:8000 to be availa
 ble............................................................................................................................................................................
 ................................................................................................................................\n\nWARNING: web interface does not seem to be 
 available!", "stdout_lines": ["splunkd 268 was not running.", "Stopping splunk helpers...", "", "Done.", "Stopped helpers.", "Removing stale pid file... done.", "", "Splunk> W
 inning the War on Error", "", "Checking prerequisites...", "\tChecking http port [8000]: open", "\tChecking mgmt port [8089]: open", "\tChecking appserver port [127.0.0.1:8065
 ]: open", "\tChecking kvstore port [8191]: open", "\tChecking configuration... Done.", "\tChecking critical directories...\tDone", "\tChecking indexes...", "\t\tValidated: _au
 dit _internal _introspection _telemetry _thefishbucket history main summary", "\tDone", "\tChecking filesystem compatibility...  Done", "\tChecking conf files for problems..."
 , "\tDone", "\tChecking default conf files for edits...", "\tValidating installed files against hashes from '/opt/splunk/splunk-7.3.0-657388c7a488-linux-2.6-x86_64-manifest'",
  "\tAll installed files intact.", "\tDone", "\tChecking replication_port port [8050]: open", "All preliminary checks passed.", "", "Starting splunk server daemon (splunkd)... 
  ", "Done", "", "", "Waiting for web server at http://127.0.0.1:8000 to be available...........................................................................................
 ...............................................................................................................................................................................
 ..................................", "", "WARNING: web interface does not seem to be available!"]}
 PLAY RECAP *********************************************************************
 localhost                  : ok=18   changed=1    unreachable=0    failed=1    skipped=16   rescued=0    ignored=0
Labels (2)
0 Karma
1 Solution

jagdish0886
Explorer

I could fix this by keeping wait time between each splunk, but I am not sure, why it needs to wait for another container to start before first starts

docker@splunk-demo-host:~$ cat start.sh

!/bin/bash

docker start 0f1f55c255ee
sleep 150
docker start 646fb5b9cecb
sleep 150
docker start bc4e86af1277
sleep 150
docker@splunk-demo-host:~$

View solution in original post

0 Karma

jagdish0886
Explorer

I could fix this by keeping wait time between each splunk, but I am not sure, why it needs to wait for another container to start before first starts

docker@splunk-demo-host:~$ cat start.sh

!/bin/bash

docker start 0f1f55c255ee
sleep 150
docker start 646fb5b9cecb
sleep 150
docker start bc4e86af1277
sleep 150
docker@splunk-demo-host:~$

0 Karma

MuS
Legend

You can check index=_internal sourcetype=splunkd for error messages for the instances.

cheers, MuS

0 Karma
Get Updates on the Splunk Community!

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...