topic Re: Splunk Web on Docker fails to start using an M1 Mac on Monterey in Splunk Enterprise

Splunk Web on Docker fails to start using an M1 Mac on Monterey

mvanberg — Wed, 19 Jan 2022 20:28:00 GMT

I'm unable to get Splunk to run in docker using a newer MBP with an M1 Max chip on a fresh install of Monterey, as well as a fresh install on an M1 Mac mini. I've played with as many settings as I could think, but can't seem to find an error that indicates what's really going on. As far as I can tell, splunkd starts and binds to port 8089, but Splunk Web fails to bind to port 8000, despite the port being available.

Things I tried and some thoughts:

- My initial thought is that port 8000 was being used by something else, so I tried many other ports with no success. Though, I had no evidence of this (using netstat)

- I then thought that maybe there was a firewall entry not being added correctly so I checked iptables, it doesn't exist. I then checked firewalld, also doesn't appear to exist. so no firewall?

- I had a friend take my exact docker compose file and install the everything on an older, non-up-to-date MacBook Air running on an intel chip. That worked...

- I also tried adjusting the timeout values listed in the sensible vars list, that didn't seem to work. Where am I supposed to mount the docker.yaml file to? Where I mounted it didn't work. the var, SPLUNK_CONNECTION_TIMEOUT, added directly to the compose file didn't make a difference either

- I even tried starting Splunk with debug mode and saw nothing helpful there.

The actual output, noticing the time taken under Start Splunk via CLI and failed=1:

sh1  | PLAY RECAP *********************************************************************

sh1  | localhost                  : ok=51   changed=7    unreachable=0    failed=1    skipped=48   rescued=0    ignored=0   

sh1  | 

sh1  | Friday 14 January 2022  21:48:52 +0000 (0:04:22.382)       0:05:46.139 ******** 

sh1  | =============================================================================== 

sh1  | splunk_common : Start Splunk via CLI ---------------------------------- 262.38s

sh1  | splunk_common : Get Splunk status --------------------------------------- 8.02s

sh1  | splunk_common : Update Splunk directory owner --------------------------- 6.01s

sh1  | Gathering Facts --------------------------------------------------------- 5.73s

sh1  | splunk_common : Generate user-seed.conf (Linux) ------------------------- 4.70s

sh1  | splunk_common : Cleanup Splunk runtime files ---------------------------- 4.30s

sh1  | splunk_common : Update /opt/splunk/etc ---------------------------------- 3.87s

sh1  | splunk_common : Check for scloud ---------------------------------------- 3.00s

sh1  | splunk_common : Hash the password --------------------------------------- 2.77s

sh1  | splunk_common : Find manifests ------------------------------------------ 2.52s

sh1  | splunk_common : Remove input SSL settings ------------------------------- 2.22s

sh1  | splunk_common : Check for existing installation ------------------------- 2.21s

sh1  | splunk_common : Create .ui_login ---------------------------------------- 2.21s

sh1  | splunk_common : Check if /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key exists --- 2.19s

sh1  | splunk_common : Enable splunktcp input ---------------------------------- 2.18s

sh1  | splunk_common : Enable Splunkd SSL -------------------------------------- 2.18s

sh1  | splunk_common : Enable Web SSL ------------------------------------------ 2.18s

sh1  | splunk_common : Trigger restart ----------------------------------------- 2.17s

sh1  | splunk_common : Remove splunktcp-ssl input ------------------------------ 2.16s

sh1  | splunk_common : Set Splunkd Connection Timeout -------------------------- 2.16s

sh1 exited with code 2

Here's the docker-compose (worked on the older Mac):

version: "3.9"

services:
  sh1:
    platform: linux/amd64
    image: splunk/splunk:latest
    container_name: sh1
    environment:
      - SPLUNK_START_ARGS=--accept-license
      - SPLUNK_PASSWORD=Passw0rd!
      - SPLUNK_ROLE=splunk_search_head
      - SPLUNK_HTTP_PORT=8000
      - SPLUNK_CONNECTION_TIMEOUT=300
   ports:
     - 8000:8000
     - 8089:8089

Any thoughts? Can someone on OS12.1 with an M1 chip get this to work? Additionally, can someone running OS12.1 with an intel chip validate that this works? Maybe the issue is with the M1 chip, not the OS version. Or maybe it's just an issue with 12.1.

Edit:

I now have evidence that the the compose file I posted works on an intel based Mac running the 12.1. Therefore, I think it's safe to say the issue is one of compatibility between the Splunk-Docker image and the M1 Mac.

Re: Splunk Web on Docker fails to start using an M1 Mac on Monterey

PickleRick — Sat, 15 Jan 2022 07:06:20 GMT

As far as I can see, Splunk for Mac is at the moment only supported on Intel platorm.

Your docker compose also refers to intel architecture whereas M1 is an ARM-derivative.

Re: Splunk Web on Docker fails to start using an M1 Mac on Monterey

mvanberg — Tue, 18 Jan 2022 16:18:30 GMT

According to a few different posts I saw, Splunk on Docker does support the M1 (ARM) Mac. I can say for sure that if I install Splunk normally, i.e. without Docker, it works just fine. Here's some information on how to handle the intel architecture on an M1 directly from Docker: https://docs.docker.com/desktop/mac/apple-silicon/. It turns out that the image does not yet exist for ARM so you must call out the platform specifically as linux/amd64 or else Docker throws an error.

Re: Splunk Web on Docker fails to start using an M1 Mac on Monterey

PickleRick — Wed, 19 Jan 2022 07:54:11 GMT

That's interesting. Honestly.

As far as I can find, the M1 does not on its own offer any layer of compatibility with x86 or x86_64.

The Splunk docs https://docs.splunk.com/Documentation/Splunk/latest/Installation/Systemrequirements say that MacOS is only supported on Intel architecture.

There is only an Intel-compatible version released for Mac on Splunk website.

So how can you install Intel Splunk version on M1 Mac? Does the Mac provide some virtualization tool for it? If so, I suppose it's gonna be highly inefficient.And it will probably not be engaged if you're trying to run the x86_64 version of docker image. And there is no M1 image since there is no M1 version of Splunk released.

So far there is a Universal Forwarder package available for M1 Mac but that's a different story. Are you sure you're not referring to the UF working on your Mac?

Re: Splunk Web on Docker fails to start using an M1 Mac on Monterey

isoutamo — Wed, 19 Jan 2022 15:16:33 GMT

My experience with M1 with splunk and docker are from the beginning of last year. At that time I cannot run splunk on docker, but it runs on macOS on M1 platform. What I have understood is that without docker, splunk use Rosetta2 when it's running and for that reason it works even it has compiled to run on Intel cpus. But when you are using docker then it don' t use Rosetta2 as docker has a native port to M1. Basically also splunk should run on docker in M1 if you could found correct splunk's docker image (Not all images are available for ARM64 architecture. You can add --platform linux/amd64 to run an Intel image under emulation). Unfortunately I haven't found (at least that time) correct image for running it on M1 😞

r. Ismo

https://docs.docker.com/desktop/mac/apple-silicon/

Re: Splunk Web on Docker fails to start using an M1 Mac on Monterey

PickleRick — Wed, 19 Jan 2022 20:13:37 GMT

Ahhh. So there is an emulator involved after all.

My experience with Macs is limited to a single Mac Pro back 10+ years ago and trying (not very successfuly) to spin up a hackintosh VM once but if said Rosetta is a thingy that is relatively easy installable (and the link you provided suggests that indeed it is), you could try modifying the dockerfile for splunk and build your own customized image. It's relatively easy although repeated tries at building (you never do it right at first try ;-)) can be annoying.

Did some docker images myself few years ago and it's not that difficult. But if it's not something you do for a living, I can understand if it's this piece of knowledge you could wish to avoid 😄

Re: Splunk Web on Docker fails to start using an M1 Mac on Monterey

mvanberg — Wed, 19 Jan 2022 20:26:29 GMT

So I admit I haven't watched the linked video yet, and maybe I should, but the post I was referring to where they say Docker-Splunk is supported by the M1 architecture is here: https://community.splunk.com/t5/Splunk-Enterprise/Splunk-enterprise-on-the-new-Mac-m1-under-Rosetta/m-p/547135

While I can likely make this all work with a custom image, especially since a local install works just fine, the point of the original post was really to let Splunk know that it's not working and to get their eyes on this.

Re: Splunk Web on Docker fails to start using an M1 Mac on Monterey

PickleRick — Thu, 20 Jan 2022 01:17:07 GMT

To be fully honest, I don't get the idea of running splunk in docker in the first place.

I'm not a very big fan of docker in the first place but it makes some limited sense in cases where you indeed don't store state within the container and the configuration is relatively easy and straightforward.

In case of splunk components you either have to configure it from the ground each time you're starting the container or have to keep most of $SPLUNK_HOME on external storage anyway which completely defeats the purpose of containerizing anyway.

Re: Splunk Web on Docker fails to start using an M1 Mac on Monterey

isoutamo — Thu, 20 Jan 2022 06:20:21 GMT

One excellent use for docker is testing TAs and apps automatically. It’s quite easy set up even several different splunk version and run tests against those.

Re: Splunk Web on Docker fails to start using an M1 Mac on Monterey

diogofgm — Fri, 21 Jan 2022 12:09:50 GMT

As stated by others the docker images available from splunk currently do not target the ARM architecture. There is the rosetta2 option but I've just read about it and not tested it myself. But having Splunk built images would be a better solution.
I use splunk docker containers a lot during development of my TAs and Apps, although I'm still using a intel based Mac, I'm about to change that soon. So here is and idea I submitted earlier that you can upvote: https://ideas.splunk.com/ideas/EID-I-1054

Alternatively, you could build your own docker images. I do not own a M1 Mac yet but this is something I would consider if this was a pressing issue.