Hello, we've encountered a problem with the TA-crowdstrike-falcon-event-streams TA, which was functional in the past. Splunk Enterprise onPrem VERSION=9.1.2 BUILD=b6b9c8185839 PRODUCT=splunk PLATFORM=Linux-x86_64 When opening the UI to configure the crowdstrike Auth we'll end up with Err 500. Same for the other views. I've tried to reinstall it, but it didn't change anything. Splunkd logs the following:     01-26-2024 16:13:29.817 +0100 ERROR AdminManagerExternal [3102377 TcpChannelThread] - Unexpected error "<class 'splunktaucclib.rest_handler.error.RestError'>" from python handler: "REST Error [500]: Internal Server Error -- Traceback (most recent call last):\n File "/opt/splunk/lib/python3.7/site-packages/urllib3/connectionpool.py", line 706, in urlopen\n chunked=chunked,\n File "/opt/splunk/lib/python3.7/site-packages/urllib3/connectionpool.py", line 382, in _make_request\n self._validate_conn(conn)\n File "/opt/splunk/lib/python3.7/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn\n conn.connect()\n File "/opt/splunk/lib/python3.7/site-packages/urllib3/connection.py", line 421, in connect\n tls_in_tls=tls_in_tls,\n File "/opt/splunk/lib/python3.7/site-packages/urllib3/util/ssl_.py", line 453, in ssl_wrap_socket\n ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls)\n File "/opt/splunk/lib/python3.7/site-packages/urllib3/util/ssl_.py", line 495, in _ssl_wrap_socket_impl\n return ssl_context.wrap_socket(sock)\n File "/opt/splunk/lib/python3.7/ssl.py", line 428, in wrap_socket\n session=session\n File "/opt/splunk/lib/python3.7/ssl.py", line 878, in _create\n self.do_handshake()\n File "/opt/splunk/lib/python3.7/ssl.py", line 1147, in do_handshake\n self._sslobj.do_handshake()\nssl.SSLError: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:1106)\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/opt/splunk/lib/python3.7/site-packages/requests/adapters.py", line 449, in send\n timeout=timeout\n File "/opt/splunk/lib/python3.7/site-packages/urllib3/connectionpool.py", line 756, in urlopen\n method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]\n File "/opt/splunk/lib/python3.7/site-packages/urllib3/util/retry.py", line 574, in increment\n raise MaxRetryError(_pool, url, error or ResponseError(cause))\nurllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='127.0.0.1', port=8089): Max retries exceeded with url: /servicesNS/nobody/TA-crowdstrike-falcon-event-streams/configs/conf-ta_crowdstrike_falcon_event_streams_settings/_reload (Caused by SSLError(SSLError(1, '[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:1106)')))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/opt/splunk/etc/apps/TA-crowdstrike-falcon-event-streams/lib/splunktaucclib/rest_handler/handler.py", line 124, in wrapper\n for name, data, acl in meth(self, *args, **kwargs):\n File "/opt/splunk/etc/apps/TA-crowdstrike-falcon-event-streams/lib/splunktaucclib/rest_handler/handler.py", line 162, in get\n self.reload()\n File "/opt/splunk/etc/apps/TA-crowdstrike-falcon-event-streams/lib/splunktaucclib/rest_handler/handler.py", line 259, in reload\n action="_reload",\n File "/opt/splunk/etc/apps/TA-crowdstrike-falcon-event-streams/lib/splunklib/binding.py", line 320, in wrapper\n return request_fun(self, *args, **kwargs)\n File "/opt/splunk/etc/apps/TA-crowdstrike-falcon-event-streams/lib/splunklib/binding.py", line 79, in new_f\n val = f(*args, **kwargs)\n File "/opt/splunk/etc/apps/TA-crowdstrike-falcon-event-streams/lib/splunklib/binding.py", line 727, in get\n response = self.http.get(path, all_headers, **query)\n File "/opt/splunk/etc/apps/TA-crowdstrike-falcon-event-streams/lib/splunklib/binding.py", line 1254, in get\n return self.request(url, { 'method': "GET", 'headers': headers })\n File "/opt/splunk/etc/apps/TA-crowdstrike-falcon-event-streams/lib/splunklib/binding.py", line 1316, in request\n response = self.handler(url, message, **kwargs)\n File "/opt/splunk/etc/apps/TA-crowdstrike-falcon-event-streams/lib/solnlib/splunk_rest_client.py", line 147, in request\n **kwargs,\n File "/opt/splunk/lib/python3.7/site-packages/requests/api.py", line 61, in request\n return session.request(method=method, url=url, **kwargs)\n File "/opt/splunk/lib/python3.7/site-packages/requests/sessions.py", line 542, in request\n resp = self.send(prep, **send_kwargs)\n File "/opt/splunk/lib/python3.7/site-packages/requests/sessions.py", line 655, in send\n r = adapter.send(request, **kwargs)\n File "/opt/splunk/lib/python3.7/site-packages/requests/adapters.py", line 514, in send\n raise SSLError(e, request=request)\nrequests.exceptions.SSLError: HTTPSConnectionPool(host='127.0.0.1', port=8089): Max retries exceeded with url: /servicesNS/nobody/TA-crowdstrike-falcon-event-streams/configs/conf-ta_crowdstrike_falcon_event_streams_settings/_reload (Caused by SSLError(SSLError(1, '[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:1106)')))\n". See splunkd.log/python.log for more details.     inputs.conf   [splunktcp-ssl:8089] disabled = 0 requireClientCert = false sslVersions = * [...] [SSL] serverCert = <path> requireClientCert = true allowSslRenegotiation = true sslCommonNameToCheck = <others> 127.0.0.1,SplunkServerDefaultCert   server.conf   [sslConfig] enableSplunkdSSL = true sslVersions = tls1.2 serverCert = /opt/splunk/etc/auth/<path>.pem sslRootCAPath = /opt/splunk/etc/auth/<path>.pem requireClientCert = true sslVerifyServerName = true sslVerifyServerCert = true sslCommonNameToCheck = <FQDNs> cliVerifyServerName = false sslPassword = <pw>     We're looking forward for your help! Thank you! ... View more

We're updating our Linux Servers to Debian 12. A few host went "missing" afterwards in Splunk. While investigating into it I found out that they were in fact not missing, but they stopped writing logfiles to /var/log. Seems like Debian switched to full journald, as I was promoted with this ReadMe in /var/log: You are looking for the traditional text log files in /var/log, and they are gone? Here's an explanation on what's going on: You are running a systemd-based OS where traditional syslog has been replaced with the Journal. The journal stores the same (and more) information as classic syslog. To make use of the journal and access the collected log data simply invoke "journalctl", which will output the logs in the identical text-based format the syslog files in /var/log used to be. For further details, please refer to journalctl(1). [...]  Of course we can simply install the rsyslog package again, but this is feels more like a step backwards. So here is my question: Is there a default and generic approach for collecting all system and service logs from journald that we can use on our UFs, since Logfiles are obviously not the future on Linux. Best regards ... View more

Hello, We're running localhost Http Event Collectors on UF for Docker Containers on the same host. However I'm unable to override the hostname from these Events. Unfortunately there is no flag to do so in the docker daemon. Therefor I've tried to do it on the Forwarder as well as on the indexer. Both unsuccessful. On the Forwarder: /opt/splunkforwarder/etc/apps/splunk_httpinput/default/inputs.conf   [http] host = wantedHostName disabled=0 port = 8088 enableSSL=0 dedicatedIoThreads=2 maxThreads = 0 maxSockets = 0 useDeploymentServer=0   /opt/splunkforwarder/etc/apps/splunk_httpinput/local/inputs.conf   [http://localhost] host = wantedHostName description = <desc> disabled = 0 index = main token = <token> useACK = false   On the indexer: /opt/splunk/etc/system/local/props.conf   [host::badHostName] TRANSFORMS-badhost = badHostName   /opt/splunk/etc/system/local/transforms.conf   [badHostName] DEST_KEY = MetaData:Host REGEX = * FORMAT = host::wantedHostName     None of these work. Can someone please help us out? Cheers!   ... View more