Could I please get assistance on how to resolve this issue and get the AlgoSec App for Security Incident Analysis and Response (2.x) Splunk application working. No changes have been made to any application files The steps in the algosec installation documentation has been followed: Integrate ASMS with Splunk (algosec.com) The Splunk Version being used: Splunk Enterprise 9.2 (Trial License) When installing the application, this error is returned: 500 Internal Server Error  Error Details: index=_internal host="*********" source=*web_service.log log_level=ERROR requestid=6694b1a1307f3b003f6d50 2024-07-15 15:20:33,402 ERROR [6694b1a1307f3b003f6d50] error:338 - Traceback (most recent call last): File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cprequest.py", line 628, in respond self._do_respond(path_info) File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cprequest.py", line 687, in _do_respond response.body = self.handler() File "/opt/splunk/lib/python3.7/site-packages/cherrypy/lib/encoding.py", line 219, in __call__ self.body = self.oldhandler(*args, **kwargs) File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/htmlinjectiontoolfactory.py", line 75, in wrapper resp = handler(*args, **kwargs) File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__ return self.callable(*self.args, **self.kwargs) File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/routes.py", line 422, in default return route.target(self, **kw) File "</opt/splunk/lib/python3.7/site-packages/decorator.py:decorator-gen-500>", line 2, in listEntities File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 41, in rundecs return fn(*a, **kw) File "</opt/splunk/lib/python3.7/site-packages/decorator.py:decorator-gen-498>", line 2, in listEntities File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 119, in check return fn(self, *a, **kw) File "</opt/splunk/lib/python3.7/site-packages/decorator.py:decorator-gen-497>", line 2, in listEntities File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 167, in validate_ip return fn(self, *a, **kw) File "</opt/splunk/lib/python3.7/site-packages/decorator.py:decorator-gen-496>", line 2, in listEntities File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 246, in preform_sso_check return fn(self, *a, **kw) File "</opt/splunk/lib/python3.7/site-packages/decorator.py:decorator-gen-495>", line 2, in listEntities File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 285, in check_login return fn(self, *a, **kw) File "</opt/splunk/lib/python3.7/site-packages/decorator.py:decorator-gen-494>", line 2, in listEntities File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 305, in handle_exceptions return fn(self, *a, **kw) File "</opt/splunk/lib/python3.7/site-packages/decorator.py:decorator-gen-489>", line 2, in listEntities File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 360, in apply_cache_headers response = fn(self, *a, **kw) File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/controllers/admin.py", line 1798, in listEntities app_name = eai_acl.get('app') AttributeError: 'NoneType' object has no attribute 'get'   Thanks Splunk Community ... View more

Check Point Skyline - Splunk Configuration Issue: Unable to get Data In  Issue Summary: Splunk Enterprise Indexer will not accept HTTP Event Collector HEC_Token from Check Point Gateway resulting in no Skyline (Open Telemetry) data being ingested into Splunk.  I need help to get splunk indexer to recognise the token and allow data to be ingested. Please note this error was also replicated on different Splunk Instance to determine potential root cause. Could potentially be attributed to the payload-no-tls.json file not being formatted or compiled correctly on the Gateway. Documentation used to configure set up: Check Point Skyline Deployment: https://support.checkpoint.com/results/sk/sk178566 Official Check Point Skyline Guide PDF: https://sc1.checkpoint.com/documents/Appliances/Skyline/CP_Skyline_AdminGuide.pdf Skyline Troubleshooting and FAQ: https://support.checkpoint.com/results/sk/sk179870 HTTP Event Collector in Splunk: https://docs.splunk.com/Documentation/SplunkCloud/latest/Data/UsetheHTTPEventCollector Environment Details: Splunk Version: Splunk Enterprise 9.2 (Trial License) Operating System: Ubuntu 22.04 Gateways (Both Virtual running on : CheckPoint_FW4 and CheckPoint_FW3 [Cluster2] Firewall Rules: Cleanup Rule to allow any communication for testing purposes.   Potential Root Cause - Log Analysis: Ran Command: tail -20 /opt/CPotelcol/otelcol.log                                 on CheckPoint_FW4 Response: go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/internal/bounded_memory_queue.go:47 2024-06-26T14:20:34.609+1000    error   exporterhelper/queued_retry.go:391      Exporting failed. The error is not retryable. Dropping data.    {"kind": "exporter", "data_type": "metrics", "name": "prometheusremotewrite", "error": "Permanent error: Permanent error: remote write returned HTTP status 401 Unauthorized; err = %!w(<nil>): Bearer token not recognized. Please contact your Splunk admin.\n", "dropped_items": 284} go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send         go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/queued_retry.go:391 go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send         go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/metrics.go:125 go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1         go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/queued_retry.go:195 go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func1   Completed Installation Steps: (Text highlighted in Green completed) Installed the Third-Party Monitoring Tool Installed the OpenTelemetry Agent and OpenTelemetry Collector on the Check Point Server Configured the OpenTelemetry Collector on the Check Point Server to work with the Third-Party Monitoring Tool: Splunk   Configure HTTP Event Collector on Splunk Enterprise Enable HTTP Event Collector on Splunk Enterprise Before you can use Event Collector to receive events through HTTP, you must enable it. For Splunk Enterprise, enable HEC through the Global Settings dialog box. Click Settings > Data Inputs. Click HTTP Event Collector. Click Global Settings. In the All Tokens toggle button, select Enabled. (Optional) Choose a Default Source Type for all HEC tokens. You can also type in the name of the source type in the text field above the drop-down list box before choosing the source type. (Optional) Choose a Default Index for all HEC tokens. (Optional) Choose a Default Output Group for all HEC tokens. (Optional) To use a deployment server to handle configurations for HEC tokens, click the Use Deployment Server check box. (Optional) To have HEC listen and communicate over HTTPS rather than HTTP, click the Enable SSL checkbox. (Optional) Enter a number in the HTTP Port Number field for HEC to listen on.     Create an Event Collector token on Splunk Enterprise To use HEC, you must configure at least one token. Click Settings > Add Data. Click monitor. Click HTTP Event Collector. In the Name field, enter a name for the token. (Optional) In the Source name override field, enter a source name for events that this input generates. (Optional) In the Description field, enter a description for the input. (Optional) In the Output Group field, select an existing forwarder output group. (Optional) If you want to enable indexer acknowledgment for this token, click the Enable indexer acknowledgment checkbox. Click Next. (Optional) Confirm the source type and the index for HEC events.     Click Review. Confirm that all settings for the endpoint are what you want. If all settings are what you want, click Submit. Otherwise, click < to make changes. (Optional) Copy the token value that Splunk Web displays and paste it into another document for reference later   Confirmed the Token is Status: Enabled Configured payload-no-tls.json in /home/admin/payload-no-tls.json   Step: Run the configuration command to apply the payload - either the CLI command, or the Gaia REST API command: n Method 1 - Run the CLI command "sklnctl": a. Save the JSON payload in a file (for example, /home/admin/payload.json). b. Run this command: sklnctl export --set "$(cat /home/admin/payload.json)" Repeated steps for FW4 Rebooted Gateway FW3 and FW4 Rebooted Splunk Server Restarted all Check Point Firewall Skyline Components Result: Data Failed to be ingested Other troubleshooting completed: Created completely new token and repeated configuration steps Updated the url within the payload.json file to end with /services/collector/raw /services/collector/events Updated “url”: http://10... Instead of https Checked the Skyline Component Log Files for Troubleshooting: What are the relevant Check Point Skyline log files? OpenTelemetry Collector: /opt/CPotelcol/otelcol.log CPView Exporter: /opt/CPviewExporter/otlp_cpview.log CPView API Service: $CPDIR/log/cpview_api_service.elg   Logs CPView API Service and CPView displayed no logs indicating causes of the issues. Confirmed that the bearer token works:   Result: Bearer Token accepted. Confirmed Collector was healthy: Alternative payload-no-tls.json formats attempted:       Gateway Log Analysis (Returned everytime:) SSH into CheckPoint_FW4 xx.xx.xx.xx via Remote Desktop Ran Command: tail /opt/CPotelcol/otelcol.log Result: go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/internal/bounded_memory_queue.go:47 2024-06-26T14:20:34.609+1000    error   exporterhelper/queued_retry.go:391      Exporting failed. The error is not retryable. Dropping data.    {"kind": "exporter", "data_type": "metrics", "name": "prometheusremotewrite", "error": "Permanent error: Permanent error: remote write returned HTTP status 401 Unauthorized; err = %!w(<nil>): Bearer token not recognized. Please contact your Splunk admin.\n", "dropped_items": 284} go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send         go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/queued_retry.go:391 go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send         go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/metrics.go:125 go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1         go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/queued_retry.go:195 go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func1   Finding: Appears to be an issue in which the HTTP Event Collector will not accept the Token Value, even when the token matches identically. Could potentially be attributed to the payload-no-tls.json file not being formatted or compiled correctly on the Gateway.   ... View more

Hello Splunk Community,  I'm encountering an issue with ingesting data from a Prometheus remote_write_agent into Splunk Enterprise – this solution utilises the ‘Prometheus Metrics for Splunk and is within a Test Environment. Problem Summary: Despite ensuring that the 'inputs.conf' file matches the configuration specifications defined in the 'inputs.conf.spec' file, the Prometheus data is not being ingested and I am receiving errors, e.g port: Not found in "btool" output (btool does not list the stanza [prometheusrw]) when viewing the inputs.conf file in the config explorer application. Details: Splunk Version: Splunk Enterprise 9.2 (Trial License) Operating System: Ubuntu 22.04 Splunk Application: Prometheus Metrics for Splunk (Latest Version 1.0.1)   inputs.conf.spec  /opt/splunk/etc/apps/modinput_prometheus/README/inputs.conf.spec (Full inputs.conf.spec - https://github.com/lukemonahan/splunk_modinput_prometheus/blob/master/modinput_prometheus/README/inputs.conf.spec As seen in image, the inputs.conf.spec file states there is a port  and maxClients configuration parameters. In the inputs.conf I updated the  /opt/splunk/etc/apps/modinput_prometheus/local/inputs.conf file to include the details below which meet the required formatting above: The inputs.conf file was saved, and the Splunk Server rebooted. After rebooting the input.conf was checked to ensure the config specification where being accepted using the Config Explorer App –  These errors where received for the following configuration parameters: However, other configuration parameters such as index, sourcetype & whitelist  Returned: 'Found in "btool" output. Exists in spec file (Stanza=[prometheusrw]) - and were accepted by Splunk. For some unknown reason, Splunk is not recognising some of the configuration parameters above that are listed within the inputs.conf.spec file, even when formatted accordingly.   Other Information: Prometheus remote-write-exporter details: Splunk Index: skyline_prometheus_metrics Any assistance is appreciated, thank you Splunk Community 🙂           ... View more