Help Needed: HTTP Event Collector Bearer Token not Recognized

[Network007]

Check Point Skyline - Splunk Configuration Issue: Unable to get Data In

 Issue Summary: Splunk Enterprise Indexer will not accept HTTP Event Collector HEC_Token from Check Point Gateway resulting in no Skyline (Open Telemetry) data being ingested into Splunk.  I need help to get splunk indexer to recognise the token and allow data to be ingested.

Please note this error was also replicated on different Splunk Instance to determine potential root cause. Could potentially be attributed to the payload-no-tls.json file not being formatted or compiled correctly on the Gateway.

Documentation used to configure set up:

Check Point Skyline Deployment: https://support.checkpoint.com/results/sk/sk178566

Official Check Point Skyline Guide PDF: https://sc1.checkpoint.com/documents/Appliances/Skyline/CP_Skyline_AdminGuide.pdf

Skyline Troubleshooting and FAQ: https://support.checkpoint.com/results/sk/sk179870

HTTP Event Collector in Splunk: https://docs.splunk.com/Documentation/SplunkCloud/latest/Data/UsetheHTTPEventCollector

Environment Details:
Splunk Version: Splunk Enterprise 9.2 (Trial License)
Operating System: Ubuntu 22.04

Gateways (Both Virtual running on : CheckPoint_FW4 and CheckPoint_FW3 [Cluster2]

Firewall Rules: Cleanup Rule to allow any communication for testing purposes.

Potential Root Cause - Log Analysis:
Ran Command: tail -20 /opt/CPotelcol/otelcol.log                                 on CheckPoint_FW4

Response:

go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/internal/bounded_memory_queue.go:47

2024-06-26T14:20:34.609+1000    error   exporterhelper/queued_retry.go:391      Exporting failed. The error is not retryable. Dropping data.    {"kind": "exporter", "data_type": "metrics", "name": "prometheusremotewrite", "error": "Permanent error: Permanent error: remote write returned HTTP status 401 Unauthorized; err = %!w(<nil>): Bearer token not recognized. Please contact your Splunk admin.\n", "dropped_items": 284}

go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send

        go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/queued_retry.go:391

go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send

        go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/metrics.go:125

go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1

        go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/queued_retry.go:195

go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func1

Completed Installation Steps:

(Text highlighted in Green completed)

• Installed the Third-Party Monitoring Tool
• Installed the OpenTelemetry Agent and OpenTelemetry Collector on the Check Point Server
• Configured the OpenTelemetry Collector on the Check Point Server to work with the Third-Party Monitoring Tool: Splunk

Configure HTTP Event Collector on Splunk Enterprise

Enable HTTP Event Collector on Splunk Enterprise

Before you can use Event Collector to receive events through HTTP, you must enable it. For Splunk Enterprise, enable HEC through the Global Settings dialog box.

1. Click Settings > Data Inputs.
2. Click HTTP Event Collector.
3. Click Global Settings.
4. In the All Tokens toggle button, select Enabled.
5. (Optional) Choose a Default Source Type for all HEC tokens. You can also type in the name of the source type in the text field above the drop-down list box before choosing the source type.
6. (Optional) Choose a Default Index for all HEC tokens.
7. (Optional) Choose a Default Output Group for all HEC tokens.
8. (Optional) To use a deployment server to handle configurations for HEC tokens, click the Use Deployment Server check box.
9. (Optional) To have HEC listen and communicate over HTTPS rather than HTTP, click the Enable SSL checkbox.
10. (Optional) Enter a number in the HTTP Port Number field for HEC to listen on.

Create an Event Collector token on Splunk Enterprise

To use HEC, you must configure at least one token.

1. Click Settings > Add Data.
2. Click monitor.
3. Click HTTP Event Collector.
4. In the Name field, enter a name for the token.
5. (Optional) In the Source name override field, enter a source name for events that this input generates.
6. (Optional) In the Description field, enter a description for the input.
7. (Optional) In the Output Group field, select an existing forwarder output group.
8. (Optional) If you want to enable indexer acknowledgment for this token, click the Enable indexer acknowledgment checkbox.
9. Click Next.
10. (Optional) Confirm the source type and the index for HEC events.

Click Review.

1. Confirm that all settings for the endpoint are what you want.
2. If all settings are what you want, click Submit. Otherwise, click < to make changes.
3. (Optional) Copy the token value that Splunk Web displays and paste it into another document for reference later

Confirmed the Token is Status: Enabled

Configured payload-no-tls.json in /home/admin/payload-no-tls.json

Step:
Run the configuration command to apply the payload - either the CLI command, or the Gaia REST API command: n Method 1 - Run the CLI command "sklnctl": a. Save the JSON payload in a file (for example, /home/admin/payload.json). b. Run this command: sklnctl export --set "$(cat /home/admin/payload.json)"

Repeated steps for FW4

• Rebooted Gateway FW3 and FW4
• Rebooted Splunk Server
• Restarted all Check Point Firewall Skyline Components

Result: Data Failed to be ingested

Other troubleshooting completed:

• Created completely new token and repeated configuration steps
• Updated the url within the payload.json file to end with
  • /services/collector/raw
  • /services/collector/events
  • Updated “url”: http://10... Instead of https

Checked the Skyline Component Log Files for Troubleshooting:

• What are the relevant Check Point Skyline log files?

• OpenTelemetry Collector:

/opt/CPotelcol/otelcol.log

• CPView Exporter:

/opt/CPviewExporter/otlp_cpview.log

• CPView API Service:

$CPDIR/log/cpview_api_service.elg

Logs CPView API Service and CPView displayed no logs indicating causes of the issues.

Confirmed that the bearer token works:

Result: Bearer Token accepted.

Confirmed Collector was healthy:

Alternative payload-no-tls.json formats attempted:

 

Gateway Log Analysis (Returned everytime:)

SSH into CheckPoint_FW4 xx.xx.xx.xx via Remote Desktop

Ran Command: tail /opt/CPotelcol/otelcol.log

Result:

go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/internal/bounded_memory_queue.go:47

2024-06-26T14:20:34.609+1000    error   exporterhelper/queued_retry.go:391      Exporting failed. The error is not retryable. Dropping data.    {"kind": "exporter", "data_type": "metrics", "name": "prometheusremotewrite", "error": "Permanent error: Permanent error: remote write returned HTTP status 401 Unauthorized; err = %!w(<nil>): Bearer token not recognized. Please contact your Splunk admin.\n", "dropped_items": 284}

go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send

        go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/queued_retry.go:391

go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send

        go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/metrics.go:125

go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1

        go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/queued_retry.go:195

go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func1

Finding:

Appears to be an issue in which the HTTP Event Collector will not accept the Token Value, even when the token matches identically.

Could potentially be attributed to the payload-no-tls.json file not being formatted or compiled correctly on the Gateway.