Splunk Cloud Platform

Google GCP cloud logging to Splunk Cloud| Error in HEC

mohammadsharukh
Path Finder

Hi All,

 

We need to push logs from Google GCP cloud logging to Splunk Cloud through Splunk HEC

  1. We are able to successfully push the GCP logs from GCP subscription to Dataflow but logs are not following past Dataflow to Splunk HEC.
  2. Current Splunk HEC URLused in Dataflow to push logs to Splunk:- A) "https://1XX.X.8.XX:8088/services/collector" (Its splunk HF) B) "https://1XX.X.8.XX:8088/services/collector/raw" (Its splunk HF) C) "https://http-inputs.abcclient.splunkcloud.com:443/services/collector" (its splunk Cloud)
  3. GCP dataflow is not accepting the Splunk HEC URL to send the data and throwing error
    1. Error appearing on GCP:-

    2021-06-18 11:10:27.021 IST Error message from worker: java.lang.RuntimeException: org.apache.beam.sdk.util.UserCodeException: java.lang.IllegalArgumentException: Invalid url format. Url format should match PROTOCOL://HOST[:PORT], where PORT is optional. Supported Protocols are http and https. eg: http://hostname:8088 org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory$1.typedApply(IntrinsicMapTaskExecutorFactory.java:197) org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory$1.typedApply(IntrinsicMapTaskExecutorFactory.java:168) org.apache.beam.runners.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:66) org.apache.beam.runners.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:53) org.apache.beam.runners.dataflow.worker.graph.Networks.replaceDirectedNetworkNodes(Networks.java:90) org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory.create(IntrinsicMapTaskExecutorFactory.java:128) org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1328) org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1100(StreamingDataflowWorker.java:163) org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$7.run(StreamingDataflowWorker.java:1105) java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) java.base/java.lang.Thread.run(Thread.java:834) Caused by: org.apache.beam.sdk.util.UserCodeException: java.lang.IllegalArgumentException: Invalid url format. Url format should match PROTOCOL://HOST[:PORT], where PORT is optional. Supported Protocols are http and https. eg: http://hostname:8088 org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:39) com.google.cloud.teleport.splunk.AutoValue_SplunkEventWriter$DoFnInvoker.invokeSetup(Unknown Source) org.apache.beam.runners.dataflow.worker.DoFnInstanceManagers$ConcurrentQueueInstanceManager.deserializeCopy(DoFnInstanceManagers.java:83) org.apache.beam.runners.dataflow.worker.DoFnInstanceManagers$ConcurrentQueueInstanceManager.peek(DoFnInstanceManagers.java:65) org.apache.beam.runners.dataflow.worker.UserParDoFnFactory.create(UserParDoFnFactory.java:99) org.apache.beam.runners.dataflow.worker.DefaultParDoFnFactory.create(DefaultParDoFnFactory.java:75) org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory.createParDoOperation(IntrinsicMapTaskExecutorFactory.java:267) org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory.access$000(IntrinsicMapTaskExecutorFactory.java:89) org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory$1.typedApply(IntrinsicMapTaskExecutorFactory.java:186) org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory$1.typedApply(IntrinsicMapTaskExecutorFactory.java:168) org.apache.beam.runners.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:66) org.apache.beam.runners.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:53) org.apache.beam.runners.dataflow.worker.graph.Networks.replaceDirectedNetworkNodes(Networks.java:90) org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory.create(IntrinsicMapTaskExecutorFactory.java:128) org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1328) org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1100(StreamingDataflowWorker.java:163) org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$7.run(StreamingDataflowWorker.java:1105) java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) java.base/java.lang.Thread.run(Thread.java:834) Caused by: java.lang.IllegalArgumentException: Invalid url format. Url format should match PROTOCOL://HOST[:PORT], where PORT is optional. Supported Protocols are http and https. eg: http://hostname:8088 org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument(Preconditions.java:122) com.google.cloud.teleport.splunk.SplunkEventWriter.setup(SplunkEventWriter.java:120)

 

Please let us know what Splunk HEC URL needs to be put? What should be the correct format???

 

Requesting all to provide a solution.

@judithsr  

@richgalloway 

@General_Talos 

@splunk_su 

@splunk 

@Anonymous 

@Anonymous 

Tags (2)
0 Karma

briqpayerik
Explorer

Hi,

@mohammadsharukh
@Ksr1982 


I found the following information here:  https://cloud.google.com/architecture/deploying-production-ready-log-exports-to-splunk-using-dataflo...


"YOUR_SPLUNK_HEC_URL must not include the HEC endpoint path, for example, /services/collector. Splunk Dataflow template currently only supports /services/collector endpoint for JSON-formatted events, and it automatically appends that path to your Splunk HEC URL input."

0 Karma

Ksr1982
Explorer

Hi All, 

Even I am facing the same issue .

https://hostname.com:8088/services/collector (HF)
https://http-inputs-hostname.com:443/services/collector (Splunk cloud)

GCP dataflow is not accepting the Splunk HEC URL to send the data and throwing error

Error message from worker: java.lang.RuntimeException: org.apache.beam.sdk.util.UserCodeException: java.lang.IllegalArgumentException: Invalid url format. Url format should match PROTOCOL://HOST[:PORT], where PORT is optional. Supported Protocols are http and https. eg: http://hostname:8088

Please respond @rphillips_splun

 

Tags (2)
0 Karma
Get Updates on the Splunk Community!

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

Industry Solutions for Supply Chain and OT, Amazon Use Cases, Plus More New Articles ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Enterprise Security Content Update (ESCU) | New Releases

In November, the Splunk Threat Research Team had one release of new security content via the Enterprise ...