Google GCP cloud logging to Splunk Cloud| Error in HEC

[mohammadsharukh]

Hi All,

We need to push logs from Google GCP cloud logging to Splunk Cloud through Splunk HEC

1. We are able to successfully push the GCP logs from GCP subscription to Dataflow but logs are not following past Dataflow to Splunk HEC.
2. Current Splunk HEC URLused in Dataflow to push logs to Splunk:- A) "https://1XX.X.8.XX:8088/services/collector" (Its splunk HF) B) "https://1XX.X.8.XX:8088/services/collector/raw" (Its splunk HF) C) "https://http-inputs.abcclient.splunkcloud.com:443/services/collector" (its splunk Cloud)
3. GCP dataflow is not accepting the Splunk HEC URL to send the data and throwing error
4. 1. Error appearing on GCP:-

   2021-06-18 11:10:27.021 IST Error message from worker: java.lang.RuntimeException: org.apache.beam.sdk.util.UserCodeException: java.lang.IllegalArgumentException: Invalid url format. Url format should match PROTOCOL://HOST[:PORT], where PORT is optional. Supported Protocols are http and https. eg: http://hostname:8088 org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory$1.typedApply(IntrinsicMapTaskExecutorFactory.java:197) org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory$1.typedApply(IntrinsicMapTaskExecutorFactory.java:168) org.apache.beam.runners.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:66) org.apache.beam.runners.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:53) org.apache.beam.runners.dataflow.worker.graph.Networks.replaceDirectedNetworkNodes(Networks.java:90) org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory.create(IntrinsicMapTaskExecutorFactory.java:128) org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1328) org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1100(StreamingDataflowWorker.java:163) org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$7.run(StreamingDataflowWorker.java:1105) java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) java.base/java.lang.Thread.run(Thread.java:834) Caused by: org.apache.beam.sdk.util.UserCodeException: java.lang.IllegalArgumentException: Invalid url format. Url format should match PROTOCOL://HOST[:PORT], where PORT is optional. Supported Protocols are http and https. eg: http://hostname:8088 org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:39) com.google.cloud.teleport.splunk.AutoValue_SplunkEventWriter$DoFnInvoker.invokeSetup(Unknown Source) org.apache.beam.runners.dataflow.worker.DoFnInstanceManagers$ConcurrentQueueInstanceManager.deserializeCopy(DoFnInstanceManagers.java:83) org.apache.beam.runners.dataflow.worker.DoFnInstanceManagers$ConcurrentQueueInstanceManager.peek(DoFnInstanceManagers.java:65) org.apache.beam.runners.dataflow.worker.UserParDoFnFactory.create(UserParDoFnFactory.java:99) org.apache.beam.runners.dataflow.worker.DefaultParDoFnFactory.create(DefaultParDoFnFactory.java:75) org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory.createParDoOperation(IntrinsicMapTaskExecutorFactory.java:267) org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory.access$000(IntrinsicMapTaskExecutorFactory.java:89) org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory$1.typedApply(IntrinsicMapTaskExecutorFactory.java:186) org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory$1.typedApply(IntrinsicMapTaskExecutorFactory.java:168) org.apache.beam.runners.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:66) org.apache.beam.runners.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:53) org.apache.beam.runners.dataflow.worker.graph.Networks.replaceDirectedNetworkNodes(Networks.java:90) org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory.create(IntrinsicMapTaskExecutorFactory.java:128) org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1328) org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1100(StreamingDataflowWorker.java:163) org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$7.run(StreamingDataflowWorker.java:1105) java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) java.base/java.lang.Thread.run(Thread.java:834) Caused by: java.lang.IllegalArgumentException: Invalid url format. Url format should match PROTOCOL://HOST[:PORT], where PORT is optional. Supported Protocols are http and https. eg: http://hostname:8088 org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument(Preconditions.java:122) com.google.cloud.teleport.splunk.SplunkEventWriter.setup(SplunkEventWriter.java:120)

Please let us know what Splunk HEC URL needs to be put? What should be the correct format???

Requesting all to provide a solution.

@judithsr  

@richgalloway 

@General_Talos 

@splunk_su 

@splunk 

@Anonymous 

@Anonymous 

[briqpayerik]

Hi,

@mohammadsharukh
@Ksr1982 


I found the following information here:  https://cloud.google.com/architecture/deploying-production-ready-log-exports-to-splunk-using-dataflo...

"YOUR_SPLUNK_HEC_URL must not include the HEC endpoint path, for example, /services/collector. Splunk Dataflow template currently only supports /services/collector endpoint for JSON-formatted events, and it automatically appends that path to your Splunk HEC URL input."

[Ksr1982]

Hi All, 

Even I am facing the same issue .

https://hostname.com:8088/services/collector (HF)
https://http-inputs-hostname.com:443/services/collector (Splunk cloud)

GCP dataflow is not accepting the Splunk HEC URL to send the data and throwing error

2021-08-18T08:08:34.108842395Z

Error message from worker: java.lang.RuntimeException: org.apache.beam.sdk.util.UserCodeException: java.lang.IllegalArgumentException: Invalid url format. Url format should match PROTOCOL://HOST[:PORT], where PORT is optional. Supported Protocols are http and https. eg: http://hostname:8088

Please respond @rphillips_splun