Unable to send events through log4j in Spark to Splunk

[davidco]

We want to use splunk-library-javalogging to send logs via Log4j  to Splunk Service

Environment: 

Spark with log4j2 in Azure Databricks ----> Splunk Enterprise

The config file log4j2.xml 

<?xml version="1.0" encoding="UTF-8"?><Configuration status="INFO" packages="com.splunk.logging,com.databricks.logging.log4j" shutdownHook="disable">
<Appenders>
    ...
    <SplunkHttp name="http-input"
      url="https://url-service"
      token="xxxx-xxxx-xxxx-xxx-xxx--xxxx"
      host=""
      index="my-index"
      source="spark-work"
      sourcetype="httpevent"
      messageFormat="text"
      middleware="HttpEventCollectorUnitTestMiddleware"
      connect_timeout="5000"
      termination_timeout="1000"
      disableCertificateValidation="true">
      <PatternLayout pattern="%m%n"/>
    </SplunkHttp>
  </Appenders>
  <Loggers>
    <Root level="INFO">
      ...
    </Root>
    ...
    <Logger name="splunk.log4j" level="DEBUG">
      <AppenderRef ref="http-input"/>
    </Logger>
  </Loggers>
</Configuration>

We use the library splunk-library-javalogging:

splunk-library-javalogging-1.11.8.jar with

• okhttp-4.11.0.jar
• okio-3.5.0.jar
• okio-jvm-3.5.0.jar

Currently we based the configuration from this example:

https://github.com/splunk/splunk-library-javalogging/blob/main/src/test/resources/log4j2.xml

Currently it doesn't work. We checked HEC via curl send a message from  Databricks to Splunk HEC and receive without problem.

Does anyone have any experience or can help us with some guidance or advice?

Thanks

[livehybrid]

Hi @davidco 

It'd be worth validating the Splunk receiving end and the logs available. Please could you check for HEC errors using:

index=_internal  reply!=0 HttpInputDataHandler

For more info on reply codes see https://docs.splunk.com/Documentation/Splunk/9.4.1/Data/TroubleshootHTTPEventCollector

Any error reply codes here may provide more insights. 

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[davidco]

Hello @livehybrid ,

First, thanks for your help.

I tried the query, but it didn't work. I mean, I got no information.

I even tested the HEC via curl, and everything seems normal.

[VatsalJagani]

@davidco- Did you check connectivity from Spark server to Splunk service on splunk HEC port?

* via telnet or curl

[davidco]

Hello @VatsalJagani ,

Yes, we checked via Curl

 curl -k -X POST 'https://hec-splunk.xxxxx.net/services/collector/event' --header 'Authorization: Splunk xxxx-xxxx-xxxx-xxx-xxxx' -d '{"sourcetype": "my_sample_data", "event": "2025-04-23-Test"}'

Result:

{"text":"Success","code":0}%

And we can see the event in Splunk 

[VatsalJagani]

Then I think definitely it something related to Log4j configuration or on Spark/Java side in which I have 0 experience, so I'm sorry I won't be able to help you, but I hope someone else in the community will be able to help.