Getting Data In

Unable to send events through log4j in Spark to Splunk

davidco
Loves-to-Learn

We want to use splunk-library-javalogging to send logs via Log4j  to Splunk Service

Environment: 

Spark with log4j2 in Azure Databricks ----> Splunk Enterprise

The config file log4j2.xml 

<?xml version="1.0" encoding="UTF-8"?><Configuration status="INFO" packages="com.splunk.logging,com.databricks.logging.log4j" shutdownHook="disable">
<Appenders>
    ...
    <SplunkHttp name="http-input"
      url="https://url-service"
      token="xxxx-xxxx-xxxx-xxx-xxx--xxxx"
      host=""
      index="my-index"
      source="spark-work"
      sourcetype="httpevent"
      messageFormat="text"
      middleware="HttpEventCollectorUnitTestMiddleware"
      connect_timeout="5000"
      termination_timeout="1000"
      disableCertificateValidation="true">
      <PatternLayout pattern="%m%n"/>
    </SplunkHttp>
  </Appenders>
  <Loggers>
    <Root level="INFO">
      ...
    </Root>
    ...
    <Logger name="splunk.log4j" level="DEBUG">
      <AppenderRef ref="http-input"/>
    </Logger>
  </Loggers>
</Configuration>

 

We use the library splunk-library-javalogging:

splunk-library-javalogging-1.11.8.jar with

  • okhttp-4.11.0.jar
  • okio-3.5.0.jar
  • okio-jvm-3.5.0.jar

Currently we based the configuration from this example:

https://github.com/splunk/splunk-library-javalogging/blob/main/src/test/resources/log4j2.xml

Currently it doesn't work. We checked HEC via curl send a message from  Databricks to Splunk HEC and receive without problem.

Does anyone have any experience or can help us with some guidance or advice?

Thanks

Labels (1)
0 Karma

livehybrid
Super Champion

Hi @davidco 

It'd be worth validating the Splunk receiving end and the logs available. Please could you check for HEC errors using:

index=_internal  reply!=0 HttpInputDataHandler

For more info on reply codes see https://docs.splunk.com/Documentation/Splunk/9.4.1/Data/TroubleshootHTTPEventCollector

Any error reply codes here may provide more insights. 

🌟 Did this answer help you? If so, please consider:

  • Adding karma to show it was useful
  • Marking it as the solution if it resolved your issue
  • Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

0 Karma

davidco
Loves-to-Learn

Hello @livehybrid ,

First, thanks for your help.

I tried the query, but it didn't work. I mean, I got no information.

I even tested the HEC via curl, and everything seems normal.

davidco_0-1745413184688.png

 

0 Karma

VatsalJagani
SplunkTrust
SplunkTrust

@davidco- Did you check connectivity from Spark server to Splunk service on splunk HEC port?

* via telnet or curl

 

 

0 Karma

davidco
Loves-to-Learn

Hello @VatsalJagani ,

Yes, we checked via Curl

 curl -k -X POST 'https://hec-splunk.xxxxx.net/services/collector/event' --header 'Authorization: Splunk xxxx-xxxx-xxxx-xxx-xxxx' -d '{"sourcetype": "my_sample_data", "event": "2025-04-23-Test"}'

 

Result:

{"text":"Success","code":0}%

 

And we can see the event in Splunk 

davidco_0-1745441475362.png

 

 

0 Karma

VatsalJagani
SplunkTrust
SplunkTrust

Then I think definitely it something related to Log4j configuration or on Spark/Java side in which I have 0 experience, so I'm sorry I won't be able to help you, but I hope someone else in the community will be able to help.

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud's AI Assistant in Action Series: Auditing Compliance and ...

This is the third post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how to ...

Splunk Community Badges!

  Hey everyone! Ready to earn some serious bragging rights in the community? Along with our existing badges ...

What You Read The Most: Splunk Lantern’s Most Popular Articles!

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...