Getting Data In

Unable to send events through log4j in Spark to Splunk

davidco
Loves-to-Learn

We want to use splunk-library-javalogging to send logs via Log4j  to Splunk Service

Environment: 

Spark with log4j2 in Azure Databricks ----> Splunk Enterprise

The config file log4j2.xml 

<?xml version="1.0" encoding="UTF-8"?><Configuration status="INFO" packages="com.splunk.logging,com.databricks.logging.log4j" shutdownHook="disable">
<Appenders>
    ...
    <SplunkHttp name="http-input"
      url="https://url-service"
      token="xxxx-xxxx-xxxx-xxx-xxx--xxxx"
      host=""
      index="my-index"
      source="spark-work"
      sourcetype="httpevent"
      messageFormat="text"
      middleware="HttpEventCollectorUnitTestMiddleware"
      connect_timeout="5000"
      termination_timeout="1000"
      disableCertificateValidation="true">
      <PatternLayout pattern="%m%n"/>
    </SplunkHttp>
  </Appenders>
  <Loggers>
    <Root level="INFO">
      ...
    </Root>
    ...
    <Logger name="splunk.log4j" level="DEBUG">
      <AppenderRef ref="http-input"/>
    </Logger>
  </Loggers>
</Configuration>

 

We use the library splunk-library-javalogging:

splunk-library-javalogging-1.11.8.jar with

  • okhttp-4.11.0.jar
  • okio-3.5.0.jar
  • okio-jvm-3.5.0.jar

Currently we based the configuration from this example:

https://github.com/splunk/splunk-library-javalogging/blob/main/src/test/resources/log4j2.xml

Currently it doesn't work. We checked HEC via curl send a message from  Databricks to Splunk HEC and receive without problem.

Does anyone have any experience or can help us with some guidance or advice?

Thanks

Labels (1)
0 Karma

livehybrid
Super Champion

Hi @davidco 

It'd be worth validating the Splunk receiving end and the logs available. Please could you check for HEC errors using:

index=_internal  reply!=0 HttpInputDataHandler

For more info on reply codes see https://docs.splunk.com/Documentation/Splunk/9.4.1/Data/TroubleshootHTTPEventCollector

Any error reply codes here may provide more insights. 

🌟 Did this answer help you? If so, please consider:

  • Adding karma to show it was useful
  • Marking it as the solution if it resolved your issue
  • Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

0 Karma

davidco
Loves-to-Learn

Hello @livehybrid ,

First, thanks for your help.

I tried the query, but it didn't work. I mean, I got no information.

I even tested the HEC via curl, and everything seems normal.

davidco_0-1745413184688.png

 

0 Karma

VatsalJagani
SplunkTrust
SplunkTrust

@davidco- Did you check connectivity from Spark server to Splunk service on splunk HEC port?

* via telnet or curl

 

 

0 Karma

davidco
Loves-to-Learn

Hello @VatsalJagani ,

Yes, we checked via Curl

 curl -k -X POST 'https://hec-splunk.xxxxx.net/services/collector/event' --header 'Authorization: Splunk xxxx-xxxx-xxxx-xxx-xxxx' -d '{"sourcetype": "my_sample_data", "event": "2025-04-23-Test"}'

 

Result:

{"text":"Success","code":0}%

 

And we can see the event in Splunk 

davidco_0-1745441475362.png

 

 

0 Karma

VatsalJagani
SplunkTrust
SplunkTrust

Then I think definitely it something related to Log4j configuration or on Spark/Java side in which I have 0 experience, so I'm sorry I won't be able to help you, but I hope someone else in the community will be able to help.

0 Karma
Get Updates on the Splunk Community!

Developer Spotlight with William Searle

The Splunk Guy: A Developer’s Path from Web to Cloud William is a Splunk Professional Services Consultant with ...

Major Splunk Upgrade – Prepare your Environment for Splunk 10 Now!

Attention App Developers: Test Your Apps with the Splunk 10.0 Beta and Ensure Compatibility Before the ...

Stay Connected: Your Guide to June Tech Talks, Office Hours, and Webinars!

What are Community Office Hours?Community Office Hours is an interactive 60-minute Zoom series where ...