Unable to see logs in my Splunk UI from my spring boot application with log4j HEC

[vaibhav1695]

I am unable to see any logs in splunk from my spring boot application. I am adding my xml property file, controller file, dependency file and splunk data input screenshots to help resolving the issue. I am breaking my head for past couple of days and unable to find what I am missing.

HEC data input config UI

HEC data input edit UI

Global Settings

The following API is logging events in my index:

curl -k http://localhost:8088/services/collector/event -H "Authorization: Splunk ***" -d '{"event": "Hello, Splunk!"}'

This is my log4j2-spring.xml:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration>
    <Appenders>
        <Console name="console" target="SYSTEM_OUT">
            <PatternLayout
                    pattern="%style{%d{ISO8601}} %highlight{%-5level }[%style{%t}{bright,blue}] %style{%C{10}}{bright,yellow}: %msg%n%throwable" />
        </Console>
        <SplunkHttp
                name="splunkhttp"
                url="http://localhost:8088"
                token="***"
                host="localhost"
                index="customer_api_dev"
                type="raw"
                source="http-event-logs"
                sourcetype="log4j"
                messageFormat="text"
                disableCertificateValidation="true">
            <PatternLayout pattern="%m" />
        </SplunkHttp>

    </Appenders>

    <Loggers>
        <!-- LOG everything at DEBUG level -->
        <Root level="debug">
            <AppenderRef ref="console" />
            <AppenderRef ref="splunkhttp" />
        </Root>
    </Loggers>
</Configuration>

This is my controller:

package com.example.advanceddbconcepts.controller;

import com.example.advanceddbconcepts.entity.Customer;
import com.example.advanceddbconcepts.entity.Order;
import com.example.advanceddbconcepts.service.CustomerService;
import lombok.Getter;
import lombok.Setter;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.*;

import java.util.List;

@RestController
@RequestMapping("/api/customers")
public class CustomerController {

    Logger logger = LogManager.getLogger(CustomerController.class);

    private final CustomerService customerService;

    public CustomerController(CustomerService customerService) {
        this.customerService = customerService;
    }

    @PostMapping
    public ResponseEntity<Customer> createCustomerWithOrder(@RequestBody CustomerRequestOrder request) {
        Customer customer = new Customer(request.getCustomerName());
        logger.info("Created a customer with name {}", request.getCustomerName());
        List<Order> orders = request
                .getProductName()
                .stream()
                .map(Order::new)
                .toList();
        Customer savedCustomer = customerService.createCustomerWithOrder(customer, orders);
        logger.info("API is successful");
        return ResponseEntity.ok().body(savedCustomer);
    }

    @Getter
    @Setter
    public static class CustomerRequestOrder {
        private String customerName;
        private List<String> productName;
    }
}

I have added below dependencies in my pom.xml

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-log4j2</artifactId>
            <version>3.3.3</version>
        </dependency>

        <dependency>
            <groupId>com.splunk.logging</groupId>
            <artifactId>splunk-library-javalogging</artifactId>
            <version>1.11.8</version>
        </dependency>
    </dependencies>

I am unable to see any logs in splunk after I hit the API. I am able to see logs in my local:

2024-09-02T19:37:00.629+05:30  INFO 24912 --- [nio-8080-exec-4] c.e.a.controller.CustomerController      : Created a customer with name John Doe
2024-09-02T19:37:00.667+05:30  INFO 24912 --- [nio-8080-exec-4] c.e.a.controller.CustomerController      : API is successful