Using the Splunk HTTP Event Collector with log4j2, why are some message field keys not being parsed?

[AlGon]

Hello,

I use Splunk HTTP Event Collector (splunk-library-javalogging-1.5.1.jar) with log4j2. Here is my (simplified) log4 configuration file:

 <?xml version="1.0" encoding="UTF-8"?>
    <Configuration status="DEBUG" monitorInterval="30">
        <Appenders>
            <!-- Splunk logger -->
            <Http name="http" url="http://%host%:8088" token="%splunk-token%" >
                <PatternLayout pattern="%m" />
            </Http>
        </Appenders>
        <Loggers>
            <Root level="info">
            </Root>
            <Logger name="eventLogger" level="info" additivity="false">
                <AppenderRef ref="http" />
            </Logger>
        </Loggers>
    </Configuration>

I use SplunkCimLogEvent class to format the logging message as explained in the class javadoc:

import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
...
private static final Logger EVENT_LOGGER = LogManager.getLogger("eventLogger");
...
SplunkCimLogEvent splunkCimLogEvent = new SplunkCimLogEvent("eventName", "eventId");
splunkCimLogEvent.addField("field1", field1Value);
splunkCimLogEvent.addField("field2", field2Value);
splunkCimLogEvent.setAuthApp("appName");
splunkCimLogEvent.setAuthUser("userName");

EVENT_LOGGER.info(splunkCimLogEvent.toString());

In Splunk, I get event formatted like following (raw text) :

{"message": "\"name=eventName\" \"event_id=eventId\" \"field1=field1\" \"field2=field2\" \"app=appName\" \"user=userName\"","severity":"INFO"} 

I get only "message" and "severity" as JSON title (see attached image)

whereas I was expected every message field key as JSON title.
Is it possible to do this with log4j?

Thank you.

[obesechicken13]

I don't think the appenders can log events with specific fields. They put everything in the message field. It's then difficult to extract these fields.

[gblock_splunk · Accepted solution]

Hi AlGon

I just looked at this again, my previous answer was incorrect which I deleted.

If you set CIM fields in the logging library, today, they are appended as key-value-pairs in the message. The logging libraries (log4j etc) are not built around objects. That message is then forwarded on within the JSON payload which HEC requires. Splunk does not support extracting kvp from a value of a field in a JSON object today. If it did, this would not be a problem and the fields would be properly extracted. We have a filed a bug for this and are exploring potentially fixing this in the future. We're also adding support in HEC for sending raw payloads (we have this already in cloud), so we may be able to add support in our logging libraries, which would also solve the problem.

In the interim, one option is to explore is using regex extractions to extract the fields. I've done some spiking on this which I posted here. Basically it introduces a sourcetype which extracts any kvp it finds in the "event" value. The same approach may work for your need.

[AlGon]

Thanks for you answer.

I chose another solution which is to build manually the HTTP request:

    InetAddress ip = InetAddress.getLocalHost();

    JSONObject bodyRequestJson = new JSONObject();
    bodyRequestJson.put("host", ip.getHostName());
    bodyRequestJson.put("event", event.toJson());
    StringEntity httpBody = new StringEntity(bodyRequestJson.toJSONString());

    HttpClient httpclient = HttpClientBuilder.create().build();
    HttpPost httpPost = new HttpPost("http://hostname:8088/services/collector/event");
    httpPost.setHeader("Content-type", "application/json");
    httpPost.setHeader("Authorization", "Splunk token");
    httpPost.setEntity(httpBody);
    HttpResponse response = httpclient.execute(httpPost);
    LOGGER.info("http event collector response: " + response);

I obtain what I want.
I will follow the changes on Splunk HEC and log4j.

[viliam]

Can you please advise what is the progress with fixing or expected ETA?
I'm using Splunk enterprise and it would help me heaps if we can get all fields parsed.

Thanks

viliam

[gblock_splunk]

Hi villam

Did you check the link I sent. That's the only mechanism in the short term. No ETA yet on the fix in Splunk.

[gblock_splunk]

I mean this link: https://gist.github.com/glennblock/0d5e6384d93449d3e7c6