HEC - Why is HTTP Event Collector not Ingesting json logs?

[rtengineer]

We have an on-prem Splunk Enterprise instance using a Deployment server, indexers, search head, etc.  The environment sits on Windows 2019 and Splunk is version 8.2.3.

We have recently setup a HTTP Event Collector token for HEC Collection.  It is working correctly from Curl both in Health check and absorbing the manual calls from curl and postman, such as:

curl -k http://deploymentserver.local:8088/services/collector/raw -H "Authorization:Splunk 1920123a-f2b1-4c46-b848-6fba456789fe7" -d '{"Sourcetype":"log4j","event":"test"}'

These particular calls are ingested and searchable within Splunk.  We have opened a ticket with Splunk, but has been less than helpful as the are just directing us to the token setup which as noted above is working.

The issue, it is not ingesting any of our actual application logs.  There are no errors, it has the correct token, it's like it's not getting there or rejected.    We've made changes to the sourcetype so there is no criteria as well as set for json, no difference either way.  So we suspect the formatting of our json is incorrect.  Are there any good samples out there?

This is what the json looks like:

<?xml version="1.0" encoding="utf-8"?>
<Configuration status="INFO" name="cloudhub"
packages="com.appforsplunk.ch.logging.appender, com.splunk.logging
,org.apache.logging.log4j">
<Appenders>
<Console name="Console" target="SYSTEM_OUT">
<PatternLayout pattern="%-5p %d [%t] [event: %X{correlationId}] %c: %m%n" />
</Console>
<Console name="ConsoleLogUtil" target="SYSTEM_OUT">
<PatternLayout pattern="%m%n" />
</Console>
<RollingFile name="file" fileName="${sys:splunkapp.home}${sys:file.separator}logs${sys:file.separator}splunkapp-custom-logging-api.log"
filePattern="${sys:splunkapp.home}${sys:file.separator}logs${sys:file.separator}splunkapp-custom-logging-api-%i.log">
<PatternLayout pattern="%-5p %d [%t] [processor: %X{processorPath}; event: %X{correlationId}] %c: %m%n" />
<SizeBasedTriggeringPolicy size="10 MB" />
<DefaultRolloverStrategy max="10"/>
</RollingFile>
<SplunkHttp name="splunk" url="http://deploymentserver.local:8088/services/collector/raw" token="Splunk 50817720-52e2-4481-a2cf-eb519716354c" disableCertificateValidation="true">
<PatternLayout pattern="%-5p %d [%t] [event: %X{correlationId}] %c: %m%n"/>
</SplunkHttp>
<Log4J2CloudhubLogAppender name="CLOUDHUB" addressProvider="com.appforsplunk.ch.logging.DefaultAggregatorAddressProvider"
applicationContext="com.appforsplunk.ch.logging.DefaultApplicationContext"
appendRetryIntervalMs="${sys:logging.appendRetryInterval}"
appendMaxAttempts="${sys:logging.appendMaxAttempts}"
batchSendIntervalMs="${sys:logging.batchSendInterval}"
batchMaxRecords="${sys:logging.batchMaxRecords}" memBufferMaxSize="${sys:logging.memBufferMaxSize}"
journalMaxWriteBatchSize="${sys:logging.journalMaxBatchSize}"
journalMaxFileSize="${sys:logging.journalMaxFileSize}"
clientMaxPacketSize="${sys:logging.clientMaxPacketSize}"
clientConnectTimeoutMs="${sys:logging.clientConnectTimeout}"
clientSocketTimeoutMs="${sys:logging.clientSocketTimeout}"
serverAddressPollIntervalMs="${sys:logging.serverAddressPollInterval}"
serverHeartbeatSendIntervalMs="${sys:logging.serverHeartbeatSendIntervalMs}"
statisticsPrintIntervalMs="${sys:logging.statisticsPrintIntervalMs}">
<PatternLayout pattern="[%d{MM-dd HH:mm:ss}] %-5p %c{1} [%t]: %m%n" />
</Log4J2CloudhubLogAppender>
</Appenders>
<Loggers>
<AsyncLogger name="org.splunkapp.service.http" level="WARN"/>
<AsyncLogger name="org.splunkapp.extension.http" level="WARN"/>
<!-- splunkapp logger -->
<AsyncLogger name="org.splunkapp.runtime.core.internal.processor.LoggerMessageProcessor" level="INFO"/>
<AsyncRoot level="INFO">
<AppenderRef ref="splunk" />
<AppenderRef ref="CLOUDHUB" />
<AppenderRef ref="Console"/>
<AppenderRef ref="file" />
</AsyncRoot>
</Loggers>
</Configuration>

Thanks in advance.