Getting Data In

HEC - Why is HTTP Event Collector not Ingesting json logs?

rtengineer
Engager

We have an on-prem Splunk Enterprise instance using a Deployment server, indexers, search head, etc.  The environment sits on Windows 2019 and Splunk is version 8.2.3.

We have recently setup a HTTP Event Collector token for HEC Collection.  It is working correctly from Curl both in Health check and absorbing the manual calls from curl and postman, such as:

curl -k http://deploymentserver.local:8088/services/collector/raw -H "Authorization:Splunk 1920123a-f2b1-4c46-b848-6fba456789fe7" -d '{"Sourcetype":"log4j","event":"test"}'

These particular calls are ingested and searchable within Splunk.  We have opened a ticket with Splunk, but has been less than helpful as the are just directing us to the token setup which as noted above is working.

The issue, it is not ingesting any of our actual application logs.  There are no errors, it has the correct token, it's like it's not getting there or rejected.    We've made changes to the sourcetype so there is no criteria as well as set for json, no difference either way.  So we suspect the formatting of our json is incorrect.  Are there any good samples out there?

This is what the json looks like:

 

 

<?xml version="1.0" encoding="utf-8"?>
<Configuration status="INFO" name="cloudhub"
packages="com.appforsplunk.ch.logging.appender, com.splunk.logging
,org.apache.logging.log4j">
<Appenders>
<Console name="Console" target="SYSTEM_OUT">
<PatternLayout pattern="%-5p %d [%t] [event: %X{correlationId}] %c: %m%n" />
</Console>
<Console name="ConsoleLogUtil" target="SYSTEM_OUT">
<PatternLayout pattern="%m%n" />
</Console>
<RollingFile name="file" fileName="${sys:splunkapp.home}${sys:file.separator}logs${sys:file.separator}splunkapp-custom-logging-api.log"
filePattern="${sys:splunkapp.home}${sys:file.separator}logs${sys:file.separator}splunkapp-custom-logging-api-%i.log">
<PatternLayout pattern="%-5p %d [%t] [processor: %X{processorPath}; event: %X{correlationId}] %c: %m%n" />
<SizeBasedTriggeringPolicy size="10 MB" />
<DefaultRolloverStrategy max="10"/>
</RollingFile>
<SplunkHttp name="splunk" url="http://deploymentserver.local:8088/services/collector/raw" token="Splunk 50817720-52e2-4481-a2cf-eb519716354c" disableCertificateValidation="true">
<PatternLayout pattern="%-5p %d [%t] [event: %X{correlationId}] %c: %m%n"/>
</SplunkHttp>
<Log4J2CloudhubLogAppender name="CLOUDHUB" addressProvider="com.appforsplunk.ch.logging.DefaultAggregatorAddressProvider"
applicationContext="com.appforsplunk.ch.logging.DefaultApplicationContext"
appendRetryIntervalMs="${sys:logging.appendRetryInterval}"
appendMaxAttempts="${sys:logging.appendMaxAttempts}"
batchSendIntervalMs="${sys:logging.batchSendInterval}"
batchMaxRecords="${sys:logging.batchMaxRecords}" memBufferMaxSize="${sys:logging.memBufferMaxSize}"
journalMaxWriteBatchSize="${sys:logging.journalMaxBatchSize}"
journalMaxFileSize="${sys:logging.journalMaxFileSize}"
clientMaxPacketSize="${sys:logging.clientMaxPacketSize}"
clientConnectTimeoutMs="${sys:logging.clientConnectTimeout}"
clientSocketTimeoutMs="${sys:logging.clientSocketTimeout}"
serverAddressPollIntervalMs="${sys:logging.serverAddressPollInterval}"
serverHeartbeatSendIntervalMs="${sys:logging.serverHeartbeatSendIntervalMs}"
statisticsPrintIntervalMs="${sys:logging.statisticsPrintIntervalMs}">
<PatternLayout pattern="[%d{MM-dd HH:mm:ss}] %-5p %c{1} [%t]: %m%n" />
</Log4J2CloudhubLogAppender>
</Appenders>
<Loggers>
<AsyncLogger name="org.splunkapp.service.http" level="WARN"/>
<AsyncLogger name="org.splunkapp.extension.http" level="WARN"/>
<!-- splunkapp logger -->
<AsyncLogger name="org.splunkapp.runtime.core.internal.processor.LoggerMessageProcessor" level="INFO"/>
<AsyncRoot level="INFO">
<AppenderRef ref="splunk" />
<AppenderRef ref="CLOUDHUB" />
<AppenderRef ref="Console"/>
<AppenderRef ref="file" />
</AsyncRoot>
</Loggers>
</Configuration>

 

 

 

 

Thanks in advance.

Labels (1)
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...