Unable to get logs in splunk from mulesoft

[fhatrick]

Hi, I have created a new token and index in splunk for my mulesoft project.

These are the configurations I have done in mulesoft to get the splunk logs.Despite this I am unable to see any logs in the dashboard when i search like index="indexname".

LOG4J2.XML FILE CHANGES

<Configuration status="INFO" name="cloudhub"

packages="com.mulesoft.ch.logging.appender,com.splunk.logging,org.apache.l

ogging.log4j">

<Appenders>

<RollingFile "Rolling file details here"

</RollingFile>

<SplunkHttp name="Splunk"

url="localhost url"

token="token" index="indexname"

batch_size_count="10" disableCertificateValidation="true">

<PatternLayout

pattern="%-5p %d [%t] [processor: %X{processorPath}; event: %X{correlationId}] %c: %m%n" />

</SplunkHttp>

<Log4J2CloudhubLogAppender name="CloudHub"

addressProvider="com.mulesoft.ch.logging.DefaultAggregatorAddressProvider"

applicationContext="com.mulesoft.ch.logging.DefaultApplicationContext"

appendRetryIntervalMs="${sys:logging.appendRetryInterval}"

appendMaxAttempts="${sys:logging.appendMaxAttempts}"

batchSendIntervalMs="${sys:logging.batchSendInterval}"

batchMaxRecords="${sys:logging.batchMaxRecords}"

memBufferMaxSize="${sys:logging.memBufferMaxSize}"

journalMaxWriteBatchSize="${sys:logging.journalMaxBatchSize}"

journalMaxFileSize="${sys:logging.journalMaxFileSize}"

clientMaxPacketSize="${sys:logging.clientMaxPacketSize}"

clientConnectTimeoutMs="${sys:logging.clientConnectTimeout}"

clientSocketTimeoutMs="${sys:logging.clientSocketTimeout}"

serverAddressPollIntervalMs="${sys:logging.serverAddressPollInterval}"

serverHeartbeatSendIntervalMs="${sys:logging.serverHeartbeatSendIntervalMs}"

statisticsPrintIntervalMs="${sys:logging.statisticsPrintIntervalMs}">

</Log4J2CloudhubLogAppender>

</Appenders>

<Loggers>

<AsyncLogger name="org.mule.service.http" level="WARN" />

<AsyncLogger name="org.mule.extension.http" level="WARN" />

<AsyncLogger name="org.mule.runtime.core.internal.processor.LoggerMessageProcessor" level="INFO" />

<AsyncRoot level="INFO">

<AppenderRef ref="file" />

<AppenderRef ref="Splunk" />

<AppenderRef ref="CloudHub" />

</AsyncRoot>

<AsyncLogger name="Splunk.Logger" level="INFO">

<AppenderRef ref="splunk" />

</AsyncLogger>

</Loggers>

</Configuration>

POM.XML FILE CHANGES

<repository>

<id>splunk-artifactory</id>

<name>Splunk Releases</name>

<url>https://splunk.jfrog.io/splunk/ext-releases-local</url>

</repository>

<dependency>

<groupId>com.splunk.logging</groupId>

<artifactId>splunk-library-javalogging</artifactId>

<version>1.7.3</version>

</dependency>

<dependency>

<groupId>org.apache.logging.log4j</groupId>

<artifactId>log4j-core</artifactId>

<version>2.10.0</version>

</dependency>

<dependency>

<groupId>org.apache.logging.log4j</groupId>

<artifactId>log4j-api</artifactId>

<version>2.10.0</version>

</dependency>

Please let me know if i am missing out on any configuration since i believe i am pretty much following what's in the mule website and other articles.

[livehybrid]

Hi @fhatrick 

Check the following points to troubleshoot why logs are not appearing in Splunk using the HEC-based SplunkHTTP log4j logging options:

1. Splunk HEC URL and Token

• Ensure the url in your config points to your Splunk HTTP Event Collector (HEC) endpoint, not localhost unless Splunk is running on the same host as MuleSoft.
• Example: url="https://<splunk-server>:8088"

• The token value must match exactly the HEC token configured in Splunk.

1. HEC Configuration in Splunk

• Confirm that HEC is enabled in Splunk (Settings > Data Inputs > HTTP Event Collector).
• The token is enabled and assigned to the correct index (indexname).

1. Index Existence and Permissions

• Verify the index (indexname) exists in Splunk and your user has permission to search it.

1. Network Connectivity

• Ensure the MuleSoft server can reach the Splunk HEC endpoint (no firewall or network issues) - use something like netcat to check this (nc -vz -w1 yourServer 8088)

1. Testing HEC Directly

• Test HEC by sending a sample event using:

curl -k https://<splunk-server>:8088/services/collector/event \ -H "Authorization: Splunk <token>" \ -d '{"event":"test event", "index":"indexname"}' 

If this event appears in Splunk, the HEC and index are working.

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[fhatrick]

My url is "http://127.0.0.1:8000" in log4j2 and localhost is running on same port. Whereas the listener is 8081 port. Do all of these have to be same? Am i missing out anywhere?

[livehybrid]

Hi @fhatrick 

Splunk HEC typically listens on port 8088 - Have you changed this default port to something else? Have you enabled SSL for HEC? If not you will need to use http:// instead of https://

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[fhatrick]

The url is  "http://127.0.0.1:8088" in log4j2  and localhost(splunk) is running on  port 8000.Whereas the project listener is 8081 port.

Yes i have enabled ssl.

Most documentation have the same setting so i followed the same ,yet cannot see the logs.

[richgalloway]

Have you enabled receiving of data in Splunk?  Go to Settings->"Forwarding and Receiving"  to turn on receiving.

Does "localhost url" include the port number (9997 by default)?

Do your firewalls allow connections between Mulesoft and Splunk?

[fhatrick]

My url is "http://127.0.0.1:8000" in log4j2 and localhost(splunk) is running on same port. Whereas the listener is 8081 port.

Earlier the url was  "http://127.0.0.1:8088" in log4j2 localhost(splunk) is running on  port 8000.Whereas the listener is 8081 port.