Recent logs are not available in splunk even if recent internal logs are missing. How to fix the issue?

[isahu]

I onboarded one production logs to splunk but after restarting the UF I am not able to see the recent logs also I am not able to see the recent internal logs. How to fix this issue please help?

[livehybrid]

Hi @isahu 

Can you have a look at $SPLUNK_HOME/var/log/splunk/splunkd.log specifically errors relating to "TcpOutputProc" or "TcpOutputFd".

Please also confirm that the outputs.conf is configured as expected using btool:

$SPLUNK_HOME/bin/splunk cmd btool outputs list --debug

As others have said, the lack of _internal logs from the UF points to an issue with sending outbound, hopefully the above troubleshooting will help determine the cause of the issue.

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[isoutamo]

Can you check on UF side what it has written into local log files? There should be information why it can’t send those into your indexers.

[richgalloway]

If neither the production logs nor the internal logs from the same UF are found then it's likely that the UF has lost its connection to the indexer(s).

Verify the UF has the right settings in outputs.conf

Verify the network still allows connections from the UF to the indexers.

If you use TLS, confirm the certificate is still valid and that the UF has the correct password.