If you find issue happens after windows server is restarted. Restarting splunk universal forwarder fixes the issue. Then try one of the following workarounds.
Use 'Delayed Start' for the Splunk Forwarder service. (
https://community.splunk.com/t5/Getting-Data-In/Why-quot-FormatMessage-error-quot-appears-in-indexed...). However it's hard to configure thousands of DCs.
Or
Configure interval as cron schedule instead.
interval = [<decimal>|<cron schedule>]
[WinEventLog]
interval=* * * * *
By default wineventlog interval is 60 sec. That means as soon as splunk is restarted, wineventlog (or any modinput) is immediately started. Subsequently every 60( configured interval) splunk checks if modinput is still running. If not re-launch modinput.
Instead of setting interval 60 sec, if we use cron schedule to run every minute, then splunk is not going to launch modinput immediately. So essentially the idea is to convert interval setting from decimal to cron schedule to introduce a delay.
If above does not solve the issue, meaning the issue is not related to windows server restart.
Try one of the following workarounds.
Workaround 1.
in inputs.conf
- Stop UF
- Set batch_size to 1.
- Start UF
[<impacted channel>]
batch_size=1
Workaround 2.
in inputs.conf
- Stop UF
- backup checkpoint file $SPLUNK_HOME\var\lib\splunk\modinputs\WinEventLog\<impacted channel> . Saves current record id
- In inputs.conf for impacted channel add use_old_eventlog_api=true
- Start UF
- Stop UF after 30 sec
- Replace current record with record id found in step 2. So that ingestion starts from right place.
- When setting use_old_eventlog_api=false ( in case if it does help). Follow all above steps so that UF correctly starts from desired record id.
[<impacted channel>]
use_old_eventlog_api=true
