Missing Splunk Logs - same configuration in development and production enviroment

[mappu]

Hello,

We have been investigating on missing 30% of Splunk logs in our production environment. I'm thinking it maybe due to TIME_FORMAT or due to high volume logs on production. Can you please let me know what should be the key-value for TIME_FORMAT on props.conf file? 
Lagsec value is 1.5seconds on source logs and the splunk forwarder log source type where we are checking has 1.13s. 
Additionally, source logs have format: 05/Mar/2024
SplunkForwarder logs have format: 2024-03-05

2048kbps on both dev and prod config file.

Also, have ignoreOlderThan=1d so, looking to remove this parameter and fix TIME_FORMAT and check out. Can you please help or provide additional information to check?

[isoutamo]

Hi

in many cases if you haven't done data onboarding correctly and setting TIME_FORMAT correctly Splunk can decide that 05/03/2024 is actually 3rd of May 2024 not 5th or March 2024.

To check this you need to look if those events are in future. That needs that you add correct end data or actually enough long span into future e.g. latest=+10mon in your SPL query.

You can also check if there is issues on those date parsing on MC and/or from internal logs.

r. Ismo

[gcusello]

Hi @mappu,

check with the following search:

index=your_index
| eval diff=_indextime-_time
| eval indextime=strftime(_indextime,"%Y-%m-%d %H:%M:$S")
| table _time indextime diff

if you have high differences between _time and indextime, you have a queue issue, if not the problem is another.

About timestamp, check if in the loosing logs you have the timestamp definition or not, but using the formats you described, you souldn't have this issue.

Ciao.

Giuseppe

[mappu]

Thank you.

index=<value> source=<sourcePath.log> host=<value>  | <evalQueryGiven>

vs

index=<sameValue> source=<splunkForwarderPath.log> host=<sameValue> | <evalQueryGiven>

[SourceLogs vs Summary logs from SplunkForwarder] [Last 15mins]

250K events vs 82K events. 

[Time difference] 
-0.023 vs -0.77 at lowest 
-0.894 vs 1.14 at highest

Missing log from source had time definition (example: 06/Mar/2024:10:08:17.894).

I couldn't say if this is a queue problem?