Monitoring Splunk

Splunk shows no logs (0 events) on it for some amount of time in a day .) 0 event count on splunk though the monitored logs have data


I have a clustered splunk environment and monitoring in place for quite a few application logs.
Lately , I have been encountering an issue with data collection in Splunk .

For some frame of time everyday(2 to 5 hours) , I do not see any data even though the application server has logs generated.
But for the rest of the day it works just fine .

Universal Forwarders and indexers are working just fine.
This is affecting the dashboards and alerts , as the data is been missed out .

Example log:
2020-02-13T05:01:45.249-0500 INFO 801 | UNIQ_ID=2AB2130 | TRANS_ID=00000170151fda6c-171dce8 | VERSION=18.09 | TYPE=AUDIT| UTC_ENTRY=2020-02-13T10:01:45.178Z | UTC_EXIT=2020-02-13T10:01:45.230Z,"Timestamp":"2020-02-13T10:01:45.062Z","Data":{"rsCommand":"","rsStatus":"executed","pqr":"2020-02-13T09:57:13.000Z","rsStatusReason":"executed","XYZ":"2020-02-13T09:57:29.000Z","rsMinutesRemaining":"6","remoDuration":"10","internTemperature":"12","ABC":"2020-02-13T10:00:20.000Z","Sucction"}}

Can anyone give some insight ,If you have faced or come across this kind of issue.
I suspect Splunk is getting confused with the time format of the actual event and the time and year value format inside the event likeabc,pqr,xyz timestamp in the example log above.. But doesn't help me how to go about and solve this issue.

0 Karma

Ultra Champion
| metadata type=sourcetypes

time picker -> all times
check this results

0 Karma


@to4kawa below is the output
firstTime lastTime recentTime sourcetype totalCount type
822873600 2145916799 1582023453 ABC 200882336 sourcetypes
1576767386 1582088399 1582023523 XYZ 106684712 sourcetypes

0 Karma
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...