Is there a way to investigate why the data ingest has changed dramatically?

[So76]

The pan logs ingested decreased significantly and nothing should have changed from the syslog point of view. Is there a way to investigate why the data ingest has changed dramatically? Attached is a screen of prior and after the drop in pan logs ingestion

I ran this command

| tstats prestats=t count WHERE index=pan by host _time span=1h | timechart partial=f span=1h count by host limit=0

Prior to drop in pan log ingestion

After the drop in pan log ingsetion

Can anyone help with a solution to this sudden drop in pan log ingestion. Since the it hasn't come up to the previous level. Solution will be appreciated

[isoutamo]

Hi

1st you need to understand what are those events which amount have decreased. Is there some special type which have vanished or are all types decreased equally? Based on that you could have better understanding what has happened and where.

You didn't say how you are collecting those events. Have you a separate syslog based service which receive those and then those are collected by splunk or are you sending those directly to splunk UF/HF/indexer? If last one then it's not a best practices and it shouldn't use ever on production!

r. Ismo

[shivanshu1593]

Are you using a syslog service like syslog-ng/rsyslogd or Splunk TCP/UDP? If its syslog-ng/rsyslogd, you may want to look in /var/log/messages and see if there's any error that the syslog service is encountering. If its Splunk TCP/UDP using a HF, then it can be a bunch of reasons behind the drop. A usual quick fix is restarting the splunk service on the HF. You can also try restarting the syslog-ng/rsyslogd service and see if that fixes the issue. If you are using a linux server as a syslog server and forwarding via a UF, then you may want to consider the queues utilization, especially the tcpout queue.

One more thing to consider is UDP. If there's a network latency/issues, the packet may get lost during the transit. More can be commented on this by me/other folks once you clarify a bit more about how are you onboarding the data.