Getting Data In

Is there a way to investigate why the data ingest has changed dramatically?

So76
Explorer

The pan logs ingested decreased significantly and nothing should have changed from the syslog point of view. Is there a way to investigate why the data ingest has changed dramatically? Attached is a screen of prior and after the drop in pan logs ingestion

 

I ran this command

| tstats prestats=t count WHERE index=pan by host _time span=1h | timechart partial=f span=1h count by host limit=0

Prior to drop in pan log ingestion

So76_0-1663876420099.png

 

After the drop in pan log ingsetion

 

So76_1-1663876534078.png

 

Can anyone help with a solution to this sudden drop in pan log ingestion. Since the it hasn't come up to the previous level. Solution will be appreciated

Labels (1)
0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

1st you need to understand what are those events which amount have decreased. Is there some special type which have vanished or are all types decreased equally? Based on that you could have better understanding what has happened and where.

You didn't say how you are collecting those events. Have you a separate syslog based service which receive those and then those are collected by splunk or are you sending those directly to splunk UF/HF/indexer? If last one then it's not a best practices and it shouldn't use ever on production!

r. Ismo

0 Karma

shivanshu1593
Builder

Are you using a syslog service like syslog-ng/rsyslogd or Splunk TCP/UDP? If its syslog-ng/rsyslogd, you may want to look in /var/log/messages and see if there's any error that the syslog service is encountering. If its Splunk TCP/UDP using a HF, then it can be a bunch of reasons behind the drop. A usual quick fix is restarting the splunk service on the HF. You can also try restarting the syslog-ng/rsyslogd service and see if that fixes the issue. If you are using a linux server as a syslog server and forwarding via a UF, then you may want to consider the queues utilization, especially the tcpout queue.

One more thing to consider is UDP. If there's a network latency/issues, the packet may get lost during the transit. More can be commented on this by me/other folks once you clarify a bit more about how are you onboarding the data.

Thank you,
Shiv
###If you found the answer helpful, kindly consider upvoting/accepting it as the answer as it helps other Splunkers find the solutions to similar issues###
0 Karma
Get Updates on the Splunk Community!

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...

What's New in Splunk Cloud Platform 9.0.2208?!

Howdy!  We are happy to share the newest updates in Splunk Cloud Platform 9.0.2208! Analysts can benefit ...

Admin Console: A Single, Unified Interface for All Your Cloud Admin Needs

WATCH NOWJoin us to learn how the admin console can save you time and give you more control over the Splunk® ...