Deployment Architecture

Splunk keeps indexing after data input is disabled

eirika
Engager

Hi,

I can't get my mind around this issue which is as follows:

We have a syslog-ng which dumps data from 100+ network devices to a log directory which then Splunk is set up to monitor and it works perfectly. The problem is that if i disable the data input in Splunk gui it seems to continue indexing for up to an hour after, but it only does so with a few devices, not all of them.

Could this be that the data format on certain hosts are unknown for Splunk and that it seemingly continues to log, and that the time stamp is being applied at index time such that it seems like it is still indexing while in reality it just running through the queue of events from before i disable the data input?

Kind regards,

Eirik

Tags (2)
0 Karma

tiagofbmm
Influencer

Hello

One possible reason for that is that you are retrieving so much data into your directory that Splunk tcp input queue is overwhelmed really, and is actually delayed indexing your files content.

This scenario makes that it still keeps ingesting data for a while after you disable the input, because it has to keep up with what he knows is in the files.

Do a test of disabling the input at a specific minute X, and wait a few minutes and recheck in Splunk if you have any event later than that X point in time.

0 Karma

tiagofbmm
Influencer

Please let me know if the answer was useful for you. If it was, accept it and upvote. If not, give us more input so we can help you with that

0 Karma

MuS
SplunkTrust
SplunkTrust

Hi eirika, that's possible check your splunkd.log for any DateParserVerbose messages to verify this.

0 Karma

eirika
Engager

Hi MuS,

Gone through the log and there are no traces of any DataParserVerbose entries.

Thanks anyway!

Kind regards,

Eirik

0 Karma

valiquet
Contributor

Hello,

Are you sending logs to indexers or UF? Please share inputs.conf

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...