Getting Data In

Splunk stopped following files.

amra
Engager

Splunk stopped following data input files for changes. This happend after I was accessing https://splunk-server:8089/services/* and https://splunk-server:8089/servicesNS/* and trying to reload configuration (after editing times.conf)

Now new log events are indexed only after splunk restart. Stopping splunk takes over 7 minutes. Previous shut down time was about 1 minute.

It's not a space issue. System has 400GB available. Splunk version is 4.1.

amra
Engager

splunk-4.1.5-85165-linux-2.6-x86_64.rpm

0 Karma

MasterOogway
Communicator

Simeon, Can you or someone else expand a bit more on what "blocked=true" is within the metrics.log? I have found multiple entries but the are defined with name=indexqueue and name=parsingqueue.

Is this a concern as well? And if yes, what does it indicated? My indexes have plenty of growth so I don't believe that is a concern. Here is a snippet of the logs:

09-22-2010 11:45:36.500 INFO Metrics - group=queue, name=indexqueue, blocked=true, max_size=1000, filled_count=1, empty_count=4459, current_size=1000, largest_size=1000, smallest_size=0 09-22-2010 11:45:36.500 INFO Metrics - group=queue, name=nullqueue, max_size=1000, filled_count=0, empty_count=1502, current_size=0, largest_size=1, smallest_size=0 09-22-2010 11:45:36.500 INFO Metrics - group=queue, name=parsingqueue, max_size=1000, filled_count=0, empty_count=3648, current_size=0, largest_size=9, smallest_size=0 09-22-2010 11:45:36.500 INFO Metrics - group=queue, name=tcpin_queue, max_size=1000, filled_count=0, empty_count=0, current_size=0, largest_size=0, smallest_size=0

Paul

0 Karma

Simeon
Splunk Employee
Splunk Employee

It sounds like indexing gets blocked after a certain amount of time. Also, Splunk typically does not take 7 minutes to shut down unless it is trying to close a lot of network connections or clean up indexing in some way. You should detail your system/hardware specifications and operating system. My recommendations:

  1. Check the $SPLUNK_HOME/var/log/splunk/metrics.log for "blocked=true". If you have current events that contain this then Splunk is not able to further index.
  2. If the data input is the same file and the header is the same 256 bytes, we are probably ignoring the file and you will need to address this in some way.
  3. It is possible you edited something else (besides times.conf) and that has broken Splunk. You should check for any recent FATAL or ERROR messages in the $SPLUNK_HOME/var/log/splunk/splunkd.log file.
0 Karma

Genti
Splunk Employee
Splunk Employee

what version? 4.1.?

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Accelerate Splunk App Development with AI

  Learn how to automate the repetitive parts of your build cycle.   Watch On-Demand   Replay the Accelerate ...

Splunk Agent Observability: Measuring AI Value with Splunk Tokenomics on AWS

The AI landscape is moving faster than ever, and our latest 2026 State of Observability report provides the ...

Enterprise Security Watch Party: What’s New in 8.8 - Recap

Splunk Community HQ hosted a Splunk Enterprise Security Watch Party: What’s New In ES 8.8 last Thursday. I had ...