Splunk stopped following data input files for changes. This happend after I was accessing https://splunk-server:8089/services/* and https://splunk-server:8089/servicesNS/* and trying to reload configuration (after editing times.conf)
Now new log events are indexed only after splunk restart. Stopping splunk takes over 7 minutes. Previous shut down time was about 1 minute.
It's not a space issue. System has 400GB available. Splunk version is 4.1.
It sounds like indexing gets blocked after a certain amount of time. Also, Splunk typically does not take 7 minutes to shut down unless it is trying to close a lot of network connections or clean up indexing in some way. You should detail your system/hardware specifications and operating system. My recommendations:
Simeon, Can you or someone else expand a bit more on what "blocked=true" is within the metrics.log? I have found multiple entries but the are defined with name=indexqueue and name=parsingqueue.
Is this a concern as well? And if yes, what does it indicated? My indexes have plenty of growth so I don't believe that is a concern. Here is a snippet of the logs:
09-22-2010 11:45:36.500 INFO Metrics - group=queue, name=indexqueue, blocked=true, max_size=1000, filled_count=1, empty_count=4459, current_size=1000, largest_size=1000, smallest_size=0 09-22-2010 11:45:36.500 INFO Metrics - group=queue, name=nullqueue, max_size=1000, filled_count=0, empty_count=1502, current_size=0, largest_size=1, smallest_size=0 09-22-2010 11:45:36.500 INFO Metrics - group=queue, name=parsingqueue, max_size=1000, filled_count=0, empty_count=3648, current_size=0, largest_size=9, smallest_size=0 09-22-2010 11:45:36.500 INFO Metrics - group=queue, name=tcpin_queue, max_size=1000, filled_count=0, empty_count=0, current_size=0, largest_size=0, smallest_size=0