Splunk stopped following files.

[amra]

Splunk stopped following data input files for changes. This happend after I was accessing https://splunk-server:8089/services/* and https://splunk-server:8089/servicesNS/* and trying to reload configuration (after editing times.conf)

Now new log events are indexed only after splunk restart. Stopping splunk takes over 7 minutes. Previous shut down time was about 1 minute.

It's not a space issue. System has 400GB available. Splunk version is 4.1.

[amra]

splunk-4.1.5-85165-linux-2.6-x86_64.rpm

[MasterOogway]

Simeon, Can you or someone else expand a bit more on what "blocked=true" is within the metrics.log? I have found multiple entries but the are defined with name=indexqueue and name=parsingqueue.

Is this a concern as well? And if yes, what does it indicated? My indexes have plenty of growth so I don't believe that is a concern. Here is a snippet of the logs:

09-22-2010 11:45:36.500 INFO Metrics - group=queue, name=indexqueue, blocked=true, max_size=1000, filled_count=1, empty_count=4459, current_size=1000, largest_size=1000, smallest_size=0 09-22-2010 11:45:36.500 INFO Metrics - group=queue, name=nullqueue, max_size=1000, filled_count=0, empty_count=1502, current_size=0, largest_size=1, smallest_size=0 09-22-2010 11:45:36.500 INFO Metrics - group=queue, name=parsingqueue, max_size=1000, filled_count=0, empty_count=3648, current_size=0, largest_size=9, smallest_size=0 09-22-2010 11:45:36.500 INFO Metrics - group=queue, name=tcpin_queue, max_size=1000, filled_count=0, empty_count=0, current_size=0, largest_size=0, smallest_size=0

Paul

[Simeon]

It sounds like indexing gets blocked after a certain amount of time. Also, Splunk typically does not take 7 minutes to shut down unless it is trying to close a lot of network connections or clean up indexing in some way. You should detail your system/hardware specifications and operating system. My recommendations:

1. Check the $SPLUNK_HOME/var/log/splunk/metrics.log for "blocked=true". If you have current events that contain this then Splunk is not able to further index.
2. If the data input is the same file and the header is the same 256 bytes, we are probably ignoring the file and you will need to address this in some way.
3. It is possible you edited something else (besides times.conf) and that has broken Splunk. You should check for any recent FATAL or ERROR messages in the $SPLUNK_HOME/var/log/splunk/splunkd.log file.

[Genti]

what version? 4.1.?