Getting Data In

Splunk stopped following files.

amra
Engager

Splunk stopped following data input files for changes. This happend after I was accessing https://splunk-server:8089/services/* and https://splunk-server:8089/servicesNS/* and trying to reload configuration (after editing times.conf)

Now new log events are indexed only after splunk restart. Stopping splunk takes over 7 minutes. Previous shut down time was about 1 minute.

It's not a space issue. System has 400GB available. Splunk version is 4.1.

amra
Engager

splunk-4.1.5-85165-linux-2.6-x86_64.rpm

0 Karma

MasterOogway
Communicator

Simeon, Can you or someone else expand a bit more on what "blocked=true" is within the metrics.log? I have found multiple entries but the are defined with name=indexqueue and name=parsingqueue.

Is this a concern as well? And if yes, what does it indicated? My indexes have plenty of growth so I don't believe that is a concern. Here is a snippet of the logs:

09-22-2010 11:45:36.500 INFO Metrics - group=queue, name=indexqueue, blocked=true, max_size=1000, filled_count=1, empty_count=4459, current_size=1000, largest_size=1000, smallest_size=0 09-22-2010 11:45:36.500 INFO Metrics - group=queue, name=nullqueue, max_size=1000, filled_count=0, empty_count=1502, current_size=0, largest_size=1, smallest_size=0 09-22-2010 11:45:36.500 INFO Metrics - group=queue, name=parsingqueue, max_size=1000, filled_count=0, empty_count=3648, current_size=0, largest_size=9, smallest_size=0 09-22-2010 11:45:36.500 INFO Metrics - group=queue, name=tcpin_queue, max_size=1000, filled_count=0, empty_count=0, current_size=0, largest_size=0, smallest_size=0

Paul

0 Karma

Simeon
Splunk Employee
Splunk Employee

It sounds like indexing gets blocked after a certain amount of time. Also, Splunk typically does not take 7 minutes to shut down unless it is trying to close a lot of network connections or clean up indexing in some way. You should detail your system/hardware specifications and operating system. My recommendations:

  1. Check the $SPLUNK_HOME/var/log/splunk/metrics.log for "blocked=true". If you have current events that contain this then Splunk is not able to further index.
  2. If the data input is the same file and the header is the same 256 bytes, we are probably ignoring the file and you will need to address this in some way.
  3. It is possible you edited something else (besides times.conf) and that has broken Splunk. You should check for any recent FATAL or ERROR messages in the $SPLUNK_HOME/var/log/splunk/splunkd.log file.
0 Karma

Genti
Splunk Employee
Splunk Employee

what version? 4.1.?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...