How to troubleshoot why indexing of events is slow only for monitored files on the Indexer?

[chrisboy68]

Hi,

I have been banging my head for a while. I have a couple of flat files that are a monitored input directly on the indexer. The events just stop getting to the indexer (I assume because they do not show up in a search), but I can clearly see in the flat file events coming in

[monitor://D:\SrvApps\Splunk\etc\apps\output\metrics.log]
disabled = false
crcSalt = <SOURCE>
index = myindex
sourcetype = metrics
alwaysOpenFile = 1
recursive  = false

Simple inputs.conf above, tried crcSlat and alwaysOpen. Now if I put this monitor on a Forwarder, the events are quickly indexed. I can't see what is going on. I tried S.O.S. and didn't see anything standing out. Also tried a few tips from here, http://wiki.splunk.com/Community:Troubleshooting_Monitor_Inputs

Any suggestions how I can narrow down what Splunk is doing? I'm on Windows, running latest 6.X

Thanks

Chris

[yannK]

• make sure that your index is not the _internal index, as they may be filtered (see outputs.conf)

• verify on the splunkd.log that the file is not skipped because the CRC of the header is the same than another search.
  see http://docs.splunk.com/Documentation/Splunk/6.2.5/Data/HowLogFileRotationIsHandled

  • Verify on the trailing processor that the file is not skipped. use the management port of the indexer (and an admin user) https://localhost:8089/services/admin/inputstatus/TailingProcessor:FileStatus or export the result and check the xml file curl -k https://localhost:8089/services/admin/inputstatus/TailingProcessor:FileStatus -u admin:changeme > tailing_status.log

[chrisboy68]

I'm baffled. Seems like it could be related to any file directly on the indexer. So now I'll just focus on splunk.log
Running the tail process I see:

file position 12955579
file size 12955579
parent $SPLUNK_HOME\var\log\splunk\rpc.log*
percent 100.00
type open file

Seems like it always stays open with subsequent calls to the tail process several minutes later and the file position does not change.

When I search the indexer (index=_internal source=D:\srvapps\splunk\var\log\splunk\splunkd.log) the last event in the indexer was 40 minutes old, but when I look directly at the file on the server, I clearly see events as recent as a few minutes ago.

When I restart Splunkd, evertything works fine for an hour or so, then stops again. Any events from the forwarder, DBX or monitored UNC path files are indexed in a timely manner.

Any where else I should look?

Thanks

Chris