Getting Data In

How to troubleshoot why indexing of events is slow only for monitored files on the Indexer?

chrisboy68
Contributor

Hi,

I have been banging my head for a while. I have a couple of flat files that are a monitored input directly on the indexer. The events just stop getting to the indexer (I assume because they do not show up in a search), but I can clearly see in the flat file events coming in

[monitor://D:\SrvApps\Splunk\etc\apps\output\metrics.log]
disabled = false
crcSalt = <SOURCE>
index = myindex
sourcetype = metrics
alwaysOpenFile = 1
recursive  = false

Simple inputs.conf above, tried crcSlat and alwaysOpen. Now if I put this monitor on a Forwarder, the events are quickly indexed. I can't see what is going on. I tried S.O.S. and didn't see anything standing out. Also tried a few tips from here, http://wiki.splunk.com/Community:Troubleshooting_Monitor_Inputs

Any suggestions how I can narrow down what Splunk is doing? I'm on Windows, running latest 6.X

Thanks

Chris

0 Karma

yannK
Splunk Employee
Splunk Employee
0 Karma

chrisboy68
Contributor

I'm baffled. Seems like it could be related to any file directly on the indexer. So now I'll just focus on splunk.log
Running the tail process I see:

file position 12955579
file size 12955579
parent $SPLUNK_HOME\var\log\splunk\rpc.log*
percent 100.00
type open file

Seems like it always stays open with subsequent calls to the tail process several minutes later and the file position does not change.

When I search the indexer (index=_internal source=D:\srvapps\splunk\var\log\splunk\splunkd.log) the last event in the indexer was 40 minutes old, but when I look directly at the file on the server, I clearly see events as recent as a few minutes ago.

When I restart Splunkd, evertything works fine for an hour or so, then stops again. Any events from the forwarder, DBX or monitored UNC path files are indexed in a timely manner.

Any where else I should look?

Thanks

Chris

0 Karma
Get Updates on the Splunk Community!

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...

Observability protocols to know about

Observability protocols define the specifications or formats for collecting, encoding, transporting, and ...

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...