Getting Data In

How to troubleshoot why indexing of events is slow only for monitored files on the Indexer?

chrisboy68
Contributor

Hi,

I have been banging my head for a while. I have a couple of flat files that are a monitored input directly on the indexer. The events just stop getting to the indexer (I assume because they do not show up in a search), but I can clearly see in the flat file events coming in

[monitor://D:\SrvApps\Splunk\etc\apps\output\metrics.log]
disabled = false
crcSalt = <SOURCE>
index = myindex
sourcetype = metrics
alwaysOpenFile = 1
recursive  = false

Simple inputs.conf above, tried crcSlat and alwaysOpen. Now if I put this monitor on a Forwarder, the events are quickly indexed. I can't see what is going on. I tried S.O.S. and didn't see anything standing out. Also tried a few tips from here, http://wiki.splunk.com/Community:Troubleshooting_Monitor_Inputs

Any suggestions how I can narrow down what Splunk is doing? I'm on Windows, running latest 6.X

Thanks

Chris

0 Karma

yannK
Splunk Employee
Splunk Employee
0 Karma

chrisboy68
Contributor

I'm baffled. Seems like it could be related to any file directly on the indexer. So now I'll just focus on splunk.log
Running the tail process I see:

file position 12955579
file size 12955579
parent $SPLUNK_HOME\var\log\splunk\rpc.log*
percent 100.00
type open file

Seems like it always stays open with subsequent calls to the tail process several minutes later and the file position does not change.

When I search the indexer (index=_internal source=D:\srvapps\splunk\var\log\splunk\splunkd.log) the last event in the indexer was 40 minutes old, but when I look directly at the file on the server, I clearly see events as recent as a few minutes ago.

When I restart Splunkd, evertything works fine for an hour or so, then stops again. Any events from the forwarder, DBX or monitored UNC path files are indexed in a timely manner.

Any where else I should look?

Thanks

Chris

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...