Monitoring Splunk

How to search splunkd.log to find tailing status of a file?

Communicator

How can we search splunkd.log effectively to know tailing status of a file and when is the last read?

0 Karma

Communicator

I think this is not the right answer. Almost nothing I can see after some noises filtered.

This is more close.

index=_internal component=WatchedFile

0 Karma

Path Finder

If you are using a heavy forwarder or indexing locally, you can query the _thefishbucket index to get run time information on the tailing of logs: “index=_thefishbucket"

Unfortunately, this isn't available on the univ/light forwarder.

It won't give you the exact timestamp of the last event exactly, but you can see the seek pointer and the last modified date and time. Based on this, you can see what it is doing (waiting for more log lines to be written), or processing a backlog, or stuck due to indexing being blocked and so forth.

More information:
http://blogs.splunk.com/2008/08/14/what-is-this-fishbucket-thing/

0 Karma

Splunk Employee
Splunk Employee

No this information is not logged in splunkd.log
Usually only the event of the first discovery of a file is listed, or when a batch processor finished to read a large file.
If you want the status of a file, use the REST-endpoint on the forwarder :
https://localhost:8089/services/admin/inputstatus/TailingProcessor:FileStatus

or use the file monitoring view in the SOS app on the local instance (if it has an UI)

Also you will never get the latest timestamp from the internal logs
For that you do a real time search on the indexed events, sort by _indextime and look at the most recent _time per source/host.

0 Karma

Communicator

Yes, I just realized SoS's "File Monitor Inputs" page is doing that. I think it's prefect for if using indexer to read logs. But badly, it doesn't work with forwarders. I'm not sure yet.

Behind the scenes an customized command bundled with SoS doing good job to parse TailingProcessor REST response

| tpstatusquery targetserver= namespace="system" owner="nobody" admin/inputstatus/TailingProcessor:FileStatus | table *

0 Karma

SplunkTrust
SplunkTrust

You can use metadata command as well for getting the most recent event timestamp by host/source/sourcetype

Communicator

This is a good start but not enough.
I wish to list all proceeded/processing logs by indexer/forwarder and the best also have the latest tailing timestamp

0 Karma

SplunkTrust
SplunkTrust

For starters, below can give all TailingProcessor activities including errors.

index=_internal component=TailingProcessor

0 Karma