Monitoring Splunk

How to search splunkd.log to find tailing status of a file?

philip_wong
Communicator

How can we search splunkd.log effectively to know tailing status of a file and when is the last read?

0 Karma

philip_wong
Communicator

I think this is not the right answer. Almost nothing I can see after some noises filtered.

This is more close.

index=_internal component=WatchedFile

0 Karma

jnicholsenernoc
Path Finder

If you are using a heavy forwarder or indexing locally, you can query the _thefishbucket index to get run time information on the tailing of logs: “index=_thefishbucket"

Unfortunately, this isn't available on the univ/light forwarder.

It won't give you the exact timestamp of the last event exactly, but you can see the seek pointer and the last modified date and time. Based on this, you can see what it is doing (waiting for more log lines to be written), or processing a backlog, or stuck due to indexing being blocked and so forth.

More information:
http://blogs.splunk.com/2008/08/14/what-is-this-fishbucket-thing/

0 Karma

yannK
Splunk Employee
Splunk Employee

No this information is not logged in splunkd.log
Usually only the event of the first discovery of a file is listed, or when a batch processor finished to read a large file.
If you want the status of a file, use the REST-endpoint on the forwarder :
https://localhost:8089/services/admin/inputstatus/TailingProcessor:FileStatus

or use the file monitoring view in the SOS app on the local instance (if it has an UI)

Also you will never get the latest timestamp from the internal logs
For that you do a real time search on the indexed events, sort by _indextime and look at the most recent _time per source/host.

0 Karma

philip_wong
Communicator

Yes, I just realized SoS's "File Monitor Inputs" page is doing that. I think it's prefect for if using indexer to read logs. But badly, it doesn't work with forwarders. I'm not sure yet.

Behind the scenes an customized command bundled with SoS doing good job to parse TailingProcessor REST response

| tpstatusquery targetserver= namespace="system" owner="nobody" admin/inputstatus/TailingProcessor:FileStatus | table *

0 Karma

somesoni2
Revered Legend

You can use metadata command as well for getting the most recent event timestamp by host/source/sourcetype

philip_wong
Communicator

This is a good start but not enough.
I wish to list all proceeded/processing logs by indexer/forwarder and the best also have the latest tailing timestamp

0 Karma

somesoni2
Revered Legend

For starters, below can give all TailingProcessor activities including errors.

index=_internal component=TailingProcessor

0 Karma
Get Updates on the Splunk Community!

New Case Study Shows the Value of Partnering with Splunk Academic Alliance

The University of Nevada, Las Vegas (UNLV) is another premier research institution helping to shape the next ...

How to Monitor Google Kubernetes Engine (GKE)

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about ...

Index This | How can you make 45 using only 4?

October 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...