How to tell if a universal-forwarder is "caught up"

[cssean]

We have universal forwarders set up on some machines which are part of an auto-scale group in EC2. In some cases the group scales up during peak load. These auto-scale events are somewhat rare, but often times leave us with more instances running than we need once things simmer down.

To combat this, we are devising a way to terminate instances during our low load times. We can terminate the application that is writing the logs and tell Amazon to terminate the instance. But we need to be sure that the forwarder is "caught up", meaning it has reached the end of each of its monitored files.

Is there a way to tell (other than tcpdump or similar) whether the forwarder is still doing anything? This would need to be a scriptable solution so no human intervention is required.

[MuS · Accepted solution]

Hi cssean,

you can use the REST end point

 /services/admin/inputstatus/TailingProcessor:FileStatus

to track if a universal forwarder is reading files monitored or completed sending events.

In the end point you can find information about "open file", and others showing "finished reading".

Some details about the endpoint information, when the percent is 100% :

"finished reading" means that the file has been read and forwarded till the end.

"open file" means the same, but in addition the handle on the file is still open (because it has been less than 3 seconds, or because it is being 'tailed', or the file has just being reopen for any update or rotation).

Splunk will monitor every file, because Splunk assumes that a new event can be added to any file.

hope this helps...

cheers, MuS

[cssean]

@MuS: I have implemented this solution but I seem to have a problem. Even after the files are no longer written to (multiple calls result in the same "file size" and "file location" values), the "type" never changes to "finished reading". Am I misunderstanding something here?

[sowings]

Note that attempting to access this endpoint from somewhere other than via loopback on the forwarder requires that you change the password from the default credentials OR allow the default credentials in remotely (set in server.conf's "allowRemoteLogin").

[MuS]

You're welcome 🙂

[cssean]

This is exactly what I needed. Thanks for the endpoint and explanation.