Does UF forward missed data(when UF was down),once it starts running?

[Ashwini008]

When UF will be stopped ,data wont be indexed. But once the UF is up and running will it forward the old data/missed data  when UF was down? I wanted to understand if the events/logs present during the downtime of UF are still forwarded to indexers once the UF starts running.

Thank you

[inventsekar]

Hi @Ashwini008 Yes, .. the overall picture... the UF reads a log file and puts the log into a "message queue", this msg queue then transfers the logs, in first in first out style, to the indexer. 

when UF is down, the message queue is still got the logs(it uses some fishbuckets... it keeps the pointers of what it read, what was transferred, etc..

https://www.splunk.com/en_us/blog/tips-and-tricks/what-is-this-fishbucket-thing.html

https://docs.splunk.com/Documentation/Splunk/8.1.0/Data/Usepersistentqueues

one more important idea - there is a feature called "indexer acknowledgement".. indexer and UF does a handshake on reading the logs. so the logs will not be lost during the travel. 

https://docs.splunk.com/Documentation/Forwarder/8.1.0/Forwarder/Protectagainstthelossofin-flightdata

Happy Splunking | Best Regards | Sekar | PS - Karma points appreciated!

[Ashwini008]

Hi @inventsekar  One of the UF on my host was down, CPU and Memory details of that host is not captured  during that down period once the UF started. Why is it so?

[inventsekar]

Hi @Ashwini008 ... i think, when the UF was "down", the UF itself was powered-off,.. so the CPU, memory details will not be available when the UF is down. If i misunderstood your question, please suggest me, thanks. 

[Ashwini008]

@inventsekar sorry if i am not being clear. So the below concept wont be applied when it is powered off ?

"when UF is down, the message queue is still got the logs(it uses some fishbuckets... it keeps the pointers of what it read, what was transferred, etc.."

Please share if you any reference links regarding this issue

[inventsekar]

Hi @Ashwini008 i got your confusion.. 

so, during normal times, the UF's "message queue" gets the CPU/memory/application logs and fwd to indexer. 

lets say the msg queue got 100MB logs in it, and UF is sent 20MB logs to indexer and the UF crashes/powered down. the remaining 80MB will be still inside the message queue and when the UF is powered on, it will send the 80MB. 

but, during the powered off situation, the CPU/memory/application logs will not be generated freshly. when UF powers on, it will generate the new logs and along with the old logs, the new logs also will be sent. hope its clear now. thanks. 

Happy Splunking | Best Regards | Sekar | PS - Karma points appreciated!

[Ashwini008]

@inventsekar Thank you for the brief explanation

"but, during the powered off situation, the CPU/memory/application logs will not be generated freshly. when UF powers on, it will generate the new logs and along with the old logs, the new logs also will be sent. hope"

When the logs are not generated during powered off situation, then When UF powers on ,you mentioned it as along with old logs ,new logs also  will be sent. 

Old logs you are referring to the one which was present before the UF was down/powered off right?

 could you let me know what are you referring to old logs here?

[inventsekar]

Old logs you are referring to the one which was present before the UF was down/powered off right? // 

Yes, exactly.