Deployment Architecture

What happens to data during a network outage?

zachvida
Path Finder

I was wondering what happens to data from scripts and logs if and when a machine is either so heavily loaded it can't get enough cycles to run the process or machine lost network connectivity.

We are HPC shop, and our machine will sometime become unresponsive when running high load jobs.

Will log data just que up somewhere and get sent to the index eventually?

Tags (2)
1 Solution

araitz
Splunk Employee
Splunk Employee

Splunk will stop seeking forward in log files it is monitoring, and once the connection is restored or resources become available it will pick back up where it left off. Data that is being dropped into spool will queue. Socket inputs will start to block and there is a small queue, but otherwise they will eventually begin to drop, so it depends on the queueing mechanisms upstream and downstream as to how they will handle congestion. If Splunk cannot forward data, scripted input data will not be captured.

View solution in original post

araitz
Splunk Employee
Splunk Employee

Splunk will stop seeking forward in log files it is monitoring, and once the connection is restored or resources become available it will pick back up where it left off. Data that is being dropped into spool will queue. Socket inputs will start to block and there is a small queue, but otherwise they will eventually begin to drop, so it depends on the queueing mechanisms upstream and downstream as to how they will handle congestion. If Splunk cannot forward data, scripted input data will not be captured.

bsanjeeva
Explorer

@araitz , My question is similar to the above but I would like to know what happens to logging in below scenarios when there is an outage. I would like to know if splunk restores logging when the systems are out of outage or does it lose the logs.

1. logs are getting forwarded from an app

2.  synced from an S3 bucket

3. pulled via API

4. data coming through heavy forwarder

Thanks in Advance

0 Karma
Get Updates on the Splunk Community!

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...