Getting Data In

Why did Splunk restart heavy forwarder?

wwhite12
Path Finder

Got an alert for a HF restarting and trying to find the root cause of unexpected restart. I'm using the search below and the results shown are at the start of the event which led to the "Starting Splunk server daemon (splunkd)... " alert

index=_internal source="/opt/splunk/var/log/splunk/splunkd_st*" host=MYHF

Results:

    4/22/20
    1:14:38.000 PM  
    Checking prerequisites...

    4/22/20
    1:14:38.000 PM  
    Splunk> The IT Search Engine.

    4/22/20
    1:14:38.000 PM  
    splunkd is not running. [FAILED]

    4/22/20
    1:14:34.824 PM  
    2020-04-22 13:14:34.824 -0400 splunkd started (build 6db836e2fb9e) pid=25388

    4/22/20
    1:14:34.000 PM  
    Bypassing local license checks since this instance is configured with a remote license master.

Is there anywhere I could look that could give a more specific reason as to why the HF restarted?

Thanks in advance

0 Karma

PavelP
Motivator

Hello @wwhite12,

among several reasons, it could be the "Forwarder Mangement" which can restart Splunkd when an app is deployed.

wwhite12
Path Finder

The actual log of "Starting splunk server daemon (splunkd)... " also came in the query results but I only included the oldest events as they all happened at the same time but the most recent events were just Splunk preliminary restart procedures

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Persistent Queue at TcpOut — One of Splunk's Most Practical Features

Splunk introduced persistent queueing at the tcpout layer as one of the most practical resilience features in ...

Build resilience for the agentic era: Join us at Splunk Go San Jose

AI is changing how organizations build, operate, and secure digital services. As teams adopt AI agents and ...

Manage Edge Processor pipelines end to end with APIs

This guide uses the Data Management APIs to create an Edge Processor, onboard one instance, deploy a pipeline ...