topic Re: Internal index data and performance data is missing in Splunk Search

Internal index data and performance data is missing

uagraw01 — Thu, 10 Oct 2024 17:27:29 GMT

To investigate the issue of missing data in Splunk for a period of 3-4 hours, where gaps were observed in the _internal index as well as in performance metrics like network and CPU data, But still can't able to find out the potential root cause of data missing in Splunk. Please help me what I need to investigate more to find out the potential root cause of the data gap in Splunk.

Gap into the _internal index data

Network performance data gap is visible

Gap in the CPU performance data

Re: Internal index data and performance data is missing

PickleRick — Thu, 10 Oct 2024 20:33:24 GMT

It looks like service downtime. Especially considering a sudden spike in throughput after a drop - the forwarders were pushing the queued data.

Check your splunkd.log immediately before and after that outage.

Re: Internal index data and performance data is missing

gcusello — Fri, 11 Oct 2024 06:27:46 GMT

Hi @uagraw01 ,

in general there can be two potential root causes:

the server is down,

there's a network or server congestion so, the internal Splunk logs have a minor priority than the other logs.

I don't think tha you can find a root cause in _internal, see the server and network logs.

Ciao.

Giuseppe

Re: Internal index data and performance data is missing

PickleRick — Fri, 11 Oct 2024 06:56:52 GMT

With congestion you would have a drop in throughput but you'd have some values if only from local internal inputs. Here you seem to have no data points whatsoever which means that it's probably an all-in-one installation or the whole splunk infrastructure was down.