Internal index data and performance data is missing

[uagraw01]

To investigate the issue of missing data in Splunk for a period of 3-4 hours, where gaps were observed in the _internal index as well as in performance metrics like network and CPU data, But still can't able to find out the potential root cause of data missing in Splunk. Please help me what I need to investigate more to find out the potential root cause of the data gap in Splunk.

Gap into the _internal index data

Network performance data gap is visible

Gap in the CPU performance data

[gcusello]

Hi @uagraw01 ,

in general there can be two potential root causes:

the server is down,

there's a network or server congestion so, the internal Splunk logs have a minor priority than the other logs.

I don't think tha you can find a root cause in _internal, see the server and network logs.

Ciao.

Giuseppe

[PickleRick]

With congestion you would have a drop in throughput but you'd have some values if only from local internal inputs. Here you seem to have no data points whatsoever which means that it's probably an all-in-one installation or the whole splunk infrastructure was down.

[PickleRick]

It looks like service downtime. Especially considering a sudden spike in throughput after a drop - the forwarders were pushing the queued data.

Check your splunkd.log immediately before and after that outage.