Why am I unable to search data from Splunk servers when the time period is set for previous week ?

[Hemnaath]

Hi All, Can any one guide me why I am unable to fetch the data from index=_internal host=splunk1 sourcetype=splunkd source="/opt/splunk/var/log/splunk/splunkd.log" when time period is set for last month? This is happening only servers related with the Splunk instances. So kindly help us in troubleshooting this issue.

thanks in advance.

[Hemnaath]

Hi Somesoni, we are getting data when we keep the duration for 15min or even I could see data for last 7 days but when we set for more that then we are getting no result found. But I need to how to check whether the _internal indexes are full ? as we have 15 splunk instance running in our environment.
Kindly guide me on this..

Wish you a merry Christmas and Happy New Year.

thanks in advance

[lguinn2]

Are your indexes full? Regardless of the retention period, once the _internal index fills the allocated space, it will remove the oldest data to ensure that it does not exceed that size.

[Hemnaath]

thanks somesoni for quick response, Yes I am able to get the data when we search with the query index=_internal sourcetype=splunkd source="/opt/splunk/var/log/splunk/splunkd.log.

Retention period is set as 30 days

but when we search with the time period for last 10 days, we are getting no result found. As per retention we should get the data right .

Kindly guide me to trouble shoot this issue.

[somesoni2]

Are you getting the data for recent time ranges like last 15 mins, last 24 hrs etc? Every Splunk instance generates splunkd logs rather frequently so if your Splunk servers (search head/deployment servers etc) are sending data to your Splunk indexers, you'll see the data for these recent time ranges. If no then probably your Splunk servers are not sending their internal data to Splunk at all. For that you need to check outputs.conf on those Splunk servers to see if it exists and if yes, are they referencing your Splunk indexers?

[somesoni2]

So if you just search this you get data?

index=_internal sourcetype=splunkd source="/opt/splunk/var/log/splunk/splunkd.log" 

What is the retention period of _internal index in your indexers? Run this query and check

| rest /services/data/indexes/_internal | table title splunk_server frozenTimePeriodInSecs | eval RetentionDays=frozenTimePeriodInSecs/86400 | rename splunk_server as Indexer