https://community.splunk.com/t5/Getting-Data-In/Why-am-I-unable-to-search-data-from-Splunk-servers-when-the-time/m-p/205908#M40650 <P>Are you getting the data for recent time ranges like last 15 mins, last 24 hrs etc? Every Splunk instance generates splunkd logs rather frequently so if your Splunk servers (search head/deployment servers etc) are sending data to your Splunk indexers, you'll see the data for these recent time ranges. If no then probably your Splunk servers are not sending their internal data to Splunk at all. For that you need to check outputs.conf on those Splunk servers to see if it exists and if yes, are they referencing your Splunk indexers?</P> Fri, 23 Dec 2016 18:53:08 GMT somesoni2 2016-12-23T18:53:08Z https://community.splunk.com/t5/Getting-Data-In/Why-am-I-unable-to-search-data-from-Splunk-servers-when-the-time/m-p/205905#M40647 <P>Hi All, Can any one guide me why I am unable to fetch the data from <CODE>index=_internal host=splunk1 sourcetype=splunkd source="/opt/splunk/var/log/splunk/splunkd.log"</CODE> when time period is set for last month? This is happening only servers related with the Splunk instances. So kindly help us in troubleshooting this issue. </P> <P>thanks in advance.</P> Fri, 23 Dec 2016 16:49:12 GMT https://community.splunk.com/t5/Getting-Data-In/Why-am-I-unable-to-search-data-from-Splunk-servers-when-the-time/m-p/205905#M40647 Hemnaath 2016-12-23T16:49:12Z https://community.splunk.com/t5/Getting-Data-In/Why-am-I-unable-to-search-data-from-Splunk-servers-when-the-time/m-p/205906#M40648 <P>So if you just search this you get data? </P> <PRE><CODE>index=_internal sourcetype=splunkd source="/opt/splunk/var/log/splunk/splunkd.log" </CODE></PRE> <P>What is the retention period of _internal index in your indexers? Run this query and check</P> <PRE><CODE>| rest /services/data/indexes/_internal | table title splunk_server frozenTimePeriodInSecs | eval RetentionDays=frozenTimePeriodInSecs/86400 | rename splunk_server as Indexer </CODE></PRE> Fri, 23 Dec 2016 17:12:44 GMT https://community.splunk.com/t5/Getting-Data-In/Why-am-I-unable-to-search-data-from-Splunk-servers-when-the-time/m-p/205906#M40648 somesoni2 2016-12-23T17:12:44Z https://community.splunk.com/t5/Getting-Data-In/Why-am-I-unable-to-search-data-from-Splunk-servers-when-the-time/m-p/205907#M40649 <P>thanks somesoni for quick response, Yes I am able to get the data when we search with the query index=_internal sourcetype=splunkd source="/opt/splunk/var/log/splunk/splunkd.log. </P> <P><STRONG>Retention period is set as 30 days</STRONG> </P> <P>but when we search with the time period for last 10 days, we are getting no result found. As per retention we should get the data right .</P> <P>Kindly guide me to trouble shoot this issue. </P> Fri, 23 Dec 2016 17:31:13 GMT https://community.splunk.com/t5/Getting-Data-In/Why-am-I-unable-to-search-data-from-Splunk-servers-when-the-time/m-p/205907#M40649 Hemnaath 2016-12-23T17:31:13Z https://community.splunk.com/t5/Getting-Data-In/Why-am-I-unable-to-search-data-from-Splunk-servers-when-the-time/m-p/205908#M40650 <P>Are you getting the data for recent time ranges like last 15 mins, last 24 hrs etc? Every Splunk instance generates splunkd logs rather frequently so if your Splunk servers (search head/deployment servers etc) are sending data to your Splunk indexers, you'll see the data for these recent time ranges. If no then probably your Splunk servers are not sending their internal data to Splunk at all. For that you need to check outputs.conf on those Splunk servers to see if it exists and if yes, are they referencing your Splunk indexers?</P> Fri, 23 Dec 2016 18:53:08 GMT https://community.splunk.com/t5/Getting-Data-In/Why-am-I-unable-to-search-data-from-Splunk-servers-when-the-time/m-p/205908#M40650 somesoni2 2016-12-23T18:53:08Z https://community.splunk.com/t5/Getting-Data-In/Why-am-I-unable-to-search-data-from-Splunk-servers-when-the-time/m-p/205909#M40651 <P>Are your indexes full? Regardless of the retention period, once the _internal index fills the allocated space, it will remove the oldest data to ensure that it does not exceed that size.</P> Sun, 25 Dec 2016 19:48:01 GMT https://community.splunk.com/t5/Getting-Data-In/Why-am-I-unable-to-search-data-from-Splunk-servers-when-the-time/m-p/205909#M40651 lguinn2 2016-12-25T19:48:01Z https://community.splunk.com/t5/Getting-Data-In/Why-am-I-unable-to-search-data-from-Splunk-servers-when-the-time/m-p/205910#M40652 <P>Hi Somesoni, we are getting data when we keep the duration for 15min or even I could see data for last 7 days but when we set for more that then we are getting no result found. But I need to how to check whether the _internal indexes are full ? as we have 15 splunk instance running in our environment.<BR /> Kindly guide me on this.. </P> <P>Wish you a merry Christmas and Happy New Year. </P> <P>thanks in advance </P> Mon, 26 Dec 2016 09:23:47 GMT https://community.splunk.com/t5/Getting-Data-In/Why-am-I-unable-to-search-data-from-Splunk-servers-when-the-time/m-p/205910#M40652 Hemnaath 2016-12-26T09:23:47Z