Data store is yellow with this message: No data was found in the short term search for the Search History storage. This likely indicates that either the backfill script is not scheduled to run, or some sort of error while running it. If the script shows Validation Success below, you should check the Troubleshooting TSIDX Population dashboard.
errors from internal log script:
09-12-2017 06:28:28.518 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/search_activity/bin/CheckDataStats-search.py" splunklib.binding.HTTPError: HTTP 400 Bad Request -- Invalid latest_time: latest_time must be after earliest_time.
09-12-2017 06:28:28.518 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/search_activity/bin/CheckDataStats-search.py" raise HTTPError(response)
09-12-2017 06:28:28.518 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/search_activity/bin/CheckDataStats-search.py" File "/opt/splunk/etc/apps/search_activity/bin/splunklib/binding.py", line 1110, in request
09-12-2017 06:28:28.518 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/search_activity/bin/CheckDataStats-search.py" return self.request(url, message)
Logs created by script:
[2017-09-12 06:28:28.513045 - id=4201015146 line=395] Just ran query (searchid="1505197707.1655"):
| rest splunk_server=local "/servicesNS/admin/-/search/jobs"| search dispatchState="RUNNING" OR dispatchState="FINALIZING" OR dispatchState="QUEUED" OR dispatchState="PARSING" title!="| rest*" title="FillSearchHistory" OR remoteSearch="info=failed OR info=completed OR info=canceled *total_run_time searchid"
[2017-09-12 06:28:28.513045 - id=4201015146 line=395] Just ran query (searchid="1505197707.1655"):
| rest splunk_server=local "/servicesNS/admin/-/search/jobs"| search dispatchState="RUNNING" OR dispatchState="FINALIZING" OR dispatchState="QUEUED" OR dispatchState="PARSING" title!="| rest*" title="FillSearchHistory" OR remoteSearch="info=failed OR info=completed OR info=canceled *total_run_time searchid"
[2017-09-12 06:28:27.472318 - id=4201015146 line=364] We are running our search over -86400 to 259200. Also:
search_time_earliest: -86400
final_time_earliest: 0
search_time_latest: 259200
maxfinaltime: 1504835471
[2017-09-12 06:28:27.472318 - id=4201015146 line=364] We are running our search over -86400 to 259200. Also:
search_time_earliest: -86400
final_time_earliest: 0
search_time_latest: 259200
maxfinaltime: 1504835471
[2017-09-12 06:28:27.472241 - id=4201015146 line=363] Just ran query (searchid="1505197703.1652"):
| tstats local=t max(_time) as maxstarttime from SA_SearchHistory
| eval range_low=maxstarttime-3600 | eval range_high=maxstarttime+3600 | map search="| tstats local=t max(_time) as maxstarttime max(finaltime) as maxfinaltime from SA_SearchHistory
where earliest=$range_low$ latest=$range_high$| eval maxfinaltime=round(coalesce(maxfinaltime, maxstarttime)-0.5,0) | eval now=now() | eval tsidxlag = now-maxfinaltime"
[2017-09-12 06:28:27.472241 - id=4201015146 line=363] Just ran query (searchid="1505197703.1652"):
| tstats local=t max(_time) as maxstarttime from SA_SearchHistory
| eval range_low=maxstarttime-3600 | eval range_high=maxstarttime+3600 | map search="| tstats local=t max(_time) as maxstarttime max(finaltime) as maxfinaltime from SA_SearchHistory
where earliest=$range_low$ latest=$range_high$| eval maxfinaltime=round(coalesce(maxfinaltime, maxstarttime)-0.5,0) | eval now=now() | eval tsidxlag = now-maxfinaltime"
[2017-09-12 06:28:27.472094 - id=4201015146 line=274] Entering time management logic: 2 - b
[2017-09-12 06:28:27.472094 - id=4201015146 line=274] Entering time management logic: 2 - b
in datastore , clicked nobackfill and waited for almost 8 hours. Now I can able to see the data getting populated for all the dashboard.