https://community.splunk.com/t5/Knowledge-Management/Summary-Index-Backfill-Jobs-Stuck/m-p/42641#M389 <P>I've set up an EC2 cluster with 19 indexers and 1 Search Head. I've loaded a large amount of data on the indexers, and I'm now backfilling a summary index. I executed the following command:</P> <PRE><CODE>nohup ./splunk cmd python fill_summary_index.py -app MyApp -name "summary_data_daily" -et -6w@d -lt @d -j 2 -auth admin:mycredentials -owner admin &amp; </CODE></PRE> <P>... 4 hours ago. The job report shows the searches at status <CODE>Running (100%)</CODE>, with runtimes around 1 hour 40 minutes. It hasn't completed the search, and it hasn't started searches for any of the following days. </P> <P>I looked through the _internal logs, and didn't see any signs of why it's happening. </P> <P>Any ideas?</P> Thu, 14 Jul 2011 06:43:16 GMT David 2011-07-14T06:43:16Z https://community.splunk.com/t5/Knowledge-Management/Summary-Index-Backfill-Jobs-Stuck/m-p/42641#M389 <P>I've set up an EC2 cluster with 19 indexers and 1 Search Head. I've loaded a large amount of data on the indexers, and I'm now backfilling a summary index. I executed the following command:</P> <PRE><CODE>nohup ./splunk cmd python fill_summary_index.py -app MyApp -name "summary_data_daily" -et -6w@d -lt @d -j 2 -auth admin:mycredentials -owner admin &amp; </CODE></PRE> <P>... 4 hours ago. The job report shows the searches at status <CODE>Running (100%)</CODE>, with runtimes around 1 hour 40 minutes. It hasn't completed the search, and it hasn't started searches for any of the following days. </P> <P>I looked through the _internal logs, and didn't see any signs of why it's happening. </P> <P>Any ideas?</P> Thu, 14 Jul 2011 06:43:16 GMT https://community.splunk.com/t5/Knowledge-Management/Summary-Index-Backfill-Jobs-Stuck/m-p/42641#M389 David 2011-07-14T06:43:16Z https://community.splunk.com/t5/Knowledge-Management/Summary-Index-Backfill-Jobs-Stuck/m-p/42642#M390 <P>With an overnight of waiting, it turns out the answer is just that the 100% is not necessarily 100% for big jobs. The jobs ended up processing for another two and a half hours after they showed <CODE>Running (100%)</CODE>, and then finally completing. The Job size here was 782 mb, with ~91 million events, so I would guess that's just a facet of big jobs being challenging to finish.</P> Thu, 14 Jul 2011 15:03:55 GMT https://community.splunk.com/t5/Knowledge-Management/Summary-Index-Backfill-Jobs-Stuck/m-p/42642#M390 David 2011-07-14T15:03:55Z