Why am I unable to generate a diag from an indexer cluster peer after upgrading to Splunk 6.4.0?

[rbal_splunk]

After upgrading to 6.4.0 today, we are unable to generate a diag from an indexer cluster peer:

When I attempt to run a ./splunk diag from my indexers, I receive the following error:

[splunk@ftlxix06 bin]$ splunk diag 

**Apps exist in both the apps and the slave-apps dir: TA-fire_brigade,_cluster 
Duplicate apps will be skipped for app-collection purposes 
An unforeseen error occurred: 
Exception: <type 'exceptions.RuntimeError'>, Value: dictionary changed size during iteration Traceback (most recent call last): 
File "/opt/splunk/lib/python2.7/site-packages/splunk/clilib/cli.py", line 1150, in main parseAndRun(argsList) 
File "/opt/splunk/lib/python2.7/site-packages/splunk/clilib/cli.py", line 977, in parseAndRun retVal = cList.getCmd(command, subCmd).call(argList, fromCLI = True) 
File "/opt/splunk/lib/python2.7/site-packages/splunk/clilib/cli.py", line 280, in call return self.func(args, fromCLI) 
File "/splunk_hotwarm/splunk/lib/python2.7/site-packages/splunk/clilib/control_api.py", line 30, in wrapperFuncreturn func(dictCopy, fromCLI) 
File "/splunk_hotwarm/splunk/lib/python2.7/site-packages/splunk/clilib/info_gather.py", line 3771, in pclMainmain() 
File "/splunk_hotwarm/splunk/lib/python2.7/site-packages/splunk/clilib/info_gather.py", line 3549, in main app_infos = discover_apps() 
File "/splunk_hotwarm/splunk/lib/python2.7/site-packages/splunk/clilib/info_gather.py", line 3445, in discover_apps for key in combined_apps: RuntimeError: dictionary changed size during iteration**

[hexx · Accepted solution]

This issue is specific to 6.4.0 and is tracked as product defect SPL-118170. This issue will be fixed in an early 6.4.x maintenance release.

In the meantime, there is a simple work-around to prevent this error, which only affects one python script - info_gather.py, which splunk diag relies upon:

• Download the patched info_gather.py from Splunk's box account
• Use it to replace file $SPLUNK_HOME/lib/python2.7/site-packages/splunk/clilib/info_gather.py
• You can now run splunk diag again without issues - no restart required!

For reference, here's the very small diff between the old file and the patched file:

$ diff -u /opt/splunk/lib/python2.7/site-packages/splunk/clilib/info_gather-6.4.0.py /opt/splunk/lib/python2.7/site-packages/splunk/clilib/info_gather.py
--- /opt/splunk/lib/python2.7/site-packages/splunk/clilib/info_gather-6.4.0.py  2016-03-25 20:05:02.000000000 -0700
+++ /opt/splunk/lib/python2.7/site-packages/splunk/clilib/info_gather.py        2016-04-18 14:36:39.700531894 -0700
@@ -3442,7 +3442,7 @@
     combined_apps.update(slave_apps)
     combined_apps.update(pooled_apps)

-    for key in combined_apps:
+    for key in combined_apps.keys():
         if key not in unique_app_set:
             del combined_apps[key]

[nnmiller]

Another work around is simply moving, deleting, or renaming the duplicate app.

[jrodman]

If for some reason, contractual requirements or similar prevent you from changing a file as part of the splunk install, it is also possible to use the updated script externally, as per the documentation http://docs.splunk.com/Documentation/Splunk/6.4.0/Troubleshooting/Generateadiag#Run_diag_with_defaul...

e.g. splunk cmd python path/to/info_gather.py