topic Re: Why am I unable to generate a diag from an indexer cluster peer after upgrading to Splunk 6.4.0? in Deployment Architecture

Why am I unable to generate a diag from an indexer cluster peer after upgrading to Splunk 6.4.0?

rbal_splunk — Mon, 18 Apr 2016 21:26:04 GMT

After upgrading to 6.4.0 today, we are unable to generate a diag from an indexer cluster peer:

When I attempt to run a ./splunk diag from my indexers, I receive the following error:

[splunk@ftlxix06 bin]$ splunk diag 

**Apps exist in both the apps and the slave-apps dir: TA-fire_brigade,_cluster 
Duplicate apps will be skipped for app-collection purposes 
An unforeseen error occurred: 
Exception: <type 'exceptions.RuntimeError'>, Value: dictionary changed size during iteration Traceback (most recent call last): 
File "/opt/splunk/lib/python2.7/site-packages/splunk/clilib/cli.py", line 1150, in main parseAndRun(argsList) 
File "/opt/splunk/lib/python2.7/site-packages/splunk/clilib/cli.py", line 977, in parseAndRun retVal = cList.getCmd(command, subCmd).call(argList, fromCLI = True) 
File "/opt/splunk/lib/python2.7/site-packages/splunk/clilib/cli.py", line 280, in call return self.func(args, fromCLI) 
File "/splunk_hotwarm/splunk/lib/python2.7/site-packages/splunk/clilib/control_api.py", line 30, in wrapperFuncreturn func(dictCopy, fromCLI) 
File "/splunk_hotwarm/splunk/lib/python2.7/site-packages/splunk/clilib/info_gather.py", line 3771, in pclMainmain() 
File "/splunk_hotwarm/splunk/lib/python2.7/site-packages/splunk/clilib/info_gather.py", line 3549, in main app_infos = discover_apps() 
File "/splunk_hotwarm/splunk/lib/python2.7/site-packages/splunk/clilib/info_gather.py", line 3445, in discover_apps for key in combined_apps: RuntimeError: dictionary changed size during iteration**

Re: Why am I unable to generate a diag from an indexer cluster peer after upgrading to Splunk 6.4.0?

hexx — Mon, 18 Apr 2016 21:46:09 GMT

This issue is specific to 6.4.0 and is tracked as product defect SPL-118170. This issue will be fixed in an early 6.4.x maintenance release.

In the meantime, there is a simple work-around to prevent this error, which only affects one python script - info_gather.py, which splunk diag relies upon:

Download the patched info_gather.py from Splunk's box account
Use it to replace file $SPLUNK_HOME/lib/python2.7/site-packages/splunk/clilib/info_gather.py
You can now run splunk diag again without issues - no restart required!

For reference, here's the very small diff between the old file and the patched file:

$ diff -u /opt/splunk/lib/python2.7/site-packages/splunk/clilib/info_gather-6.4.0.py /opt/splunk/lib/python2.7/site-packages/splunk/clilib/info_gather.py
--- /opt/splunk/lib/python2.7/site-packages/splunk/clilib/info_gather-6.4.0.py  2016-03-25 20:05:02.000000000 -0700
+++ /opt/splunk/lib/python2.7/site-packages/splunk/clilib/info_gather.py        2016-04-18 14:36:39.700531894 -0700
@@ -3442,7 +3442,7 @@
     combined_apps.update(slave_apps)
     combined_apps.update(pooled_apps)

-    for key in combined_apps:
+    for key in combined_apps.keys():
         if key not in unique_app_set:
             del combined_apps[key]

Re: Why am I unable to generate a diag from an indexer cluster peer after upgrading to Splunk 6.4.0?

jrodman — Mon, 18 Apr 2016 21:52:11 GMT

If for some reason, contractual requirements or similar prevent you from changing a file as part of the splunk install, it is also possible to use the updated script externally, as per the documentation http://docs.splunk.com/Documentation/Splunk/6.4.0/Troubleshooting/Generateadiag#Run_diag_with_default_settings

e.g. splunk cmd python path/to/info_gather.py

Re: Why am I unable to generate a diag from an indexer cluster peer after upgrading to Splunk 6.4.0?

nnmiller — Mon, 18 Apr 2016 21:52:47 GMT

Another work around is simply moving, deleting, or renaming the duplicate app.