Deployment Architecture

Why am I unable to generate a diag from an indexer cluster peer after upgrading to Splunk 6.4.0?

Splunk Employee
Splunk Employee

After upgrading to 6.4.0 today, we are unable to generate a diag from an indexer cluster peer:

When I attempt to run a ./splunk diag from my indexers, I receive the following error:

[splunk@ftlxix06 bin]$ splunk diag 

**Apps exist in both the apps and the slave-apps dir: TA-fire_brigade,_cluster 
Duplicate apps will be skipped for app-collection purposes 
An unforeseen error occurred: 
Exception: <type 'exceptions.RuntimeError'>, Value: dictionary changed size during iteration Traceback (most recent call last): 
File "/opt/splunk/lib/python2.7/site-packages/splunk/clilib/cli.py", line 1150, in main parseAndRun(argsList) 
File "/opt/splunk/lib/python2.7/site-packages/splunk/clilib/cli.py", line 977, in parseAndRun retVal = cList.getCmd(command, subCmd).call(argList, fromCLI = True) 
File "/opt/splunk/lib/python2.7/site-packages/splunk/clilib/cli.py", line 280, in call return self.func(args, fromCLI) 
File "/splunk_hotwarm/splunk/lib/python2.7/site-packages/splunk/clilib/control_api.py", line 30, in wrapperFuncreturn func(dictCopy, fromCLI) 
File "/splunk_hotwarm/splunk/lib/python2.7/site-packages/splunk/clilib/info_gather.py", line 3771, in pclMainmain() 
File "/splunk_hotwarm/splunk/lib/python2.7/site-packages/splunk/clilib/info_gather.py", line 3549, in main app_infos = discover_apps() 
File "/splunk_hotwarm/splunk/lib/python2.7/site-packages/splunk/clilib/info_gather.py", line 3445, in discover_apps for key in combined_apps: RuntimeError: dictionary changed size during iteration**
1 Solution

Splunk Employee
Splunk Employee

This issue is specific to 6.4.0 and is tracked as product defect SPL-118170. This issue will be fixed in an early 6.4.x maintenance release.

In the meantime, there is a simple work-around to prevent this error, which only affects one python script - info_gather.py, which splunk diag relies upon:

  • Download the patched info_gather.py from Splunk's box account
  • Use it to replace file $SPLUNK_HOME/lib/python2.7/site-packages/splunk/clilib/info_gather.py
  • You can now run splunk diag again without issues - no restart required!

For reference, here's the very small diff between the old file and the patched file:

$ diff -u /opt/splunk/lib/python2.7/site-packages/splunk/clilib/info_gather-6.4.0.py /opt/splunk/lib/python2.7/site-packages/splunk/clilib/info_gather.py
--- /opt/splunk/lib/python2.7/site-packages/splunk/clilib/info_gather-6.4.0.py  2016-03-25 20:05:02.000000000 -0700
+++ /opt/splunk/lib/python2.7/site-packages/splunk/clilib/info_gather.py        2016-04-18 14:36:39.700531894 -0700
@@ -3442,7 +3442,7 @@
     combined_apps.update(slave_apps)
     combined_apps.update(pooled_apps)

-    for key in combined_apps:
+    for key in combined_apps.keys():
         if key not in unique_app_set:
             del combined_apps[key]

View solution in original post

Splunk Employee
Splunk Employee

This issue is specific to 6.4.0 and is tracked as product defect SPL-118170. This issue will be fixed in an early 6.4.x maintenance release.

In the meantime, there is a simple work-around to prevent this error, which only affects one python script - info_gather.py, which splunk diag relies upon:

  • Download the patched info_gather.py from Splunk's box account
  • Use it to replace file $SPLUNK_HOME/lib/python2.7/site-packages/splunk/clilib/info_gather.py
  • You can now run splunk diag again without issues - no restart required!

For reference, here's the very small diff between the old file and the patched file:

$ diff -u /opt/splunk/lib/python2.7/site-packages/splunk/clilib/info_gather-6.4.0.py /opt/splunk/lib/python2.7/site-packages/splunk/clilib/info_gather.py
--- /opt/splunk/lib/python2.7/site-packages/splunk/clilib/info_gather-6.4.0.py  2016-03-25 20:05:02.000000000 -0700
+++ /opt/splunk/lib/python2.7/site-packages/splunk/clilib/info_gather.py        2016-04-18 14:36:39.700531894 -0700
@@ -3442,7 +3442,7 @@
     combined_apps.update(slave_apps)
     combined_apps.update(pooled_apps)

-    for key in combined_apps:
+    for key in combined_apps.keys():
         if key not in unique_app_set:
             del combined_apps[key]

View solution in original post

Contributor

Another work around is simply moving, deleting, or renaming the duplicate app.

Splunk Employee
Splunk Employee

If for some reason, contractual requirements or similar prevent you from changing a file as part of the splunk install, it is also possible to use the updated script externally, as per the documentation http://docs.splunk.com/Documentation/Splunk/6.4.0/Troubleshooting/Generateadiag#Run_diag_with_defaul...

e.g. splunk cmd python path/to/info_gather.py