Deployment Architecture

Why am I unable to generate a diag from an indexer cluster peer after upgrading to Splunk 6.4.0?

rbal_splunk
Splunk Employee
Splunk Employee

After upgrading to 6.4.0 today, we are unable to generate a diag from an indexer cluster peer:

When I attempt to run a ./splunk diag from my indexers, I receive the following error:

[splunk@ftlxix06 bin]$ splunk diag 

**Apps exist in both the apps and the slave-apps dir: TA-fire_brigade,_cluster 
Duplicate apps will be skipped for app-collection purposes 
An unforeseen error occurred: 
Exception: <type 'exceptions.RuntimeError'>, Value: dictionary changed size during iteration Traceback (most recent call last): 
File "/opt/splunk/lib/python2.7/site-packages/splunk/clilib/cli.py", line 1150, in main parseAndRun(argsList) 
File "/opt/splunk/lib/python2.7/site-packages/splunk/clilib/cli.py", line 977, in parseAndRun retVal = cList.getCmd(command, subCmd).call(argList, fromCLI = True) 
File "/opt/splunk/lib/python2.7/site-packages/splunk/clilib/cli.py", line 280, in call return self.func(args, fromCLI) 
File "/splunk_hotwarm/splunk/lib/python2.7/site-packages/splunk/clilib/control_api.py", line 30, in wrapperFuncreturn func(dictCopy, fromCLI) 
File "/splunk_hotwarm/splunk/lib/python2.7/site-packages/splunk/clilib/info_gather.py", line 3771, in pclMainmain() 
File "/splunk_hotwarm/splunk/lib/python2.7/site-packages/splunk/clilib/info_gather.py", line 3549, in main app_infos = discover_apps() 
File "/splunk_hotwarm/splunk/lib/python2.7/site-packages/splunk/clilib/info_gather.py", line 3445, in discover_apps for key in combined_apps: RuntimeError: dictionary changed size during iteration**
1 Solution

hexx
Splunk Employee
Splunk Employee

This issue is specific to 6.4.0 and is tracked as product defect SPL-118170. This issue will be fixed in an early 6.4.x maintenance release.

In the meantime, there is a simple work-around to prevent this error, which only affects one python script - info_gather.py, which splunk diag relies upon:

  • Download the patched info_gather.py from Splunk's box account
  • Use it to replace file $SPLUNK_HOME/lib/python2.7/site-packages/splunk/clilib/info_gather.py
  • You can now run splunk diag again without issues - no restart required!

For reference, here's the very small diff between the old file and the patched file:

$ diff -u /opt/splunk/lib/python2.7/site-packages/splunk/clilib/info_gather-6.4.0.py /opt/splunk/lib/python2.7/site-packages/splunk/clilib/info_gather.py
--- /opt/splunk/lib/python2.7/site-packages/splunk/clilib/info_gather-6.4.0.py  2016-03-25 20:05:02.000000000 -0700
+++ /opt/splunk/lib/python2.7/site-packages/splunk/clilib/info_gather.py        2016-04-18 14:36:39.700531894 -0700
@@ -3442,7 +3442,7 @@
     combined_apps.update(slave_apps)
     combined_apps.update(pooled_apps)

-    for key in combined_apps:
+    for key in combined_apps.keys():
         if key not in unique_app_set:
             del combined_apps[key]

View solution in original post

hexx
Splunk Employee
Splunk Employee

This issue is specific to 6.4.0 and is tracked as product defect SPL-118170. This issue will be fixed in an early 6.4.x maintenance release.

In the meantime, there is a simple work-around to prevent this error, which only affects one python script - info_gather.py, which splunk diag relies upon:

  • Download the patched info_gather.py from Splunk's box account
  • Use it to replace file $SPLUNK_HOME/lib/python2.7/site-packages/splunk/clilib/info_gather.py
  • You can now run splunk diag again without issues - no restart required!

For reference, here's the very small diff between the old file and the patched file:

$ diff -u /opt/splunk/lib/python2.7/site-packages/splunk/clilib/info_gather-6.4.0.py /opt/splunk/lib/python2.7/site-packages/splunk/clilib/info_gather.py
--- /opt/splunk/lib/python2.7/site-packages/splunk/clilib/info_gather-6.4.0.py  2016-03-25 20:05:02.000000000 -0700
+++ /opt/splunk/lib/python2.7/site-packages/splunk/clilib/info_gather.py        2016-04-18 14:36:39.700531894 -0700
@@ -3442,7 +3442,7 @@
     combined_apps.update(slave_apps)
     combined_apps.update(pooled_apps)

-    for key in combined_apps:
+    for key in combined_apps.keys():
         if key not in unique_app_set:
             del combined_apps[key]

nnmiller
Contributor

Another work around is simply moving, deleting, or renaming the duplicate app.

jrodman
Splunk Employee
Splunk Employee

If for some reason, contractual requirements or similar prevent you from changing a file as part of the splunk install, it is also possible to use the updated script externally, as per the documentation http://docs.splunk.com/Documentation/Splunk/6.4.0/Troubleshooting/Generateadiag#Run_diag_with_defaul...

e.g. splunk cmd python path/to/info_gather.py

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...