The lookup table 'splunk_instances_info' is invalid - SOS

[gerald_huddlest]

Installed Splunk on Splunk 2.1 on 4.2.1 but when opening app and app is retrieving information about splunk server in a glimpse of your server, it returns "The lookup table 'splunk_instances_info' is invalid" in pop up bar on menu and no fields are populated in a glimpse of your server frame.
Sideview utils 1.2.5 is installed.
have not found any otehr issues with splunk server

[hexx]

This can happen for one of two reasons, depending on the version of Splunk you are running.

If you are running Splunk 4.3.3:

There is a core bug in Splunk 4.3.3 (reference SPL-52862) that causes scheduled search "sos_splunk_instances_info" to fail with an error of this kind:

Encountered an error while reading file '/opt/splunk/var/run/splunk/dispatchtmp/subsearch_1341344801.10_1341344802.1/prereport_0.csv.gz'.

This issue will be fixed in Splunk 4.3.4. SoS 2.2 will have a modified version of the "sos_splunk_instances_info" to overcome this problem until 4.3.4 comes out, but if you want to fix the lookup table manually in the meantime, follow these instructions:

• Go to http[s]://[splunkweb hostname]:[splunkweb port]/en-US/app/sos/flashtimeline
• Run the following search: | serverinfo | stats values(*) AS * | append [entity namespace=None properties/server/license] | eval title=_raw | eval master_uri=if(title=="master_uri",entityContent,"") | eval license_role=case(master_uri=="self","License master",like(master_uri,"http%"),"License slave, reporting to master_uri : \"".master_uri."\"",isnull(master_uri),"n/a") | append [entity namespace=None properties/authentication/authentication] | eval title=_raw | eval auth_method=if(title=="authType",entityContent,"") | eval auth_method=case(isnull(auth_method),"not available",auth_method=="Splunk","Splunk built-in",isnotnull(auth_method),auth_method) | appendcols [entity namespace=None server/info] | appendcols [entity namespace=None server/settings] | search source=serverinfo OR license_role!="" OR auth_method!="" | fields sos_server cpu_count host_fqdn total_phys_mem_gb version build os_name cpu_arch SPLUNK_HOME SPLUNK_DB license_role auth_method | fields - _time _raw | stats values(*) AS * by sos_server | outputlookup createinapp=t splunk_instances_info

This should heal the $SPLUNK_HOME/etc/apps/sos/lookups/splunk_instances_info.csv lookup table. The problem will go away for good once you upgrade to SoS 2.2 or Splunk 4.3.4.

If you are running Splunk 4.2, 4.2.1 or 4.2.2:

This error can be caused by core Splunk bug SPL-40222 which causes lookup files dynamically created by scheduled searches to be always gzipped, even when it's explicitly specified otherwise.

The result is that in Splunk version 4.2, 4.2.1 and 4.2.2 the lookup file $SPLUNK_HOME/etc/apps/sos/lookups/splunk_instances_info.csv.gz is created when the app actually expects splunk_instances_info.csv.

The fix is to upgrade Splunk to 4.2.3 or higher.

If, however, you find yourself unable to upgrade, there is a simple work-around. In $SPLUNK_HOME/etc/apps/sos/local/transforms.conf (create this directory if it doesn't exist), add the following stanza :

[splunk_instances_info]
filename = splunk_instances_info.csv.gz

This should fix the problem without even needing a Splunk restart. Remember to back out this change once you upgrade to 4.2.3 or higher, though!

[hexx]

@rroberts : Sorry for the mistake, the <code> tags I used to wrap the search string still required backslashes to be escaped. I fixed the string and tested a copy/paste of it with success.

[rroberts]

Got an error "malformed eval expected )." Changed to
| eval license_role=case(master_uri=="self","License master",like(master_uri,"http%"),"License slave reporting to master_uri : '.master_uri.'" , isnull(master_uri), "n/a")
Note single quotes around '.master_uri.' Seemed to solve it.

[hexx]

@rroberts : Thank you for reporting this issue. This bug has been identified as SoS bug SUP-457/ Splunk bug SPL-52862. I have amended my answer at the top to include instructions on how to overcome it until SoS 2.2 is released.

[rroberts]

Having the same issue however after running
| get_splunk_instances_info | outputlookup createinapp=t splunk_instances_info

Encountered an error while reading file '/opt/splunk/var/run/splunk/dispatchtmp/subsearch_1341344801.10_1341344802.1/prereport_0.csv.gz'.

[rcovert]

That worked! Thanks!

[hexx]

Ok, so it should have run right after the app's installation but didn't. Let's try this :

- Go to http[s]://[splunkweb_hostname]:[splunkweb_port]/en-US/app/sos/flashtimeline
- Run the following search:

| `get_splunk_instances_info` | outputlookup createinapp=t splunk_instances_info
- Check for the existence of splunk_instances_info.csv in /opt/splunk/etc/apps/sos/lookups/
- Check the functionality of the SoS home view again

[rcovert]

I installed it this morning from the UI.

[hexx]

Ok so this means that the lookup-populating search hasn't run yet. Normally, it should run :

- When the app is installed/upgraded if that is done from the UI.

- Every day at 00:42.

How did you install the app? How long has it been installed?

[rcovert]

Yes, I have refreshed the home view several times and still get the error.

The contents of /opt/splunk/etc/apps/sos/lookups/ is:

cron2basic.csv
version2build.csv

[hexx]

This should not be an issue in 4.3. You must be having a different problem.

- Does this happen even after reloading the home view several times?
- Can you show us the contents of $SPLUNK_HOME/etc/apps/sos/lookups/? Is there a splunk_instances_info.csv file there?

[rcovert]

I am also having this error but I have Splunk 4.3. Should I try the workaround?

[romantercero]

Thanks for the troubleshooting steps! they helped me narrow down a typo in the transforms.conf file

[hexx]

Ah! So, the lookup definition still point to the csv file, not to the csv.gz :
sos splunk_instances_info
sos filename = splunk_instances_info.csv
Are you sure that you have fully followed the instructions of the work-around?
You need to have a transforms.conf file in $SPLUNK_HOME/etc/apps/sos/local with the following content :
[splunk_instances_info]
filename = splunk_instances_info.csv.gz
This change needs to be reflected in the configuration file viewer search that you just ran.

[gerald_huddlest]

results of transforms.conf search:
sos [splunk_instances_info]
system CAN_OPTIMIZE = True
system CLEAN_KEYS = True
system DEFAULT_VALUE =
system DEST_KEY =
system FORMAT =
system KEEP_EMPTY_VALS = False
system LOOKAHEAD = 4096
system MV_ADD = False
system REGEX =
system SOURCE_KEY = _raw
system WRITE_META = False
sos filename = splunk_instances_info.csv
sos_server=DTCI-CBTCPPK01 | stanza=splunk_instances_info | app=sos

[hexx]

Ok, so you do have the lookup table in there, and in .csv.gz format as predicted. In this condition, the work-around should work like a charm... Let's see what's going on with your transforms.conf definition. Please go to the SoS configuration file viewer, pick transforms.conf, search for stanza="splunk_instances_info" and post the resulting stanza here. Alternatively, you can run and provide the output of : splunk cmd btool transforms list splunk_instances_info --debug.

[gerald_huddlest]

Contents of $SPLUNK_HOME/etc/apps/sos/lookups:

cron2basic.csv
splunk_instances_info.csv.gz
version2build.csv

thanks

[hexx]

So it's possible that another bug is part of the problem here. Can you list the contents of $SPLUNK_HOME/etc/apps/sos/lookups for me?

[gerald_huddlest]

Thanks for the response.
Under $SPLUNK_HOME/etc/apps/sos/ there isnt a local directory - there is a transforms.conf in the sos/default - changing the stanza in that had no effect.

Added sos/local directory and copied the default transforms.conf to this directory, amended to .gz but this did not work either.

Upgrading is a longer option, would like to resolve it in 4.2.1 if possible.

thanks