Unable to initialize modular input "mscs_storage_table" defined in the app "Splunk_TA_microsoft-cloudservices"

[lukasmecir]

Hello,
I have a problem with Splunk Add-on for Microsoft Cloud Services. It is installed on Heavy Forwarder in a distributed environment and is used to get data from MS Azure. Everything was fine till yesterday - after a restart of Heavy Forwarder I got this message:

"Unable to initialize modular input "mscs_storage_table" defined in the app "Splunk_TA_microsoft-cloudservices": Introspecting scheme=mscs_storage_table: script running failed (exited with code 1).."

I ran command:

splunk cmd splunkd print-modinput-config --debug mscs_storage_table

and got this output (just interesting part):

Found scheme="mscs_storage_table".
Locating script for scheme="mscs_storage_table"...
No regular file="/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/linux_x86_64/bin/mscs_storage_table.sh".
No regular file="/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/linux_x86_64/bin/mscs_storage_table.py".
No regular file="/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/linux_x86_64/bin/mscs_storage_table.js".
No regular file="/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/linux_x86_64/bin/mscs_storage_table".
No script found in dir="/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/linux_x86_64/bin"
No regular file="/opt/splunk/etc/slave-apps/Splunk_TA_microsoft-cloudservices/linux_x86_64/bin/mscs_storage_table.sh".
No regular file="/opt/splunk/etc/slave-apps/Splunk_TA_microsoft-cloudservices/linux_x86_64/bin/mscs_storage_table.py".
No regular file="/opt/splunk/etc/slave-apps/Splunk_TA_microsoft-cloudservices/linux_x86_64/bin/mscs_storage_table.js".
No regular file="/opt/splunk/etc/slave-apps/Splunk_TA_microsoft-cloudservices/linux_x86_64/bin/mscs_storage_table".
No script found in dir="/opt/splunk/etc/slave-apps/Splunk_TA_microsoft-cloudservices/linux_x86_64/bin"
No regular file="/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/mscs_storage_table.sh".
Found script "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/mscs_storage_table.py" to handle scheme "mscs_storage_table".
Ignoring parameter "index" from the spec file: it is an internal arg that gets handled in a special way.
Ignoring parameter "sourcetype" from the spec file: it is an internal arg that gets handled in a special way.
stderr Introspecting scheme=mscs_storage_table:  Traceback (most recent call last):
stderr Introspecting scheme=mscs_storage_table:    File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/mscs_storage_table.py", line 7, in module
stderr Introspecting scheme=mscs_storage_table:      import splunktaucclib.data_collection.ta_mod_input as ta_input
stderr Introspecting scheme=mscs_storage_table:    File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/splunktaucclib/data_collection/ta_mod_input.py", line 15, in module
stderr Introspecting scheme=mscs_storage_table:      from splunktaucclib.data_collection import ta_checkpoint_manager as cpmgr
stderr Introspecting scheme=mscs_storage_table:  ImportError: cannot import name ta_checkpoint_manager
Introspecting scheme=mscs_storage_table: script running failed (exited with code 1).
Unable to initialize modular input "mscs_storage_table" defined in the app "Splunk_TA_microsoft-cloudservices": Introspecting scheme=mscs_storage_table: script running failed (exited with code 1)..

Unfortunately I have very little experience with modular inputs and I had never been facing this kind of problem.
Could someone please help me? It would be higly appreciated. Thank you very much in advance.
Best regards
Lukas Mecir

[sean_wong]

Hi,

Can you described more about "uploading and installing new add-on"?
I tried to uploaded and re-install Splunk_TA_microsoft-cloudservices but the problem is still occurs. Can you advise how to fix it? Thanks.

[KevinMurray]

Chances are that some python scripts have "invalid syntax" errors.

Check your splunkd logs for "mscs_storage" ERROR events.

You may find the scripts:

/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/mscs_storage_blob.py

/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/mscs_storage_table.py

have this error:

print(hlp % sys.argv[0], file=sys.stderr) <-- the equal sign is INVALID!!!!!

During my conversion to Splunk 8, a lot of these problem scripts are appearing although they are supposed to be Splunk 8 compatible. 

[KevinMurray]

Splunkd logs may have this error:

raise ImportError('Unable to load system certificate authority files')

ImportError: Unable to load system certificate authority files

looks like it is being generated by this script:

/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/ca_certs_locater.py

[plaftaric]

Switching python from 3 to 2 (as recommended by bpaul_splunk) did not work for me (Using latest Splunk 8.0.5).

I still get the same error as originally reported by lukasmecir, and now by KevinMurray.

[vanvan]

Splunk 8.x and Python 2.x is supposed to be a "no go"

[Armando]

Does your Splunk box where the input lives have Python 2 installed? I had to manually install it on my CentOS 8 box as the default for that distro is Python 3. Also, did you make any other changes besides what was outlined? If you did, I'd recommend removing and reinstalling the MSCS app and rebooting Splunk after making the Python changes. That's what ended up working for me.

[plaftaric]

Does your Splunk box where the input lives have Python 2 installed?

Yep.

I have CentOS Linux release 7.7, and Python 2 is already here

Also, did you make any other changes besides what was outlined?

Nope.

I  tried again, clean install no luck. Downgrade to python 2 no luck.

Thanks anyway.

[KevinMurray]

I tried using both versions of python.  That is how I produced the top 2 entries in this communication stream. 

My back end for Splunk is SUSE 15 SP1. 

I wonder if there are components of python, other libraries, etc.  that are needed to support Splunk apps and add-ons that are not present in every Linux distro that the developer may have in their environment?

It would be nice if the dependencies were listed in the app/add-on entries in splunkbase to assist in troubleshooting.

Thanks to all that have replied to this stream so far, I hope we find a solution soon!

[tomasmoser · Accepted solution]

Hi,

This issue was fixed. We found out that we had broken add-on Splunk TA for MS Cloud Services. Lot's of files that were supposed to be present in a Gitlab repository for this add-on were missing. Uploading and installing new add-on fixed all issues.

Tomas

[Armando]

Would you be able to be more specific with your solution (version, source of app package, etc.)? I'm having the same issue described by OP as of 8/4/2020. I've tried installing from Splunk Web and by manually copying the app folder to the appropriate Splunk directory. Neither one of those worked.

[bpaul_splunk]

I suspect that may be an issue with Python compatibility.  Try forcing the App to use Python 2 by doing the following.

1. Make a new directory called “local” in $SPLUNK_HOME/etc/apps/Splunk_TA_microsoft_cloudservices.
2. Copy the $SPLUNK_HOME/etc/apps/Splunk_TA_microsoft_cloudservices/default/inputs.conf and $SPLUNK_HOME/etc/apps/Splunk_TA_microsoft_cloudservices/default/restmap.conf files to the new $SPLUNK_HOME/etc/apps/Splunk_TA_microsoft_cloudservices/local directory.
3. Edit the inputs.conf and restmap.conf files in the $SPLUNK_HOME/etc/apps/Splunk_TA_microsoft_cloudservices/local directory and change “python3” to “python2”.
4. Restart Splunk.

If that works, you can remove the changes in the future by deleting the $SPLUNK_HOME/etc/apps/Splunk_TA_microsoft_cloudservices/local directory when the compatibility issue is corrected.

[louismai]

This error is raised in version 4.1.3 too.

I have changed the inputs.conf and restmap.conf to python2, but it doesn't work

Splunk: 7.7.3

Python: 2.7

[Dave1]

This worked for me as well.  Thanks for posting this! This is really frustrating trying to figure it out.  Knowing this workaround is frustrating because I'm using the unmodified Splunk docker image.  

[Armando]

Yeah. I had to manually install Python2 because CentOS 8 doesn't appear to have it by default. Other than that, config seems to be working as intended. I was able to bring in storage table data containing Windows Event logs from VMs. I'm currently working on formatting that data to fit the syntax that the Windows TA uses to parse fields out, etc. 

[Armando]

I tried what you said and so far so good! I haven't really had a chance to create the necessary configuration on the Azure side, but the error within Splunk cleared. I'll report back tomorrow once I've had a chance to do the Azure end of the deal. Thanks!

[plaftaric]

What version is the "new" add-on?  Currently latest 4.0.1 reports the same problem.

[amyers16]

I am in the same boat with this one. Anyone have suggestions?