All Apps and Add-ons

CrowdStrike app fails: Fail to decrypt the encrypted credential information - cannot concatenate 'str' and 'NoneType' objects'.

CMSchelin
Path Finder

I installed the app CrowdStrike Falcon Intelligence Add-on on our Splunk heavy forwarder. I attempted to configure it, but the configure page doesn't load at all. When I check the browser's console, I see:

External handler failed with code '1' and output: 'REST ERROR[1021]: Fail to decrypt the encrypted credential information - cannot concatenate 'str' and 'NoneType' objects'.  See splunkd.log for stderr output.

From splunkd.log:

06-01-2020 11:46:55.999 +0000 ERROR AdminManagerExternal - Unexpected error "<class 'splunktaucclib.rest_handler.error.RestError'>" from python handler: "REST Error [500]: Internal Server Error -- Traceback (most recent call last):
  File "/opt/splunk/etc/apps/TA-crowdstrike_falcon_intel/bin/ta_crowdstrike_falcon_intel/splunktaucclib/rest_handler/handler.py", line 113, in wrapper
    for name, data, acl in meth(self, *args, **kwargs):
  File "/opt/splunk/etc/apps/TA-crowdstrike_falcon_intel/bin/ta_crowdstrike_falcon_intel/splunktaucclib/rest_handler/handler.py", line 299, in _format_response
    masked = self.rest_credentials.decrypt_for_get(name, data)
  File "/opt/splunk/etc/apps/TA-crowdstrike_falcon_intel/bin/ta_crowdstrike_falcon_intel/splunktaucclib/rest_handler/credentials.py", line 184, in decrypt_for_get
    clear_password = self._get(name)
  File "/opt/splunk/etc/apps/TA-crowdstrike_falcon_intel/bin/ta_crowdstrike_falcon_intel/splunktaucclib/rest_handler/credentials.py", line 389, in _get
    string = mgr.get_password(user=context.username())
  File "/opt/splunk/etc/apps/TA-crowdstrike_falcon_intel/bin/ta_crowdstrike_falcon_intel/solnlib/utils.py", line 154, in wrapper
    return func(*args, **kwargs)
  File "/opt/splunk/etc/apps/TA-crowdstrike_falcon_intel/bin/ta_crowdstrike_falcon_intel/solnlib/credentials.py", line 118, in get_password
    all_passwords = self._get_all_passwords()
  File "/opt/splunk/etc/apps/TA-crowdstrike_falcon_intel/bin/ta_crowdstrike_falcon_intel/solnlib/utils.py", line 154, in wrapper
    return func(*args, **kwargs)
  File "/opt/splunk/etc/apps/TA-crowdstrike_falcon_intel/bin/ta_crowdstrike_falcon_intel/solnlib/credentials.py", line 272, in _get_all_passwords
    clear_password += field_clear[index]
TypeError: cannot concatenate 'str' and 'NoneType' objects
".  See splunkd.log for more details.

I tried installing the app on my local trial version of Splunk Enterprise, and the configure page loads, and I'm able to add the streaming API key and secret successfully.

I tried being hacky and copying my local passwords.conf file onto the heavy forwarder server in the same path/location, and making sure the file permissions were the same, to no avail. The config page still doesn't load, and the app still isn't configured.

What am I missing?

(Updated: My bad, there are multiple CrowdStrike issues.)

Labels (2)
0 Karma

tsullivan06
Explorer

CMSchelin,

The CrowdStrike TAs must be deployed locally as opposed to through a deployment server. This is in part because the credentials are stored locally - which is why the work around you attempted did not work. 

The API Secret value is stored in the the KV store and is encrypted. This is to prevent the exact scenario that you are attempting to do - simply copy the passwords.conf file over. If that was something that was possible then anyone could pull that file, drop it into a Splunk system and collect your data - attacker, competitor, former employee, etc. The encryption keys from the original Splunk system are what's needed to decrypt that information.  

Thanks
Tim

davidveuve
Engager

For anyone who comes across this message:

The UI Behavior is: Any Add-on Builder Setup page will spin and spin and spin (e.g., it will load the overall Splunk UI, but when you try to go to the configuration page, the main panel will show the normal loading icon but it won't go away).

The back-end behavior is: you'll see a 500 error and the stack trace below. 

Root Cause: a passwords.conf that cannot be decrypted (and poor error handling in the Add-on Builder). The most common scenario for this is that you copied the local/passwords.conf from one system (encrypted using the key on that system) to another system (which uses a different key).

There has been one report of this behavior happening in a totally different app:  app1 uses the add-on builder setup. app2 used anything to create a passwords.conf but it was copied from another system so it wouldn't work there. The invalid passwords.conf in app2 prevents the app1 add-on builder from working. 

Solution: go through and clear out any local passwords.conf. You can always move the files away and move then back to figure out which one is broken. Because that passwords.conf was never valid, it shouldn't cause any system impact to do so. 

Enhancement for the Add-on Builder Team: replace this line of code with a try: e

Tags (1)

a_m_s
Explorer

Thank you @davidveuve  for the solution , was searching from a long time . Also i was wondering how did you came to observe this behavior of AOB.

0 Karma

Piiit
Engager

Thanks for posting this solution. For anyone else having this problem, look for passwords.conf in every app that is is installed on the same searchhead as the Crowdstrike app and manually try to decrypt/dehash the values to figure out which passwords.conf Splunk (and therefore also Crowdstrike) is unable to decrypt.

  find $SPLUNK_HOME/  -name "passwords.conf" 

  $SPLUNK_HOME/bin/splunk show-decrypted --value '$value'

where $value is what is after  password =  ...  under each stanza in passwords.conf

If your decryption fails, that means the password was hashed on another splunk server with another splunk.secret than this, and then copied over which makes the passwords.conf hash value unreadable. Decrypt the password on the splunk instance that the app came from originally to figure out the password. Then encrypt it using the same splunk.secret on the destination splunk server whichever way you prefer (I used a tool called splunksecrets: pip3 install splunksecrets). Past the value back into passwords.conf. Try to decrypt again to make sure the password stayed the same.

Get Updates on the Splunk Community!

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...