Splunk Add-on for Nessus Tenable API: Getting error "Fail to decrypt the encrypted credential information - not well-formed (invalid token)"?

[jonathan_cooper]

I have the latest TA Nessus installed and it was working fine for about a week importing nessus reports through the Tenable API calls. It then stopped indexing events and reported the following error(s):

2016-08-08 17:04:27,658 +0000 log_level=ERROR, pid=18084, tid=MainThread, file=ta_mod_input.py, func_name=main, code_line_no=186 | Tenable task encounter exception Traceback (most recent call last): File "/opt/splunk/etc/apps/Splunk_TA_nessus/bin/splunk_ta_nessus/splunktaucclib/data_collection/ta_mod_input.py", line 183, in main config_cls=configer_cls) File "/opt/splunk/etc/apps/Splunk_TA_nessus/bin/splunk_ta_nessus/splunktaucclib/data_collection/ta_mod_input.py", line 100, in run tconfig = tc.create_ta_config(settings, config_cls or tc.TaConfig) File "/opt/splunk/etc/apps/Splunk_TA_nessus/bin/splunk_ta_nessus/splunktaucclib/data_collection/ta_config.py", line 181, in create_ta_config return config_cls(meta_config, settings) File "/opt/splunk/etc/apps/Splunk_TA_nessus/bin/splunk_ta_nessus/splunktaucclib/data_collection/ta_config.py", line 24, in __init__ self._load_task_configs() File "/opt/splunk/etc/apps/Splunk_TA_nessus/bin/splunk_ta_nessus/splunktaucclib/data_collection/ta_config.py", line 48, in _load_task_configs self._client_schema) File "/opt/splunk/etc/apps/Splunk_TA_nessus/bin/splunk_ta_nessus/splunktaucclib/data_collection/ta_helper.py", line 67, in __init__ self._load_conf_contents() File "/opt/splunk/etc/apps/Splunk_TA_nessus/bin/splunk_ta_nessus/splunktaucclib/data_collection/ta_helper.py", line 93, in _load_conf_contents self._all_conf_contents = self._config.load() File "/opt/splunk/etc/apps/Splunk_TA_nessus/bin/splunk_ta_nessus/splunktaucclib/config.py", line 127, in load raise ConfigException(msg) ConfigException: Fail to load endpoint "global_settings" - Unspecified internal server error. reason={"messages":[{"type":"ERROR","text":"\n In handler 'ta_tenable_settings': External handler failed with code '1' and output: 'REST ERROR[1021]: Fail to decrypt the encrypted credential information - not well-formed (invalid token): line 135, column 41'. See splunkd.log for stderr output."}]}

as well as:

2016-08-08 17:04:27,658 +0000 log_level=ERROR, pid=18084, tid=MainThread, file=config.py, func_name=log, code_line_no=50 | UCC Config Module: Fail to load endpoint "global_settings" - Unspecified internal server error. reason={"messages":[{"type":"ERROR","text":"\n In handler 'ta_tenable_settings': External handler failed with code '1' and output: 'REST ERROR[1021]: Fail to decrypt the encrypted credential information - not well-formed (invalid token): line 135, column 41'.  See splunkd.log for stderr output."}]}
  File "/opt/splunk/etc/apps/Splunk_TA_nessus/bin/tenable_sc.py", line 21, in <module>
    ta_run()
  File "/opt/splunk/etc/apps/Splunk_TA_nessus/bin/tenable_sc.py", line 17, in ta_run
    ta_input.main(collector_cls, schema_file_path, 'tenable_sc')
  File "/opt/splunk/etc/apps/Splunk_TA_nessus/bin/splunk_ta_nessus/splunktaucclib/data_collection/ta_mod_input.py", line 183, in main
    config_cls=configer_cls)
  File "/opt/splunk/etc/apps/Splunk_TA_nessus/bin/splunk_ta_nessus/splunktaucclib/data_collection/ta_mod_input.py", line 100, in run
    tconfig = tc.create_ta_config(settings, config_cls or tc.TaConfig)
  File "/opt/splunk/etc/apps/Splunk_TA_nessus/bin/splunk_ta_nessus/splunktaucclib/data_collection/ta_config.py", line 181, in create_ta_config
    return config_cls(meta_config, settings)
  File "/opt/splunk/etc/apps/Splunk_TA_nessus/bin/splunk_ta_nessus/splunktaucclib/data_collection/ta_config.py", line 24, in __init__
    self._load_task_configs()
  File "/opt/splunk/etc/apps/Splunk_TA_nessus/bin/splunk_ta_nessus/splunktaucclib/data_collection/ta_config.py", line 48, in _load_task_configs
    self._client_schema)
  File "/opt/splunk/etc/apps/Splunk_TA_nessus/bin/splunk_ta_nessus/splunktaucclib/data_collection/ta_helper.py", line 67, in __init__
    self._load_conf_contents()
  File "/opt/splunk/etc/apps/Splunk_TA_nessus/bin/splunk_ta_nessus/splunktaucclib/data_collection/ta_helper.py", line 93, in _load_conf_contents
    self._all_conf_contents = self._config.load()
  File "/opt/splunk/etc/apps/Splunk_TA_nessus/bin/splunk_ta_nessus/splunktaucclib/config.py", line 126, in load
    log(msg, level=logging.ERROR, need_tb=True)
  File "/opt/splunk/etc/apps/Splunk_TA_nessus/bin/splunk_ta_nessus/splunktaucclib/config.py", line 48, in log
    stack = ''.join(traceback.format_stack())
None

I've tried restarting the Heavy Forwarder that is collecting it, as well as changing the "start_time" located in the tenable_sc_inputs.conf to try and reset the checkpoint information, but no luck.

[jbailey_splunk]

This answer evolved over time as there were two issues eventually listed - the first related to "Fail to decrypt the encrypted credential information - not well-formed (invalid token)", and the second related to the following message: "APIError: 'status=403, error_code=12, error_msg=This request contains an invalid token".

The second issue was resolved by the latest TA release... From following the answer thread, the first issue was resolved from the post on August 25 @ 8:07 am, which states: "I was able to resolve the curl issue, but only after removing passwords.conf and replacing the password in tenable_sc_server.conf and then restarting Splunk a few times."

I would recommend re-configuring the TA, ensuring the passwords.conf and tenable_sc_server.conf files are correct. Also, make sure Splunk is restarted - just in case.

Hope that helps...

[worshamn]

I opened a support ticket with Splunk. The issue of the "This request contains an invalid token" in my case is because we are using Security Center 5.4. Splunk informed me that in 5.4, Tenable changed their set-cookie format (which was that it returns 2 cookies, one of which is valid). Splunk knows of the issue and is planning on adding support for 5.4 in a future version, but could not provide a timeline.

[jbailey_splunk]

@worshamn - I received the same answer with support... Same issue, same version of Security Center.

[worshamn]

Here are the lines on our heavy forwarder in the TA app on the file: $SPLUNK_HOME/etc/apps/Splunk_TA_nessus/bin/splunk_ta_nessus/security_center.py

135 
136         if response.get('set-cookie') is not None:
137             self._cookie = response.get('set-cookie')
138             self._cookie = self._cookie[74:]
139         return result['response']
140 

[jat75]

Thanks! But hmm, my code looks nothing like that haha. I guess i'll keep poking.
$SPLUNK_HOME/etc/apps/Splunk_TA_nessus/bin/splunk_ta_nessus/security_center.py

132 response, content = http.request(
133 self._uri(path), method, data, headers)
134
135 if path.find('download') != -1:
136 return content
137
138 result = json.loads(content)
139
140 self._error_check(response, result)
141
142 set_cookie = response.get('set-cookie')
143
144 if set_cookie:
145 self._cookie = set_cookie[set_cookie.find(',') + 1:].strip()
146 stulog.logger.debug('{} set-cookie={}'.format(self._logger_prefix,
147 set_cookie))
148 stulog.logger.debug('{} self._cookie={}'.format(
149 self._logger_prefix, self._cookie))
150
151 return result['response']

[worshamn]

Oh right, looks we are running an older version of the app (at the time of original posting was the latest). Version 5.0.0

[jat75]

Yeah i have Splunk_TA_Nessus 5.1.0 currently and I think that's what shipped with the latest version of ES.

[jonathan_cooper]

Latest error I'm receiving. I was able to resolve the curl issue, but only after removing passwords.conf and replacing the password in tenable_sc_server.conf and then restarting Splunk a few times. Still doesn't explain why it just stops working randomly. I have verified the credentials are good, and curl returns the right information, but it's still not working:

2016-08-25 15:00:05,516 +0000 log_level=ERROR, pid=3366, tid=Thread-7, file=ta_data_collector.py, func_name=_do_safe_index, code_line_no=161 | [stanza_name="sc_input" data="sc_vulnerability" server="prod_sc"] Failed to get msg
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/Splunk_TA_nessus/bin/splunk_ta_nessus/splunktaucclib/data_collection/ta_data_collector.py", line 151, in _do_safe_index
    events, ckpt = self._client.get()
  File "/opt/splunk/etc/apps/Splunk_TA_nessus/bin/splunk_ta_nessus/splunktaucclib/data_collection/ta_data_client.py", line 73, in get
    return self._gen.next()
  File "/opt/splunk/etc/apps/Splunk_TA_nessus/bin/splunk_ta_nessus/ta_tenable_sc_data_collector.py", line 93, in _process_sc_vulnerability
    _pre_process_ckpt(sc, task_config, ckpt, logger_prefix)
  File "/opt/splunk/etc/apps/Splunk_TA_nessus/bin/splunk_ta_nessus/ta_tenable_sc_data_collector.py", line 212, in _pre_process_ckpt
    job_start_time, end_time))
  File "/opt/splunk/etc/apps/Splunk_TA_nessus/bin/splunk_ta_nessus/security_center.py", line 134, in perform_request
    self._error_check(response, result)
  File "/opt/splunk/etc/apps/Splunk_TA_nessus/bin/splunk_ta_nessus/security_center.py", line 177, in _error_check
    result['error_msg'])
APIError: 'status=403, error_code=12, error_msg=This request contains an invalid token.'

[jonathan_cooper]

This problem seems to have reared it's ugly head again, and this time it's not the passwords.conf.

After running fine for a few days, it began erroring out, this time with ERROR regarding invalid credentials and invalid tokens in the request. I verified my login credentials directly, which worked, and updated the app, while removing the passwords.conf and allowing it to regenerate, which had no effect.

I ran some curl commands against the passwords API to validate it was getting the right results and began seeing odd behavior. For instance, I ran the following:

curl -q --insecure -u 'admin:password' 'https://localhost:8089//servicesNS/nobody/Splunk_TA_nessus/storage/passwords?count=1'

The result returned for the Splunk_TA_cisco-ise (WHAAAT?) app instead. If I changed the "count" to 0 or -1, I would get the right app to return, but the following text for clear password:

<s:key name="clear_password">proxy_password``splunk_cred_sep````splunk_cred_sep``proxy_username``splunk_cred_sep``</s:key>

Could this be a bug? I'm not sure what could cause this.

[dwaddle]

First guess is that it's having trouble decrypting the password stored in passwords.conf. Maybe it was made on a node with a different splunk.secret? Does resetting the credentials by hand help?

[jonathan_cooper]

So that was part of the issue, it was in fact that the passwords.conf that was generated was messed up during an ES upgrade and merging of TA's. The unfortunate issue is how this gets deployed via the Deployment Server. Being that it generates a passwords.conf after restart, it doesn't jive well with how the Deployment Server works.

Once manually installed on the Heavy Forwarder and the passwords.conf cleared and re-generated, it seems to be working fine.

[jonathan_cooper]

Unaccepted just because I'm still having on-going issues, this was the answer to the original problem however.

[supreetsingh]

I am having the same issues and looking for a resolution.

APIError: 'status=403, error_code=12, error_msg=This request contains an invalid token.

[worshamn]

Same issue here too:

  File "/opt/splunk/etc/apps/Splunk_TA_nessus/bin/splunk_ta_nessus/security_center.py", line 177, in _error_check
    result['error_msg'])
APIError: 'status=403, error_code=12, error_msg=This request contains an invalid token.'

Has anyone tried a support ticket yet?

[jonathan_cooper]

I had another consultant get a Jira ticket opened internally, I'll see if I can get a status