Qualys Technology Add-on (TA) for Splunk: Are these errors related to lack of API calls to Qualys or add-on configuration?

[becksyboy]

Hi,

We are testing the Qualys Technology Add-on (TA) for Splunk v 1.1.0 within our Dev Splunk environment v 6.4.2.

At the moment, the Dev environment does not have external internet access to make API calls to Qualys. However, we are seeing a few errors and was wondering if any of them relate to the lack of internet access or a Splunk or the add-on configuration issue:

Issue [1]

We are seeing the following error within the Splunk web console messages drop down:

msg="A script exited abnormally" input="/opt/splunk/etc/apps/Splunk_TA_QualysCloudPlatform/bin/qualys.py" stanza="qualys://knowledge_base" status="exited with code 1"

I think this maybe related to the lack of internet access?

Issue [2]

When we restart Splunk, from the Linux command line console we are seeing the following:

    Checking conf files for problems...
                Invalid key in stanza [apply_qualys_tag_to_webapp] in /opt/splunk/etc/apps/TA-QualysCloudPlatform/default/alert_actions.conf, line 11: param.tag_ids  (value:  0).
                Invalid key in stanza [apply_qualys_tag_to_webapp] in /opt/splunk/etc/apps/TA-QualysCloudPlatform/default/alert_actions.conf, line 17: param._cam  (value:  {
"supports_adhoc": true,
"category": ["Information Gathering"],
"task": ["create"],
"subject": ["process.reputation-service"],
"technology": [ {"vendor": "Qualys"},{"product": "WAS"},{"version": "0.1"}]}).
                Invalid key in stanza [qualys] in /opt/splunk/etc/apps/TA-QualysCloudPlatform/default/inputs.conf, line 5: passAuth  (value:  admin).
                Your indexes and inputs configurations are not internally consistent. For more information, run 'splunk btool check --debug'

We have disabled this;

splunk ~/etc/apps/TA-QualysCloudPlatform/local cat alert_actions.conf 
[apply_qualys_tag_to_webapp]
disabled = 1
ttl = 0

But we are still seeing the errors, which are related to the "custom params" in the alert_actions.conf.

thanks

[princemanto2580]

For me also getting some kind of different error for both was_findings and knowledge_base. Any please or suggestion will be highly appreciate. Log source path is "ta_QualysCloudPlatform.log" under /opt/splunk/var/log/splunk.

[was_findings]

TA-QualysCloudPlatform: 2017-10-15T10:12:15Z PID=9027 [MainThread] DEBUG: TA-QualysCloudPlatform [knowledge_base] - Loading custom settings
TA-QualysCloudPlatform: 2017-10-15T10:12:15Z PID=9027 [MainThread] INFO: TA-QualysCloudPlatform [knowledge_base] - Start
TA-QualysCloudPlatform: 2017-10-15T10:12:15Z PID=9027 [MainThread] INFO: TA-QualysCloudPlatform [knowledge_base] - Start logging knowledgebase
TA-QualysCloudPlatform: 2017-10-15T10:12:15Z PID=9027 [MainThread] INFO: TA-QualysCloudPlatform [knowledge_base] - Outputting logs to stdout
TA-QualysCloudPlatform: 2017-10-15T10:12:15Z PID=9027 [MainThread] INFO: TA-QualysCloudPlatform [knowledge_base] - Making request: https://qualysapi.qualys.eu/api/2.0/fo/knowledge_base/vuln/ with params={'details': 'Basic', 'action': 'list'}
TA-QualysCloudPlatform: 2017-10-15T10:12:16Z PID=9027 [MainThread] ERROR: TA-QualysCloudPlatform [knowledge_base] - Unsuccessful while calling API [403 : Forbidden]. Retrying: https://qualysapi.qualys.eu/api/2.0/fo/knowledge_base/vuln/ with params={'details': 'Basic', 'action': 'list'}. Retry count: 1

[knowledge_base]

TA-QualysCloudPlatform: 2017-10-15T13:14:20Z PID=13220 [MainThread] DEBUG: TA-QualysCloudPlatform [was_findings] - Loading custom settings
TA-QualysCloudPlatform: 2017-10-15T13:14:20Z PID=13220 [MainThread] INFO: TA-QualysCloudPlatform [was_findings] - Start
TA-QualysCloudPlatform: 2017-10-15T13:14:20Z PID=13220 [MainThread] INFO: TA-QualysCloudPlatform [was_findings] - Start
TA-QualysCloudPlatform: 2017-10-15T13:14:20Z PID=13220 [MainThread] INFO: TA-QualysCloudPlatform [was_findings] - WAS findings were last fetched on 1999-01-01T00:00:00Z
TA-QualysCloudPlatform: 2017-10-15T13:14:20Z PID=13220 [MainThread] INFO: TA-QualysCloudPlatform [was_findings] - Fetching WAS findings data for Hosts which were scanned after 1999-01-01T00:00:00Z
TA-QualysCloudPlatform: 2017-10-15T13:14:20Z PID=13220 [MainThread] INFO: TA-QualysCloudPlatform [was_findings] - Fetching all WAS detection data
TA-QualysCloudPlatform: 2017-10-15T13:14:20Z PID=13220 [MainThread] INFO: TA-QualysCloudPlatform [was_findings] - Running in single thread mode
TA-QualysCloudPlatform: 2017-10-15T13:14:20Z PID=13220 [MainThread] INFO: TA-QualysCloudPlatform [was_findings] - Making request: https://qualysapi.qualys.eu/qps/rest/3.0/search/was/finding with params=true1999-01-01T00:00:00Z
TA-QualysCloudPlatform: 2017-10-15T13:14:21Z PID=13220 [MainThread] INFO: TA-QualysCloudPlatform [was_findings] - WAS detection fetched
TA-QualysCloudPlatform: 2017-10-15T13:14:21Z PID=13220 [MainThread] INFO: TA-QualysCloudPlatform [was_findings] - Parsing WAS detection XML
TA-QualysCloudPlatform: 2017-10-15T13:14:21Z PID=13220 [MainThread] INFO: TA-QualysCloudPlatform [was_findings] - API Response Code = UNAUTHORIZED
TA-QualysCloudPlatform: 2017-10-15T13:14:21Z PID=13220 [MainThread] DEBUG: TA-QualysCloudPlatform [was_findings] - Exception while parsing. API ERROR. Message=UNAUTHORIZED :: Traceback (most recent call last):
TA-QualysCloudPlatform: 2017-10-15T13:14:21Z PID=13220 [MainThread] ERROR: TA-QualysCloudPlatform [was_findings] - could not load API response
TA-QualysCloudPlatform: 2017-10-15T13:14:21Z PID=13220 [MainThread] INFO: TA-QualysCloudPlatform [was_findings] - Done
TA-QualysCloudPlatform: 2017-10-15T13:14:21Z PID=13220 [MainThread] INFO: TA-QualysCloudPlatform [was_findings] - was_findings run completed. Sleeping for 10800s

[prabhasgupte]

Hi,

I am assuming that you have properly configured the TA and set your Qualys API credentials.

Splunk might have encountered some exception while initializing the scheme. To confirm that, you can make use of run.py in TA/bin directory.
If your SPLUNK_HOME is /opt/splunk, then from SPLUNK_HOME/etc/apps/TA-QualysCloudPlatform run following command.
/opt/splunk/bin/splunk cmd python ./bin/run.py -h
From the help, you will come to know how to run knowledgebase input. Then prepare that command and run it.
See what it outputs to stdout. Since you don't have Internet access, it will obviously not be able to download knowledgebase, but the error should only be related to Internet access (like no host found etc.). If it throws any other error which is not related to URL access, then that's the reason of your issue 1.
Please post the output here, if that does not help you.

For issue 2, that is not a issue to worry about as such. Qualys have added a custom alert action to this TA version, which is for Splunk's Enterprise Security App. For some reason, Splunk does complain about some of the custom action settings and hence those messages. Trust me, that does not affect any of TA or Splunk functionality. That actually should have been written somewhere in TA as known thing.

[becksyboy]

Hi,

thanks for your clarification with this.

Issue [1]
After running the run.py script to try and download the knowledgebase input and the host-detection inputs; the output would seem to point to the issue being a lack of external access, output below. Once we can get external access enabled to our Dev environment we can re-test.

Issue [2]
At the moment we do not need to expose the custom alert actions, and would like to remove those start up error messages. What would be the correct way to disable the custom alert actions for the time being?

============================================================================
splunk ~/bin ./splunk cmd python /opt/splunk/etc/apps/TA-QualysCloudPlatform/bin/run.py -k
QG Username:
QG Password:
TA-QualysCloudPlatform: 2016-11-17T11:32:22Z PID=14837 [MainThread] INFO: TA-QualysCloudPlatform - Making request: https://qualysapi.qualys.com/msp/about.php with params={}
_internal
TA-QualysCloudPlatform: 2016-11-17T11:33:25Z PID=14837 [MainThread] ERROR: TA-QualysCloudPlatform - Error during request to /msp/about.php, [None] [Errno 110] Connection timed out
_internal
Traceback (most recent call last):
File "/opt/splunk/etc/apps/TA-QualysCloudPlatform/bin/run.py", line 138, in
qapi.client.validate()
File "/opt/splunk/etc/apps/TA-QualysCloudPlatform/bin/qualysModule/lib/api/Client.py", line 200, in validate
response = self.get("/msp/about.php", {}, SimpleAPIResponse())
File "/opt/splunk/etc/apps/TA-QualysCloudPlatform/bin/qualysModule/lib/api/Client.py", line 259, in get
raise APIRequestError("Error during request to %s, [%s] %s" % (end_point, ue.errno, ue.reason))
qualysModule.lib.api.Client.APIRequestError: Error during request to /msp/about.php, [None] [Errno 110] Connection timed out

splunk ~/bin ./splunk cmd python /opt/splunk/etc/apps/TA-QualysCloudPlatform/bin/run.py -d
QG Username:
QG Password:
TA-QualysCloudPlatform: 2016-11-17T11:35:44Z PID=22002 [MainThread] INFO: TA-QualysCloudPlatform - Making request: https://qualysapi.qualys.com/msp/about.php with params={}
_internal
TA-QualysCloudPlatform: 2016-11-17T11:36:47Z PID=22002 [MainThread] ERROR: TA-QualysCloudPlatform - Error during request to /msp/about.php, [None] [Errno 110] Connection timed out
_internal
Traceback (most recent call last):
File "/opt/splunk/etc/apps/TA-QualysCloudPlatform/bin/run.py", line 138, in
qapi.client.validate()
File "/opt/splunk/etc/apps/TA-QualysCloudPlatform/bin/qualysModule/lib/api/Client.py", line 200, in validate
response = self.get("/msp/about.php", {}, SimpleAPIResponse())
File "/opt/splunk/etc/apps/TA-QualysCloudPlatform/bin/qualysModule/lib/api/Client.py", line 259, in get
raise APIRequestError("Error during request to %s, [%s] %s" % (end_point, ue.errno, ue.reason))
qualysModule.lib.api.Client.APIRequestError: Error during request to /msp/about.php, [None] [Errno 110] Connection timed out

[prabhasgupte]

You can safely edit those files and comment out below mentioned lines.

Open default/inputs.conf and change passAuth = admin to # passAuth = admin.
Open default/alert_actions.conf and change following section

# custom params
param.tag_ids = 0
param._cam = {\
"supports_adhoc": true,\
"category": ["Information Gathering"],\
"task": ["create"],\
"subject": ["process.reputation-service"],\
"technology": [ {"vendor": "Qualys"},{"product": "WAS"},{"version": "0.1"}]}

to

# custom params
# param.tag_ids = 0
# param._cam = {\
# "supports_adhoc": true,\
# "category": ["Information Gathering"],\
# "task": ["create"],\
# "subject": ["process.reputation-service"],\
# "technology": [ {"vendor": "Qualys"},{"product": "WAS"},{"version": "0.1"}]}

Do not forget to restart your Splunk after these changes!

[becksyboy]

Hi,

when we edit:
/default/inputs.conf
/default/alert_actions.conf

And make the suggested changes, then we can see there are no start up errors listed.

We like to create a custom Inputs TA with it's own local folder with settings to overwrite the TA. Hence if the Qualys TA was updated we could install the new version whilst knowing our custom inputs TA would overwrite the default settings.

For testing we left the local inputs within the TA-QualysCloudPlatform folder under:
TA-QualysCloudPlatform/local/inputs.conf
TA-QualysCloudPlatform/local/alert_actions.conf

However, the settings in /local do not seem to take precedence over the settings in /default

====================
TA-QualysCloudPlatform/local/inputs.conf

[qualys]
index = nbs_qualys
disabled = false
interval = 60

passAuth = admin

[qualys://knowledge_base]
duration = 86400
index = nbs_qualys
start_date = 1999-01-01T00:00:00Z
disabled = 0

[qualys://host_detection]
duration = 86400
index = nbs_qualys
start_date = 1999-01-01T00:00:00Z
disabled = 0

=================
TA-QualysCloudPlatform/local/alert_actions.conf

[apply_qualys_tag_to_webapp]
disabled = 1
ttl = 0

custom params

param.tag_ids = 0

param._cam = {\

"supports_adhoc": true,\

"category": ["Information Gathering"],\

"task": ["create"],\

"subject": ["process.reputation-service"],\

"technology": [ {"vendor": "Qualys"},{"product": "WAS"},{"version": "0.1"}]}

[prabhasgupte]

Try putting those files in SPLUNK_HOME/etc/apps/search/local directory and restart the Splunk. That should do the job.

[becksyboy]

Hi, not sure why we should have to add the files to this location, it should work from the /local folder within the TA? either way we tried this and it didn't work.