Why does the TA_QualysCloudPlatform app installed by Splunk logs errors and leaves behind temporary files?

[rsantoso_splunk]

Files with names like:

kb_2019-01-05-30-08_MainThread_102498_batch_1.xml 

started building up in directory:

/opt/splunk/etc/apps/TA-QualysCloudPlatform/tmp 

The files are 200MB in size, and up to 25 of them are being created per day.
The only error we could find in log:

/opt/splunk/var/log/splunk/ta_QualysCloudPlatform.log 

is:

TA-QualysCloudPlatform: 2019-01-29T05:57:48Z PID=5451 [MainThread] ERROR: TA-QualysCloudPlatform [was_findings] - Unsuccessful while calling API [404 : Not Found].

Traceback (most recent call last):

File "/opt/splunk/etc/apps/TA-QualysCloudPlatform/bin/qualysModule/lib/api/Client.py", line 246, in get 
request = urllib2.urlopen(req, timeout=300) # timeout set to bail in case of timeouts 
File "/opt/splunk/lib/python2.7/urllib2.py", line 154, in urlopen 
return opener.open(url, data, timeout) 
File "/opt/splunk/lib/python2.7/urllib2.py", line 437, in open 
response = meth(req, response) 
File "/opt/splunk/lib/python2.7/urllib2.py", line 550, in http_response 
'http', request, response, code, msg, hdrs) 
File "/opt/splunk/lib/python2.7/urllib2.py", line 475, in error 
return self._call_chain(*args) 
File "/opt/splunk/lib/python2.7/urllib2.py", line 409, in _call_chain 
result = func(*args) 
File "/opt/splunk/lib/python2.7/urllib2.py", line 558, in http_error_default 
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) 
HTTPError: HTTP Error 404: Not Found 

The error is logged at 5 minute intervals.

The app is running on an instance of Splunk configured as a heavy forwarder. Splunk version is 6.5.1.
TA_QualysCloudPlatform version is: 1.2.3

[rsantoso_splunk · Accepted solution]

The issue is due to there was an upgrade of Qualys from version 8.15 to version 8.16

Cause:

The cause was a Qualys TA API that was used to pull the qualys_kb data, there was a new field included by Qualys in the later version 8.16.
The parsing script cannot handle it, then the system was unable to download the data.

Recommendation:

To add the new field in the parsing script (kbpopulator.py). Then, the data can be downloaded and the lookup can start working again.

Within the kbpopulator.py file, change the following line to include the new field CVSS_VECTOR_STRING:

File location:
/opt/splunk/etc/apps/TA-QualysCloudPlatform/bin/qualysModule/splunkpopulator/kbpopulator.py

Modify the following line:
CVS_HEADER_COLUMNS = ["QID", "SEVERITY"] + QID_EXTRA_FIELDS_TO_LOG + ["CVSS_BASE", "CVSS_TEMPORAL", "CVSS_VECTOR_STRING", "CVE", "VENDOR_REFERENCE"]
def init(self, logger=None):

Detail Log

After the upgrade of Qualys Web version 8.16, there was message "Exception while parsing. dict contains fields not in fieldnames: 'CVSS_VECTOR_STRING'" causing the API could not be loaded.

Before the upgrade of Qualys Web version 8.15

2019-01-05T07:30:11Z PID=27895 [MainThread] INFO: TA-QualysCloudPlatform - Start qualys TA
...
2019-01-05T07:30:13Z PID=27895 [MainThread] INFO: TA-QualysCloudPlatform - Found QWEB_VERSION=8.15
2019-01-05T07:30:13Z PID=27895 [MainThread] INFO: TA-QualysCloudPlatform - Running for qualys://knowledge_base
2019-01-05T07:30:13Z PID=27895 [MainThread] INFO: TA-QualysCloudPlatform - knowledge_base cron/duration: 30 * * * *
2019-01-05T07:30:13Z PID=27895 [MainThread] INFO: TA-QualysCloudPlatform - Current time (2019-01-05 07:30:00) matches cron format (30 * * * *) defined for knowledge_base. Running now.
...
2019-01-05T07:36:38Z PID=27895 [MainThread] INFO: TA-QualysCloudPlatform [knowledge_base] - Update lookup file: /opt/splunk/etc/apps/TA-QualysCloudPlatform/lookups/qualys_kb.csv with 41069 QIDs
2019-01-05T07:36:39Z PID=27895 [MainThread] INFO: TA-QualysCloudPlatform [knowledge_base] - Updated lookup file: /opt/splunk/etc/apps/TA-QualysCloudPlatform/lookups/qualys_kb.csv with 41069 QIDs
2019-01-05T07:36:39Z PID=27895 [MainThread] INFO: TA-QualysCloudPlatform [knowledge_base] - Parsed 41069 knowledgebase entry. Logged=0
2019-01-05T07:36:39Z PID=27895 [MainThread] DEBUG: TA-QualysCloudPlatform [knowledge_base] - Done with parsing, returning.
2019-01-05T07:36:39Z PID=27895 [MainThread] DEBUG: TA-QualysCloudPlatform [knowledge_base] - Removing tmp file /opt/splunk/etc/apps/TA-QualysCloudPlatform/bin/../tmp/kb_2019-01-05-30-11_MainThread_27895_batch_1.xml
2019-01-05T07:36:39Z PID=27895 [MainThread] INFO: TA-QualysCloudPlatform [knowledge_base] - Done logging knowledgebase
2019-01-05T07:36:39Z PID=27895 [MainThread] INFO: TA-QualysCloudPlatform [knowledge_base] - End qualys TA

After the upgrade to Qualys Web version 8.16

2019-01-05T14:30:08Z PID=61781 [MainThread] INFO: TA-QualysCloudPlatform - Start qualys TA
...
2019-01-05T14:30:10Z PID=61781 [MainThread] INFO: TA-QualysCloudPlatform - Found QWEB_VERSION=8.16
2019-01-05T14:30:10Z PID=61781 [MainThread] INFO: TA-QualysCloudPlatform - Running for qualys://knowledge_base
2019-01-05T14:30:10Z PID=61781 [MainThread] INFO: TA-QualysCloudPlatform - knowledge_base cron/duration: 30 * * * *
2019-01-05T14:30:10Z PID=61781 [MainThread] INFO: TA-QualysCloudPlatform - Current time (2019-01-05 14:30:00) matches cron format (30 * * * *) defined for knowledge_base. Running now.
...
2019-01-05T14:35:59Z PID=61781 [MainThread] INFO: TA-QualysCloudPlatform [knowledge_base] - Update lookup file: /opt/splunk/etc/apps/TA-QualysCloudPlatform/lookups/qualys_kb.csv with 41069 QIDs
2019-01-05T14:35:59Z PID=61781 [MainThread] DEBUG: TA-QualysCloudPlatform [knowledge_base] - Exception while parsing. dict contains fields not in fieldnames: 'CVSS_VECTOR_STRING' :: Traceback (most recent call last):
2019-01-05T14:35:59Z PID=61781 [MainThread] ERROR: TA-QualysCloudPlatform [knowledge_base] - could not load API response
2019-01-05T14:35:59Z PID=61781 [MainThread] INFO: TA-QualysCloudPlatform [knowledge_base] - Done logging knowledgebase
2019-01-05T14:35:59Z PID=61781 [MainThread] INFO: TA-QualysCloudPlatform [knowledge_base] - End qualys TA