Hi
What change i have to make and where to pull extra information from qualys suing TA-qualys Add-on?
Anyone had any luck with prabhasgupte's method of getting the "RESULTS" field extracted? I've followed the steps but still not getting anything unfortunately.
So there was a bug that was listed as fixed in the latest version of the TA. It wasn't until the TA was upgraded that I was able to pull the results fields with no additional editing.
Unfortunately, prabhasgutpe's comments about multi-lined results fields rings very true. While it's able to pull the data, it's absolutely useless because of incorrect handling of the multi-lined fields. It'd be great for the data source to be cleaned up so it presents the data in a usable fashion OR the add-on is updated to accommodate the multi-lined fields.
This is fantastic.
Any help you can provide for pulling back the RESULTS field would be appreciated!
I assume that you are using TA 1.2.2+
To get the RESULTS field indexed in host detection input, do the followings:
Let me draw your attention to the fact that, RESULTS in host detection API output could be multi-line text. As KV_MODE is set to auto for hostDetection input in props.conf, we are not sure how Splunk will treat the events when RESULTS field is multi-line text. It may or may not consider the multi-line text to be the part of same single event. The newline character might confuse Splunk's event detection.
I was eventually able to make it work, after patching to version 1.1.0 and starting from prabhasgupte's steps:
Step 1: edit qualys.conf in $SPLUNK_HOME/etc/apps/TA-QualysCloudPlatform/local/
append the following line to the file
detection_params = {"show_tags":1}
Step 2: edit detectionpopulator.py in $SPLUNK_HOME/etc/apps/TA-QualysCloudPlatform/bin/qualysModule/splunkpopulator/
(starting from a default file, here are the sections of my working file that I changed - line numbers and unmodified lines shown to make it easier to locate the items that need to be changed and the relative code block indentations)
2.1
100 class HostDetectionPopulator(BasePopulator):
101 PLUGINS = []
102 OBJECT_TYPE = "detection"
103 FILE_PREFIX = "host_detection"
104 ROOT_TAG = 'HOST'
105
106 detection_fields_to_log = ["QID", "TYPE", "PORT", "PROTOCOL", "SSL", "STATUS", "LAST_UPDATE_DATETIME",
107 "LAST_FOUND_DATETIME", "FIRST_FOUND_DATETIME", "LAST_TEST_DATETIME"]
108 host_fields_to_log = ["ID", "IP", "TRACKING_METHOD", "DNS", "NETBIOS", "OS", "LAST_SCAN_DATETIME", "TAGS"]
109
2.2
155 def _process_root_element(self, elem):
156 HostDetectionPopulator.host_fields_to_log = ["ID", "IP", "TRACKING_METHOD", "DNS", "NETBIOS", "OS", "LAST_SCAN_DATETIME", "TAGS"]
157 if elem.tag == "HOST":
2.3
174 if name in HostDetectionPopulator.host_fields_to_log:
175 # TAGS parsing starts here
176 if name == "TAGS":
177 host_tags = []
178 tag_elements = sub_ele.findall('./TAG/NAME')
179 for tag_element in list(tag_elements):
180 host_tags.append(tag_element.text)
181
182 val = ",".join(host_tags)
183 else:
184 # TAGS parsing ends here
185 val = sub_ele.text
186
187 if name in fields_to_encode:
188 val = val.encode('utf-8')
189 host_summary.append("%s=\"%s\"" % (name, val))
Step 3
restart splunk
Step 4
Confirm it is working
search for sourcetype="qualys:hostDetection" eventtype=qualys_host_summary_event | stats count by TAGS
If troubleshooting try searching index=_internal sourcetype=splunkd TA-QualysCloudPlatform source="/opt/splunk/var/log/splunk/splunkd.log"
OR if debug logs are enabled, try looking in $SPLUNK_HOME/var/log/splunk/ta_QualysCloudPlatform.log
I hope that helps clear up the confusion for anyone else with this problem.
Did you manage to figure this out? I am running into the same dilemma and the solutions posted did not work.
I tried, it din't work
we tried as described above
- the xml is enhanced by the TAGS information (Tag + Name)
any changes necessary to props.conf / transforms.conf / fields.conf
Thanks
You should be able to see tags in HOSTSUMMARY events (eventtype=qualys_host_summary_event). Tags would be in comma-separated string. Can you add extraction for that on your own?
Hi,
following are working steps to parse the host tags. Please note, Tags will be added to host summary events (lines starting with HOSTSUMMARY:)
Steps:
1) On setup page, set Host detection extra parameter field: 'show_tags=1'
2) Take a backup of this file: /etc/apps/TA-QualysCloudPlatform/bin/qualysModule/splunkpopulator/detectionpopulator.py
3) Open and Edit code file: /etc/apps/TA-QualysCloudPlatform/bin/qualysModule/splunkpopulator/detectionpopulator.py
Tags are shown like this in API response:
<TAGS>
<TAG>
<TAG_ID>
<![CDATA[12345]]>
</TAG_ID>
<NAME>
<![CDATA[Test]]>
</NAME>
</TAG>
<TAG>
<TAG_ID>
<![CDATA[12346]]>
</TAG_ID>
<NAME>
<![CDATA[Test-2]]>
</NAME>
</TAG>
</TAGS>
So, you will have to add TAGS to host_fields_to_log.
3.1 - Telling code to parse tags
search for method "_process_root_element" in class "HostDetectionPopulator".
Put following line as fist line of this method:
HostDetectionPopulator.host_fields_to_log = ["ID", "IP", "TRACKING_METHOD", "DNS", "NETBIOS", "OS", "LAST_SCAN_DATETIME", "TAGS"]
Edit it as per your need.
3.2 - outputting tags inline with current style
in the same file, go to Look for 'if' condition shown below.
class HostDetectionPopulator(BasePopulator):
...
def _process_root_element(self, elem):
...
if name in HostDetectionPopulator.host_fields_to_log:
val = sub_ele.text
if name in fields_to_encode:
val = val.encode('utf-8')
host_summary.append("%s=\"%s\"" % (name, val))
Now, because of #3.1 above, your code enters this if block. Check if name == 'TAGS'. If yes, you will have to parse the sub-XML (You will have to read TAG.NAME for each child of TAGS element.)
Since there could be multiple tags associated with host, its better to put them in list and then joining them while outputing.
Refering to same if block (indicated above), replace following line:
val = sub_ele.text
with code block below:
if name == "TAGS":
host_tags = []
tag_elements = sub_ele.findall('./TAG/NAME')
for tag_element in list(tag_elements):
host_tags.append(tag_element.text)
# for
val = ",".join(host_tags)
# tags parsing ends here
else:
val = sub_ele.text
4) Save the file and restart your Splunk.
Now, your _process_root_element method should look similar to this:
class HostDetectionPopulator(BasePopulator):
...
def _process_root_element(self, elem):
HostDetectionPopulator.host_fields_to_log = ["ID", "IP", "TRACKING_METHOD", "DNS", "NETBIOS", "OS", "LAST_SCAN_DATETIME", "TAGS"]
...
if name in HostDetectionPopulator.host_fields_to_log:
if name == "TAGS":
host_tags = []
tag_elements = sub_ele.findall('./TAG/NAME')
for tag_element in list(tag_elements):
host_tags.append(tag_element.text)
# for
val = ",".join(host_tags)
# tags parsing ends here
else:
val = sub_ele.text
if name in fields_to_encode:
val = val.encode('utf-8')
host_summary.append("%s=\"%s\"" % (name, val))
Check if you are now getting host tags in host summary events (Splunk events starting with HOSTSUMMARY:)
If anything goes wrong, restore the backup and restart your Splunk.
Let me know if that helps you.
Hello kiran,
The answer is yes. This add-on is written by python. By modifying some script, you can pull all information which Qualys API 2 supports.
Here is my answer for this question
https://answers.splunk.com/answers/447343/qualys-asset-tag-data.html#answer-447363
Thanks
Hi Lindaiyu,
I tried this one, but it din't pull the Tag information, I added the ""show_tags":1" in the existing line
def get_api_parameters(self): return dict({"action": "list", "show_igs": 1,"show_tags":1, 'truncation_limit': self.truncation_limit}.items() + self._detection_api_filters.items())
and for 2nd part can you let me know how can i extract the TAG?
The file downloaded from Qualys is in format xml, then script reads and extract fields from the xml file.
The fucntion "_process_root_element(self, elem)" is used to realise it.
Try to read the function, add a few sentences to extract "tag"
Just a question of python.
here is what I do, it works:
tg=elem.find('TAGS')
if tg is not None:
for tag_item in list(tg):
tag_id=tag_item.find('TAG_ID')
if tag_id is not None:
for sub_ele in list(tag_item):
name=sub_ele.tag
val=sub_ele.text.upper()
if val.find("BU_") == 0:
host_summary.append("%s=\"%s\"" % ("BU_NAME", val))
else:
host_summary.append("%s=\"%s\"" % ("TAG_NAME", val))
else:
host_summary.append("%s=\"%s\"" % ("TAG_ID", 0))
host_summary.append("%s=\"%s\"" % ("NAME", "NULL"))
#end the piece
Hi Lindaiyu,
I have _process_root_element(self, elem) in two places, if i place it in 1st one its show error.
"Unable to initialize modular input "qualys" defined inside the app "TA-QualysCloudPlatform": Introspecting scheme=qualys: script running failed (exited with code 1)."
Do i have to place it in the 2nd process?
@kiran331, have you tried this one?