All Apps and Add-ons

IS there way to pull extra information from Qualys like asset Tags, Groups?

kiran331
Builder

Hi

What change i have to make and where to pull extra information from qualys suing TA-qualys Add-on?

0 Karma

Robbie1194
Communicator

Anyone had any luck with prabhasgupte's method of getting the "RESULTS" field extracted? I've followed the steps but still not getting anything unfortunately.

0 Karma

todd_miller
Communicator

So there was a bug that was listed as fixed in the latest version of the TA. It wasn't until the TA was upgraded that I was able to pull the results fields with no additional editing.

Unfortunately, prabhasgutpe's comments about multi-lined results fields rings very true. While it's able to pull the data, it's absolutely useless because of incorrect handling of the multi-lined fields. It'd be great for the data source to be cleaned up so it presents the data in a usable fashion OR the add-on is updated to accommodate the multi-lined fields.

0 Karma

todd_miller
Communicator

This is fantastic.

Any help you can provide for pulling back the RESULTS field would be appreciated!

0 Karma

prabhasgupte
Communicator

I assume that you are using TA 1.2.2+

To get the RESULTS field indexed in host detection input, do the followings:

  1. On TA Setup page, in VM Detection extra parameters, set show_results=0
  2. Open bin/qualysModule/splunkpopulator/detectionpopulator.py and find class HostDetectionPopulator.
  3. In this class, find _process_root_element(self, elem) method.
  4. In that method, we have a list named HostDetectionPopulator.detection_fields_to_log. Its a list of fields to parse from detection tag. If you don't find it, you can copy one from the top of the same class.
  5. In that list, add "RESULTS" at the end. As a best practice, add a comment describing why you edited this list. This will tell code to parse that XML tag as well and output it while printing the event data.
  6. Save the file and restart your Splunk.

Let me draw your attention to the fact that, RESULTS in host detection API output could be multi-line text. As KV_MODE is set to auto for hostDetection input in props.conf, we are not sure how Splunk will treat the events when RESULTS field is multi-line text. It may or may not consider the multi-line text to be the part of same single event. The newline character might confuse Splunk's event detection.

0 Karma

jpittman_splunk
Splunk Employee
Splunk Employee

I was eventually able to make it work, after patching to version 1.1.0 and starting from prabhasgupte's steps:

Step 1: edit qualys.conf in $SPLUNK_HOME/etc/apps/TA-QualysCloudPlatform/local/
append the following line to the file

detection_params = {"show_tags":1}

Step 2: edit detectionpopulator.py in $SPLUNK_HOME/etc/apps/TA-QualysCloudPlatform/bin/qualysModule/splunkpopulator/
(starting from a default file, here are the sections of my working file that I changed - line numbers and unmodified lines shown to make it easier to locate the items that need to be changed and the relative code block indentations)
2.1

100 class HostDetectionPopulator(BasePopulator):
101     PLUGINS = []
102     OBJECT_TYPE = "detection"
103     FILE_PREFIX = "host_detection"
104     ROOT_TAG = 'HOST'
105
106     detection_fields_to_log = ["QID", "TYPE", "PORT", "PROTOCOL", "SSL", "STATUS", "LAST_UPDATE_DATETIME",
107                                "LAST_FOUND_DATETIME", "FIRST_FOUND_DATETIME", "LAST_TEST_DATETIME"]
108     host_fields_to_log = ["ID", "IP", "TRACKING_METHOD", "DNS", "NETBIOS", "OS", "LAST_SCAN_DATETIME", "TAGS"]
109

2.2

155     def _process_root_element(self, elem):
156         HostDetectionPopulator.host_fields_to_log = ["ID", "IP", "TRACKING_METHOD", "DNS", "NETBIOS", "OS", "LAST_SCAN_DATETIME", "TAGS"]
157         if elem.tag == "HOST":

2.3

174                 if name in HostDetectionPopulator.host_fields_to_log:
175                 # TAGS parsing starts here
176                     if name == "TAGS":
177                         host_tags = []
178                         tag_elements = sub_ele.findall('./TAG/NAME')
179                         for tag_element in list(tag_elements):
180                             host_tags.append(tag_element.text)
181
182                         val = ",".join(host_tags)
183                     else:
184                 # TAGS parsing ends here
185                         val = sub_ele.text
186
187                     if name in fields_to_encode:
188                         val = val.encode('utf-8')
189                     host_summary.append("%s=\"%s\"" % (name, val))

Step 3
restart splunk

Step 4
Confirm it is working
search for sourcetype="qualys:hostDetection" eventtype=qualys_host_summary_event | stats count by TAGS

If troubleshooting try searching index=_internal sourcetype=splunkd TA-QualysCloudPlatform source="/opt/splunk/var/log/splunk/splunkd.log" OR if debug logs are enabled, try looking in $SPLUNK_HOME/var/log/splunk/ta_QualysCloudPlatform.log

I hope that helps clear up the confusion for anyone else with this problem.

0 Karma

n0sidd02
New Member

Did you manage to figure this out? I am running into the same dilemma and the solutions posted did not work.

0 Karma

kiran331
Builder

I tried, it din't work

0 Karma

maada
Explorer

we tried as described above
- the xml is enhanced by the TAGS information (Tag + Name)

  • but don't geht the "TAGS" field extracted in Splunk - all other fields appear ...any recommendation? we are using Splunk 6.5.2.

any changes necessary to props.conf / transforms.conf / fields.conf

Thanks

0 Karma

prabhasgupte
Communicator

You should be able to see tags in HOSTSUMMARY events (eventtype=qualys_host_summary_event). Tags would be in comma-separated string. Can you add extraction for that on your own?

0 Karma

prabhasgupte
Communicator

Hi,

following are working steps to parse the host tags. Please note, Tags will be added to host summary events (lines starting with HOSTSUMMARY:)

Steps:
1) On setup page, set Host detection extra parameter field: 'show_tags=1'
2) Take a backup of this file: /etc/apps/TA-QualysCloudPlatform/bin/qualysModule/splunkpopulator/detectionpopulator.py
3) Open and Edit code file: /etc/apps/TA-QualysCloudPlatform/bin/qualysModule/splunkpopulator/detectionpopulator.py
Tags are shown like this in API response:

 <TAGS>
    <TAG>
        <TAG_ID>
            <![CDATA[12345]]>
        </TAG_ID>
        <NAME>
            <![CDATA[Test]]>
        </NAME>
    </TAG>
    <TAG>
        <TAG_ID>
            <![CDATA[12346]]>
        </TAG_ID>
        <NAME>
            <![CDATA[Test-2]]>
        </NAME>
    </TAG>
</TAGS>

So, you will have to add TAGS to host_fields_to_log.

3.1 - Telling code to parse tags
search for method "_process_root_element" in class "HostDetectionPopulator".
Put following line as fist line of this method:

HostDetectionPopulator.host_fields_to_log = ["ID", "IP", "TRACKING_METHOD", "DNS", "NETBIOS", "OS", "LAST_SCAN_DATETIME", "TAGS"]

Edit it as per your need.

3.2 - outputting tags inline with current style
in the same file, go to Look for 'if' condition shown below.

class HostDetectionPopulator(BasePopulator):
    ...
    def _process_root_element(self, elem):
        ...
        if name in HostDetectionPopulator.host_fields_to_log:
            val = sub_ele.text
            if name in fields_to_encode:
                val = val.encode('utf-8')
            host_summary.append("%s=\"%s\"" % (name, val))

Now, because of #3.1 above, your code enters this if block. Check if name == 'TAGS'. If yes, you will have to parse the sub-XML (You will have to read TAG.NAME for each child of TAGS element.)
Since there could be multiple tags associated with host, its better to put them in list and then joining them while outputing.

Refering to same if block (indicated above), replace following line:

val = sub_ele.text

with code block below:

if name == "TAGS":
    host_tags = []
    tag_elements = sub_ele.findall('./TAG/NAME')
    for tag_element in list(tag_elements):
        host_tags.append(tag_element.text)
    # for
    val = ",".join(host_tags)
# tags parsing ends here
else:
    val = sub_ele.text

4) Save the file and restart your Splunk.
Now, your _process_root_element method should look similar to this:

class HostDetectionPopulator(BasePopulator):
    ...
    def _process_root_element(self, elem):
        HostDetectionPopulator.host_fields_to_log = ["ID", "IP", "TRACKING_METHOD", "DNS", "NETBIOS", "OS", "LAST_SCAN_DATETIME", "TAGS"]
        ...
        if name in HostDetectionPopulator.host_fields_to_log:
            if name == "TAGS":
                host_tags = []
                tag_elements = sub_ele.findall('./TAG/NAME')
                for tag_element in list(tag_elements):
                    host_tags.append(tag_element.text)
                # for
                val = ",".join(host_tags)
            # tags parsing ends here
            else:
                val = sub_ele.text

            if name in fields_to_encode:
                val = val.encode('utf-8')
            host_summary.append("%s=\"%s\"" % (name, val))

Check if you are now getting host tags in host summary events (Splunk events starting with HOSTSUMMARY:)

If anything goes wrong, restore the backup and restart your Splunk.

Let me know if that helps you.

Lindaiyu
Path Finder

Hello kiran,
The answer is yes. This add-on is written by python. By modifying some script, you can pull all information which Qualys API 2 supports.
Here is my answer for this question
https://answers.splunk.com/answers/447343/qualys-asset-tag-data.html#answer-447363
Thanks

0 Karma

kiran331
Builder

Hi Lindaiyu,

I tried this one, but it din't pull the Tag information, I added the ""show_tags":1" in the existing line

def get_api_parameters(self): return dict({"action": "list", "show_igs": 1,"show_tags":1, 'truncation_limit': self.truncation_limit}.items() + self._detection_api_filters.items())

and for 2nd part can you let me know how can i extract the TAG?

0 Karma

Lindaiyu
Path Finder
  1. show_tags=1 tells the API to get this field
  2. The file downloaded from Qualys is in format xml, then script reads and extract fields from the xml file.
    The fucntion "_process_root_element(self, elem)" is used to realise it.
    Try to read the function, add a few sentences to extract "tag"
    Just a question of python.
    here is what I do, it works:

            tg=elem.find('TAGS')
            if tg is not None:
                for tag_item in list(tg):
                    tag_id=tag_item.find('TAG_ID')
                    if tag_id is not None:
                        for sub_ele in list(tag_item): 
                            name=sub_ele.tag
                            val=sub_ele.text.upper()
    
                            if val.find("BU_") == 0:
                              host_summary.append("%s=\"%s\"" % ("BU_NAME", val))
                            else:
                              host_summary.append("%s=\"%s\"" % ("TAG_NAME", val))
    
            else:
    
                    host_summary.append("%s=\"%s\"" % ("TAG_ID", 0))
                    host_summary.append("%s=\"%s\"" % ("NAME", "NULL"))
    
            #end the piece
    
0 Karma

kiran331
Builder

Hi Lindaiyu,

I have _process_root_element(self, elem) in two places, if i place it in 1st one its show error.

"Unable to initialize modular input "qualys" defined inside the app "TA-QualysCloudPlatform": Introspecting scheme=qualys: script running failed (exited with code 1)."

Do i have to place it in the 2nd process?

0 Karma

prabhasgupte
Communicator

@kiran331, have you tried this one?

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!