All Apps and Add-ons

Splunk Add-on for Nessus: How to import nessus field's "plugin_output"?

simmppllee
New Member

Hi,
I have installed Splunk add-on for Nessus, but there isn't field "plugin_output". I try to edit script nessus2splunk.py with adding a row - ('plugin_output', 'Data', str), - it doesn't help. Can anyone provide some solution to fix it? Thanks!

0 Karma

Grumpalot
Communicator

Ok after further efforts I have been able to update the def _collect_one_host_scan_info inside of nessus_data_collector. I will post the code below which will replace lines 170-222 of the nessus_data_collector.py used by 6x. Please understand that this code is not supported by Splunk currently, so use with knowing support may not be given for this code if something does not work. I left my re.sub for plugin_output_info noted out but if you would like to do any regex before the data is sent into splunk you can do so there on the clean variable then pass to the append(). Please let me know how it works for you and if any issues.

def _collect_one_host_scan_info(self, host_id, sid, scan_info):
        """
        The method to collect all the vulnerabilities of one host and generate the event data.
        """
        count = 0
        host_uri = self.endpoint + '/' + str(sid) + '/hosts/' + str(host_id)
        result = self.client.request(host_uri).get("content")
        # if there is exception in request, return None
        if result is None:
            _LOGGER.info("There is exception in request, return None")
            return None
        else:
            host_info = result.get("info", {})
            host_end_time = host_info.get("host_end", "")
            if self.ckpt.is_new_host_scan(host_end_time,
                                          self.config.get("start_date")):
                self.source = self.url + self.endpoint + '/' + str(
                    sid) + '/hosts/' + str(host_id)
                for vuln in result.get("vulnerabilities", []):
                    vuln["sid"] = sid
                    vuln["host_id"] = host_id
                    plugin_id = vuln.get("plugin_id", "")

                    # get plugin_output data
                    plugin_output_info = []
                    if plugin_id:
                        plugin_uri = "{}/plugins/{}".format(host_uri,
                                                            plugin_id)
                        plugin_outputs = self.client.request(plugin_uri).get(
                            "content", {}).get("outputs", [])
                        data_output = []
                        for output in plugin_outputs:
                            items = output.get("plugin_output", 'no value')
                            item = str(items)
                            #clean = re.sub('[^a-zA-Z0-9-()_*.(:\\)]', ' ', item)
                            plugin_output_info.append(item)

                    # get the port info        
                    port_info = []
                    if plugin_id:
                        plugin_uri = "{}/plugins/{}".format(host_uri,
                                                            plugin_id)
                        plugin_outputs = self.client.request(plugin_uri).get(
                            "content", {}).get("outputs", [])
                        ports = []
                        for output in plugin_outputs:
                            ports.extend(output.get("ports", {}).keys())
                        for port in ports:
                            port_elem = {}
                            port_items = re.split(r"\s*/\s*", port)
                            port_elem["port"] = int(port_items[0])
                            if port_items[1]:
                                port_elem["transport"] = port_items[1]
                            if port_items[2]:
                                port_elem["protocol"] = port_items[2]
                            port_info.append(port_elem)

                    vuln = dict(vuln, **scan_info)
                    vuln = dict(vuln, **host_info)
                    if port_info:
                        vuln["ports"] = port_info
                    if plugin_output_info:
                        vuln["plugin_output"] = plugin_output_info
                    entry = NessusObject(
                        vuln.get("timestamp"), self.sourcetype, self.source,
                        vuln)
                    self._print_stream(entry)
                    count += 1
        return count

simmppllee
New Member

Yep, thank you very much, all are work great!!!

0 Karma

Grumpalot
Communicator

Wanted to check back in and see if anyone has had any issues with the above code if they have implemented the change.

0 Karma

willi358
Engager

I had the same requirement that I wanted the plugin output information available in Splunk. So I have tested your code. It is working. However, there a some minor issues:

The field is named plugin_output{} instead of plugin_output. Do the curly braces have any meaning?

Sometimes, the line breaks are not working. When I view the plugin output in Nessus then the output is extended over several lines. However, in Splunk, all lines are pressed into one line. That makes it hard to read. (I have Nessus and Splunk running on Linux). Would it be possible to fix that issue?

0 Karma

Grumpalot
Communicator

Thank you for reaching out @willi358. I have moved on from the position where I wrote this code for and would need to get an environment setup to test further.

You can do a field alias for plugin_output{} changing it too plugin_output for your sourcetype=nessus:scan

http://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Addaliasestofields

and for the line breaking issue can you supply me with an event so I can test further, for possible formatting within Splunk?

0 Karma

willi358
Engager

Thanks @Grumpalot for your quick reply. So here is such an example:
Nessus: https://postimg.org/image/60wd5vzcr/
Splunk: https://postimg.org/image/830vwywvp/
In this examle, the plugin output consists of two sections because it applies to two ports. In Splunk, it is all grouped together into one section without the port/host information and without proper line breaks.
If you could fix that issue, that would be great.

Edit:
Here is what the JSON output from the Nessus API looks like:
https://postimg.org/image/t40t8fw1x/
In nessus_data_collector.py there is the code section "get the port info". Since the plugin output is specific for each port, perhaps in this code section we can extract the plugin output data for each port.

Edit 2:
Okay so I made some changes myself. As I said, I put your "#get plugin_output data" and "# get the port info" sections together because the plugin output is different for each port. In the original nessus_data_collector.py script, I replaced the lines 192-211 with the following:

                    # get the port info
                    plugin_id = vuln.get("plugin_id", "")
                    port_info = []
                    if plugin_id:
                        plugin_uri = "{}/plugins/{}".format(host_uri,
                                                            plugin_id)
                        plugin_outputs = self.client.request(plugin_uri).get(
                            "content", {}).get("outputs")
                        for output in plugin_outputs:
                            ports = output.get("ports", {}).keys()
                            plugin_output = output.get("plugin_output", "")
                            if plugin_output:
                                plugin_output = str(plugin_output)
                                plugin_output = plugin_output.replace("\n", "#NL#")
                                plugin_output = plugin_output.replace("\\", "#BL#")
                            else:
                                plugin_output = "N/A"
                            for port in ports:
                                port_elem = {}
                                port_items = re.split(r"\s*/\s*", port)
                                port_elem["port"] = int(port_items[0])
                                if port_items[1]:
                                    port_elem["transport"] = port_items[1]
                                else:
                                    port_elem["transport"] = "N/A"
                                if port_items[2]:
                                    port_elem["protocol"] = port_items[2]
                                else:
                                    port_elem["protocol"] = "N/A"
                                port_elem["plugin_output"] = plugin_output
                                port_info.append(port_elem)

I don't understand why I don't see newline and backslash characters in Splunk. So in the collector script I replace them with "#NL#" and "#BL#" and later in Splunk convert them back to \n and \. It's not pretty. But at least it's now working for me.

0 Karma

Grumpalot
Communicator

Great update to the code @willi358. I do not have an environment with nessus data to test this out anymore but I'm sure this update will make the plugin_output field data that much more useful now.

0 Karma

Grumpalot
Communicator

The script you are using for import for 6x is the nessus_data_collector.py

Currently with the new 6x version I have been unable to find a way to pull the "plugin_output" field since its not written into the Class NessusScanCollector def _collect_one_host_scan_info inside of the nessus_data_collector.py. I have been working on a way to include the data into Splunks add-on with no luck so I opened as feature request with Splunk over a month ago.

If you happen to know a python expert who can look at the code from line 170-222 in the nessus_datat_collector.py you may be able to find a way to include that portion of the JSON since its already pulling "ports".

0 Karma

simmppllee
New Member

Thanks, for your help, try to find expert if I will have some solution I will write there!

0 Karma

Grumpalot
Communicator

are you using Nessus 5x? just wanted to verify first.

0 Karma

simmppllee
New Member

no, Nessus 6x and add-on has been updated

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!