All Apps and Add-ons

Qualys App for Splunk Enterprise: After changing python scripts to add more fields than the defaults, why are fields being truncated?

Ovi
Path Finder

Hi all

I need to pull into SPlunk more fields from Qualys than the defaults.
I changed the detectionpupulator.py and kppopulator.py scripts to add these fields and they do work - however I have a few challenges with field truncation

Especially the RESULTS fields is severely truncated reduced to just a few characcters from the first line

I am pasting below the details, if anyone had experience with these and can give me hint I would greatly appreciate it

  1. Adding RESULTS field = Main Issue = severely truncated I changed detectionpopulator.py and added RESULTS as a detection fields to log detection_fields_to_log = ["QID", "TYPE", "PORT", "PROTOCOL", "SSL", "STATUS", "LAST_UPDATE_DATETIME", "LAST_FOUND_DATETIME","FIRST_FOUND_DATETIME", "LAST_TEST_DATETIME", "RESULTS"]

This works fine but the RESULTS field is severely truncated – it only contains the first few characters up to 10-20, always only form the first line.(I tried to change self.truncation_limit but that doesn’t seem to make any difference)
Question: how do I change this so that it will parse the full RESULTS field?

  1. Similar question for the SOLUTION field I added in kbpopulator.py three additional fields. They are getting parsed fine but the SOLUTION field is still being truncated after 2000-3000 characters....and they want it all 🙂

QID_EXTRA_FIELDS_TO_LOG = ["VULN_TYPE", "PATCHABLE", "PCI_FLAG", "TITLE", "CATEGORY", "DIAGNOSIS", "CONSEQUENCE", "SOLUTION", "PUBLISHED_DATETIME"]
Question: basically same question for the SOLUTION field – how can I make sure the field is not truncated

  1. HTML tags in SOLUTION field – any way to get rid of them? The SOLUTION field will have a lot of hyperlinks, the data comes in with all the HTML tags so it’s obviously hard to read and much lengthier Any way these tags can be stripped during the file parsing?

Thank you in advance

muralianup
Communicator

I got rid of the HTML Tags using 4SED statements to get rid of all the HTML tags & formatting:

rex field=SOLUTION mode=sed "s/&q u o t //g" ->(without spaces)
rex field=SOLUTION mode=sed "s/<[^>]*>//g
rex field=SOLUTION mode=sed "s/- - & g t//g --> (without spaces)
rex field=SOLUTION mode=sed "s/& g t//g --> (without spaces)

0 Karma

ltrand
Contributor

Is truncation happening when you run the script and output to a local file? Or is truncation happening after the script output? I would suggest looking in props.conf for the app and verify that TRUNCATE=0 is set. This will ensure that Splunk isn't truncating the script output. If the truncation is happening before that, I will defer to those that have a Qualys system to test with.

0 Karma

srinathd
Contributor

In kbpopulator.py file i have added SOLUTION as additional field. I restarted it after the modification. But the solution values are not coming. How to reload this file?

0 Karma

muralianup
Communicator

Did you try append this to the search ?

| lookup qualys_kb_lookup QID OUTPUT TITLE SEVERITY CATEGORY SOLUTION

0 Karma
Get Updates on the Splunk Community!

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...

Get the T-shirt to Prove You Survived Splunk University Bootcamp

As if Splunk University, in Las Vegas, in-person, with three days of bootcamps and labs weren’t enough, now ...

Wondering How to Build Resiliency in the Cloud?

IT leaders are choosing Splunk Cloud as an ideal cloud transformation platform to drive business resilience,  ...