PAVO Getwatchlist - nested array parsing and other woes

K_Sukumar — Mon, 22 Jan 2024 14:46:59 GMT

Good day,

First I want to say that this add-on is an absolute lifesaver when it comes to getting structured data into Splunk, and if you ever put it up on GitHub please let me know - I'd be happy to contribute.

I have found a few minor issues. I'll be using the following json in my examples:

{"total":52145,"rows": [ {"discoverable_guid":"94937859-A157-4C43-94AC-290172D50C4D","component_cpe":{"cpe23":"cpe:2.3:a:oracle:java_runtime_environment:1.8.0_381"},"cve":[]}, {"discoverable_guid":"2B933591-6192-4E42-9DFC-32C361D32208","component_cpe":{"cpe23":"cpe:2.3:a:oracle:jdk\\/sdk:1.8.0_201"},"cve":[]}, {"discoverable_guid":"DD854B8C-5900-518C-B8B6-096285936816","component_cpe":{"cpe23":"cpe:2.3:o:microsoft:windows_defender:4.18.1909.6"},"cve":[{"name":"CVE-2006-5270"},{"name":"CVE-2018-0986"},{"name":"CVE-2021-24092"},{"name":"CVE-2021-1647"},{"name":"CVE-2020-1170"},{"name":"CVE-2020-1163"},{"name":"CVE-2020-0835"},{"name":"CVE-2017-8558"},{"name":"CVE-2017-8541"},{"name":"CVE-2017-8540"},{"name":"CVE-2017-8538"},{"name":"CVE-2017-0290"},{"name":"CVE-2019-1255"},{"name":"CVE-2013-0078"},{"name":"CVE-2011-0037"},{"name":"CVE-2020-1461"},{"name":"CVE-2020-1002"},{"name":"CVE-2019-1161"},{"name":"CVE-2017-8542"},{"name":"CVE-2017-8539"},{"name":"CVE-2017-8537"},{"name":"CVE-2017-8536"},{"name":"CVE-2017-8535"},{"name":"CVE-2008-1438"},{"name":"CVE-2008-1437"}]}, {"discoverable_guid":"ADF7E72A-4A72-4D92-B278-F644E27EA88F","component_cpe":{"cpe23":"cpe:2.3:a:microsoft:.net_framework:4.8.04084"},"cve":[{"name":"CVE-2020-0646"},{"name":"CVE-2020-0606"},{"name":"CVE-2020-0605"},{"name":"CVE-2020-1147"},{"name":"CVE-2022-26832"},{"name":"CVE-2021-24111"},{"name":"CVE-2020-1108"},{"name":"CVE-2019-1083"},{"name":"CVE-2019-1006"},{"name":"CVE-2019-0981"},{"name":"CVE-2019-0980"},{"name":"CVE-2019-0820"},{"name":"CVE-2023-36873"},{"name":"CVE-2022-41064"},{"name":"CVE-2020-16937"},{"name":"CVE-2020-1476"},{"name":"CVE-2019-0864"},{"name":"CVE-2022-30130"}]}, {"discoverable_guid":"2B933591-6192-4E42-9DFC-32C361D32208","component_cpe":{"cpe23":"cpe:2.3:a:oracle:jdk\\/sdk:1.8.0_261"},"cve":[]} ]}

1. There are certain cases where nested json is rendered in splunk with single quotes (') instead of double-quotes("):

which makes me have to use a

| rex mode=sed field=<field_with_nested_json> "s/\'/\"/g"

to make it compatible with spath.

~~2. The "autoextract=0" option when pulling down json does not put the contents into a _raw field (as stated in your docs), but instead seems to do first-level extraction -~~

~~So a page that contains the following json:~~ EDIT - covered in #3 below

Renders looking like this when I use getwatchlist json <url> autoextract=0

3. ~~the "dataKey" parameter~~ All of the parameters seem to be case-sensitive - "dataKey=rows" produces correct content (below) vs "datakey=rows", which seems to ignore the parameter entirely

4. your docs don't seem to match the feature set or version in all places -

Splunkbase "details" tab	still refers to 1.3.2
Add-on "About" tab (after install)	refers to 1.3.3, but does not include details of the url parsing features that can only be found in your release notes on Splunkbase

5. The flattenJson parameter does not seem to be working at all. I find references to it in the code, but if I put it into the search as a parameter Splunk does not recognize it as such, but it also does not treat it as a custom field either.

As I said above, this add-on is great work, and literally the only things I could ask for "extra" are maybe xml parsing, and being able to perhaps pass URL parameters as an array.

EDIT:
A little more testing made me realize that a lot of my problems are specific to capitalization of the command parameters. I've edited #3 above

topic PAVO Getwatchlist - nested array parsing and other woes in Getting Data In

PAVO Getwatchlist - nested array parsing and other woes