Getting Data In

PAVO Getwatchlist - nested array parsing and other woes

K_Sukumar
Loves-to-Learn

Good day,

First I want to say that this add-on is an absolute lifesaver when it comes to getting structured data into Splunk, and if you ever put it up on GitHub please let me know - I'd be happy to contribute.

I have found a few minor issues.  I'll be using the following json in my examples:

 

 

{"total":52145,"rows":
[
{"discoverable_guid":"94937859-A157-4C43-94AC-290172D50C4D","component_cpe":{"cpe23":"cpe:2.3:a:oracle:java_runtime_environment:1.8.0_381"},"cve":[]},
{"discoverable_guid":"2B933591-6192-4E42-9DFC-32C361D32208","component_cpe":{"cpe23":"cpe:2.3:a:oracle:jdk\\/sdk:1.8.0_201"},"cve":[]},
{"discoverable_guid":"DD854B8C-5900-518C-B8B6-096285936816","component_cpe":{"cpe23":"cpe:2.3:o:microsoft:windows_defender:4.18.1909.6"},"cve":[{"name":"CVE-2006-5270"},{"name":"CVE-2018-0986"},{"name":"CVE-2021-24092"},{"name":"CVE-2021-1647"},{"name":"CVE-2020-1170"},{"name":"CVE-2020-1163"},{"name":"CVE-2020-0835"},{"name":"CVE-2017-8558"},{"name":"CVE-2017-8541"},{"name":"CVE-2017-8540"},{"name":"CVE-2017-8538"},{"name":"CVE-2017-0290"},{"name":"CVE-2019-1255"},{"name":"CVE-2013-0078"},{"name":"CVE-2011-0037"},{"name":"CVE-2020-1461"},{"name":"CVE-2020-1002"},{"name":"CVE-2019-1161"},{"name":"CVE-2017-8542"},{"name":"CVE-2017-8539"},{"name":"CVE-2017-8537"},{"name":"CVE-2017-8536"},{"name":"CVE-2017-8535"},{"name":"CVE-2008-1438"},{"name":"CVE-2008-1437"}]},
{"discoverable_guid":"ADF7E72A-4A72-4D92-B278-F644E27EA88F","component_cpe":{"cpe23":"cpe:2.3:a:microsoft:.net_framework:4.8.04084"},"cve":[{"name":"CVE-2020-0646"},{"name":"CVE-2020-0606"},{"name":"CVE-2020-0605"},{"name":"CVE-2020-1147"},{"name":"CVE-2022-26832"},{"name":"CVE-2021-24111"},{"name":"CVE-2020-1108"},{"name":"CVE-2019-1083"},{"name":"CVE-2019-1006"},{"name":"CVE-2019-0981"},{"name":"CVE-2019-0980"},{"name":"CVE-2019-0820"},{"name":"CVE-2023-36873"},{"name":"CVE-2022-41064"},{"name":"CVE-2020-16937"},{"name":"CVE-2020-1476"},{"name":"CVE-2019-0864"},{"name":"CVE-2022-30130"}]},
{"discoverable_guid":"2B933591-6192-4E42-9DFC-32C361D32208","component_cpe":{"cpe23":"cpe:2.3:a:oracle:jdk\\/sdk:1.8.0_261"},"cve":[]}
]} 

 

 


1. There are certain cases where nested json is rendered in splunk with  single quotes (') instead of double-quotes("):

K_Sukumar_1-1705929951046.png

which makes me have to use a 

 

 

| rex mode=sed field=<field_with_nested_json> "s/\'/\"/g"

 

 

to make it compatible with spath.

2. The "autoextract=0" option when pulling down json does not put the contents into a _raw field (as stated in your docs), but instead seems to do first-level extraction - 

So a page that contains the following json:    EDIT - covered in #3 below

Renders looking like this when I use getwatchlist json <url> autoextract=0

K_Sukumar_0-1705929552322.png

3.  the "dataKey" parameter All of the parameters seem to be case-sensitive - "dataKey=rows" produces correct content (below) vs "datakey=rows", which seems to ignore the parameter entirely

4. your docs don't seem to match the feature set or version in all places -

Splunkbase "details" tabstill refers to 1.3.2
Add-on "About" tab (after install)refers to 1.3.3, but does not include details of the url parsing features that can only be found in your release notes on Splunkbase


5. The flattenJson parameter does not seem to be working at all.  I find references to it in the code, but if I put it into the search as a parameter Splunk does not recognize it as such, but it also does not treat it as a custom field either.

As I said above, this add-on is great work, and literally the only things I could ask for "extra" are maybe xml parsing, and being able to perhaps pass URL parameters as an array.

EDIT:
A little more testing made me realize that a lot of my problems are specific to capitalization of the command parameters.  I've edited #3 above

Labels (3)
Tags (2)
0 Karma
Get Updates on the Splunk Community!

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

SignalFlow: What? Why? How?

What is SignalFlow? Splunk Observability Cloud’s analytics engine, SignalFlow, opens up a world of in-depth ...

Federated Search for Amazon S3 | Key Use Cases to Streamline Compliance Workflows

Modern business operations are supported by data compliance. As regulations evolve, organizations must ...