Splunk not extracting JSON from RAPID7 InsightVM Properly

st1cky — Tue, 09 May 2023 13:45:36 GMT

Hello,

I am running into a bit of a challenge getting the data from the Rapid7 InsightVM TA to extract properly. The data in question is a nested JSON structure-- when I run a search on the data, the object appears to visualize properly, however the extracted fields are a mess and result in the data being unsearchable. I would expect that the nested fields get extracted into their own keys and values, however instead they get extracted into field* and in most cases those fields are comprised of a value from another key and the key itself.

Environment is distributed-- the TA is deployed on the HF ( which forwards to a UF forwarding tier, no TA there), Indexer Cluster, and Search Head cluster. The App is deployed on the SH cluster only.

Props.conf (HF, Indexer Cluster, SHC):

[source::...ta-rapid7-insightvm*.log*]
SHOULD_LINEMERGE = true
sourcetype = tarapid7insightvm:log

[source::...ta_rapid7_insightvm*.log*]
SHOULD_LINEMERGE = true
sourcetype = tarapid7insightvm:log

[rapid7:insightvm:asset]
FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_1 = host_name AS name
FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_2 = os_description AS version
FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_3 = os_description AS os
FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_4 = os_architecture AS family
FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_5 = os_system_name AS vendor_product
SHOULD_LINEMERGE = 0
pulldown_type = 1
DATETIME_CONFIG = CURRENT

[rapid7:insightvm:asset:vulnerability_finding]
FIELDALIAS-aob_gen_rapid7:insightvm:asset:vulnerability_finding_alias_1 = asset_hostname AS name
FIELDALIAS-aob_gen_rapid7:insightvm:asset:vulnerability_finding_alias_2 = asset_ip AS ip
FIELDALIAS-aob_gen_rapid7:insightvm:asset:vulnerability_finding_alias_3 = asset_ip AS dest
SHOULD_LINEMERGE = 0
pulldown_type = 1
TRUNCATE = 50000
DATETIME_CONFIG = CURRENT

[rapid7:insightvm:vulnerability_definition]
FIELDALIAS-aob_gen_rapid7:insightvm:vulnerability_definition_alias_1 = categories AS category
FIELDALIAS-aob_gen_rapid7:insightvm:vulnerability_definition_alias_2 = cves AS cve
SHOULD_LINEMERGE = 0
pulldown_type = 1
TRUNCATE = 50000

btool output from SH:

/opt/splunk/bin/splunk btool props list rapid7:insightvm:asset --debug
/opt/splunk/etc/apps/search/local/props.conf [rapid7:insightvm:asset]
/opt/splunk/etc/system/default/props.conf ADD_EXTRA_TIME_FIELDS = True
/opt/splunk/etc/system/default/props.conf ANNOTATE_PUNCT = True
/opt/splunk/etc/system/default/props.conf AUTO_KV_JSON = true
/opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE =
/opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE_DATE = True
/opt/splunk/etc/system/default/props.conf CHARSET = UTF-8
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf DATETIME_CONFIG = CURRENT
/opt/splunk/etc/system/default/props.conf DEPTH_LIMIT = 1000
/opt/splunk/etc/system/default/props.conf DETERMINE_TIMESTAMP_DATE_WITH_SYSTEM_TIME = false
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_1 = host_name AS name
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_2 = os_description AS version
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_3 = os_description AS os
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_4 = os_architecture AS family
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_5 = os_system_name AS vendor_product
/opt/splunk/etc/system/default/props.conf HEADER_MODE =
/opt/splunk/etc/system/default/props.conf LB_CHUNK_BREAKER_TRUNCATE = 2000000
/opt/splunk/etc/system/default/props.conf LEARN_MODEL = true
/opt/splunk/etc/system/default/props.conf LEARN_SOURCETYPE = true
/opt/splunk/etc/system/default/props.conf LINE_BREAKER_LOOKBEHIND = 100
/opt/splunk/etc/system/default/props.conf MATCH_LIMIT = 100000
/opt/splunk/etc/system/default/props.conf MAX_DAYS_AGO = 2000
/opt/splunk/etc/system/default/props.conf MAX_DAYS_HENCE = 2
/opt/splunk/etc/system/default/props.conf MAX_DIFF_SECS_AGO = 3600
/opt/splunk/etc/system/default/props.conf MAX_DIFF_SECS_HENCE = 604800
/opt/splunk/etc/system/default/props.conf MAX_EVENTS = 256
/opt/splunk/etc/system/default/props.conf MAX_TIMESTAMP_LOOKAHEAD = 128
/opt/splunk/etc/system/default/props.conf MUST_BREAK_AFTER =
/opt/splunk/etc/system/default/props.conf MUST_NOT_BREAK_AFTER =
/opt/splunk/etc/system/default/props.conf MUST_NOT_BREAK_BEFORE =
/opt/splunk/etc/apps/search/local/props.conf REPORT-rapid7 = REPORT-rapid7
/opt/splunk/etc/apps/search/local/props.conf REPORT-vm-cred-check = REPORT-vm-cred-check
/opt/splunk/etc/system/default/props.conf SEGMENTATION = indexing
/opt/splunk/etc/system/default/props.conf SEGMENTATION-all = full
/opt/splunk/etc/system/default/props.conf SEGMENTATION-inner = inner
/opt/splunk/etc/system/default/props.conf SEGMENTATION-outer = outer
/opt/splunk/etc/system/default/props.conf SEGMENTATION-raw = none
/opt/splunk/etc/system/default/props.conf SEGMENTATION-standard = standard
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf SHOULD_LINEMERGE = 0
/opt/splunk/etc/system/default/props.conf TRANSFORMS =
/opt/splunk/etc/system/default/props.conf TRUNCATE = 10000
/opt/splunk/etc/system/default/props.conf detect_trailing_nulls = false
/opt/splunk/etc/system/default/props.conf maxDist = 100
/opt/splunk/etc/system/default/props.conf priority =
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf pulldown_type = 1
/opt/splunk/etc/system/default/props.conf sourcetype =
/opt/splunk/etc/system/default/props.conf termFrequencyWeightedDist = false
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf [rapid7:insightvm:asset:vulnerability_finding]
/opt/splunk/etc/system/default/props.conf ADD_EXTRA_TIME_FIELDS = True
/opt/splunk/etc/system/default/props.conf ANNOTATE_PUNCT = True
/opt/splunk/etc/system/default/props.conf AUTO_KV_JSON = true
/opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE =
/opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE_DATE = True
/opt/splunk/etc/system/default/props.conf CHARSET = UTF-8
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf DATETIME_CONFIG = CURRENT
/opt/splunk/etc/system/default/props.conf DEPTH_LIMIT = 1000
/opt/splunk/etc/system/default/props.conf DETERMINE_TIMESTAMP_DATE_WITH_SYSTEM_TIME = false
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf FIELDALIAS-aob_gen_rapid7:insightvm:asset:vulnerability_finding_alias_1 = asset_hostname AS name
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf FIELDALIAS-aob_gen_rapid7:insightvm:asset:vulnerability_finding_alias_2 = asset_ip AS ip
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf FIELDALIAS-aob_gen_rapid7:insightvm:asset:vulnerability_finding_alias_3 = asset_ip AS dest
/opt/splunk/etc/system/default/props.conf HEADER_MODE =
/opt/splunk/etc/system/default/props.conf LB_CHUNK_BREAKER_TRUNCATE = 2000000
/opt/splunk/etc/system/default/props.conf LEARN_MODEL = true
/opt/splunk/etc/system/default/props.conf LEARN_SOURCETYPE = true
/opt/splunk/etc/system/default/props.conf LINE_BREAKER_LOOKBEHIND = 100
/opt/splunk/etc/system/default/props.conf MATCH_LIMIT = 100000
/opt/splunk/etc/system/default/props.conf MAX_DAYS_AGO = 2000
/opt/splunk/etc/system/default/props.conf MAX_DAYS_HENCE = 2
/opt/splunk/etc/system/default/props.conf MAX_DIFF_SECS_AGO = 3600
/opt/splunk/etc/system/default/props.conf MAX_DIFF_SECS_HENCE = 604800
/opt/splunk/etc/system/default/props.conf MAX_EVENTS = 256
/opt/splunk/etc/system/default/props.conf MAX_TIMESTAMP_LOOKAHEAD = 128
/opt/splunk/etc/system/default/props.conf MUST_BREAK_AFTER =
/opt/splunk/etc/system/default/props.conf MUST_NOT_BREAK_AFTER =
/opt/splunk/etc/system/default/props.conf MUST_NOT_BREAK_BEFORE =
/opt/splunk/etc/system/default/props.conf SEGMENTATION = indexing
/opt/splunk/etc/system/default/props.conf SEGMENTATION-all = full
/opt/splunk/etc/system/default/props.conf SEGMENTATION-inner = inner
/opt/splunk/etc/system/default/props.conf SEGMENTATION-outer = outer
/opt/splunk/etc/system/default/props.conf SEGMENTATION-raw = none
/opt/splunk/etc/system/default/props.conf SEGMENTATION-standard = standard
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf SHOULD_LINEMERGE = 0
/opt/splunk/etc/system/default/props.conf TRANSFORMS =
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf TRUNCATE = 50000
/opt/splunk/etc/system/default/props.conf detect_trailing_nulls = false
/opt/splunk/etc/system/default/props.conf maxDist = 100
/opt/splunk/etc/system/default/props.conf priority =
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf pulldown_type = 1
/opt/splunk/etc/system/default/props.conf sourcetype =
/opt/splunk/etc/system/default/props.conf termFrequencyWeightedDist = false

This appears to only be happening with the rapid7:insightvm:asset sourcetype, however that is the type that I have primarily been focused on while trying to get things working. It may be happening in the other sourcetypes however I have not yet seen it in my limited time experimenting with that data.

Any help is much appreciated, thank you!

Re: Splunk ont extracting JSON from RAPID7 InsightVM Properly

st1cky — Tue, 09 May 2023 13:22:48 GMT

Also, the raw event text:

topic Re: Splunk ont extracting JSON from RAPID7 InsightVM Properly in Getting Data In

Splunk not extracting JSON from RAPID7 InsightVM Properly

Re: Splunk ont extracting JSON from RAPID7 InsightVM Properly