Why is Splunk not extracting Rapid7 JSON data correctly?

st1cky — Tue, 09 May 2023 16:37:03 GMT

Hello,

I am running into a bit of a challenge getting the data from the Rapid7 InsightVM TA to extract properly. The data in question is a nested JSON structure-- when I run a search on the data, the object appears to visualize properly, however the extracted fields are a mess and result in the data being unsearchable. I would expect that the nested fields get extracted into their own keys and values, however instead they get extracted into field* and in most cases those fields are comprised of a value from another key and the key itself.

Environment is distributed-- the TA is deployed on the HF ( which forwards to a UF forwarding tier, no TA there), Indexer Cluster, and Search Head cluster. The App is deployed on the SH cluster only.

Props.conf (HF, Indexer Cluster, SHC):

[source::...ta-rapid7-insightvm*.log*]
SHOULD_LINEMERGE = true
sourcetype = tarapid7insightvm:log

[source::...ta_rapid7_insightvm*.log*]
SHOULD_LINEMERGE = true
sourcetype = tarapid7insightvm:log

[rapid7:insightvm:asset]
FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_1 = host_name AS name
FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_2 = os_description AS version
FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_3 = os_description AS os
FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_4 = os_architecture AS family
FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_5 = os_system_name AS vendor_product
SHOULD_LINEMERGE = 0
pulldown_type = 1
DATETIME_CONFIG = CURRENT

[rapid7:insightvm:asset:vulnerability_finding]
FIELDALIAS-aob_gen_rapid7:insightvm:asset:vulnerability_finding_alias_1 = asset_hostname AS name
FIELDALIAS-aob_gen_rapid7:insightvm:asset:vulnerability_finding_alias_2 = asset_ip AS ip
FIELDALIAS-aob_gen_rapid7:insightvm:asset:vulnerability_finding_alias_3 = asset_ip AS dest
SHOULD_LINEMERGE = 0
pulldown_type = 1
TRUNCATE = 50000
DATETIME_CONFIG = CURRENT

[rapid7:insightvm:vulnerability_definition]
FIELDALIAS-aob_gen_rapid7:insightvm:vulnerability_definition_alias_1 = categories AS category
FIELDALIAS-aob_gen_rapid7:insightvm:vulnerability_definition_alias_2 = cves AS cve
SHOULD_LINEMERGE = 0
pulldown_type = 1
TRUNCATE = 50000

btool output from SH:

/opt/splunk/bin/splunk btool props list rapid7:insightvm:asset --debug
/opt/splunk/etc/apps/search/local/props.conf [rapid7:insightvm:asset]
/opt/splunk/etc/system/default/props.conf ADD_EXTRA_TIME_FIELDS = True
/opt/splunk/etc/system/default/props.conf ANNOTATE_PUNCT = True
/opt/splunk/etc/system/default/props.conf AUTO_KV_JSON = true
/opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE =
/opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE_DATE = True
/opt/splunk/etc/system/default/props.conf CHARSET = UTF-8
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf DATETIME_CONFIG = CURRENT
/opt/splunk/etc/system/default/props.conf DEPTH_LIMIT = 1000
/opt/splunk/etc/system/default/props.conf DETERMINE_TIMESTAMP_DATE_WITH_SYSTEM_TIME = false
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_1 = host_name AS name
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_2 = os_description AS version
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_3 = os_description AS os
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_4 = os_architecture AS family
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_5 = os_system_name AS vendor_product
/opt/splunk/etc/system/default/props.conf HEADER_MODE =
/opt/splunk/etc/system/default/props.conf LB_CHUNK_BREAKER_TRUNCATE = 2000000
/opt/splunk/etc/system/default/props.conf LEARN_MODEL = true
/opt/splunk/etc/system/default/props.conf LEARN_SOURCETYPE = true
/opt/splunk/etc/system/default/props.conf LINE_BREAKER_LOOKBEHIND = 100
/opt/splunk/etc/system/default/props.conf MATCH_LIMIT = 100000
/opt/splunk/etc/system/default/props.conf MAX_DAYS_AGO = 2000
/opt/splunk/etc/system/default/props.conf MAX_DAYS_HENCE = 2
/opt/splunk/etc/system/default/props.conf MAX_DIFF_SECS_AGO = 3600
/opt/splunk/etc/system/default/props.conf MAX_DIFF_SECS_HENCE = 604800
/opt/splunk/etc/system/default/props.conf MAX_EVENTS = 256
/opt/splunk/etc/system/default/props.conf MAX_TIMESTAMP_LOOKAHEAD = 128
/opt/splunk/etc/system/default/props.conf MUST_BREAK_AFTER =
/opt/splunk/etc/system/default/props.conf MUST_NOT_BREAK_AFTER =
/opt/splunk/etc/system/default/props.conf MUST_NOT_BREAK_BEFORE =
/opt/splunk/etc/apps/search/local/props.conf REPORT-rapid7 = REPORT-rapid7
/opt/splunk/etc/apps/search/local/props.conf REPORT-vm-cred-check = REPORT-vm-cred-check
/opt/splunk/etc/system/default/props.conf SEGMENTATION = indexing
/opt/splunk/etc/system/default/props.conf SEGMENTATION-all = full
/opt/splunk/etc/system/default/props.conf SEGMENTATION-inner = inner
/opt/splunk/etc/system/default/props.conf SEGMENTATION-outer = outer
/opt/splunk/etc/system/default/props.conf SEGMENTATION-raw = none
/opt/splunk/etc/system/default/props.conf SEGMENTATION-standard = standard
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf SHOULD_LINEMERGE = 0
/opt/splunk/etc/system/default/props.conf TRANSFORMS =
/opt/splunk/etc/system/default/props.conf TRUNCATE = 10000
/opt/splunk/etc/system/default/props.conf detect_trailing_nulls = false
/opt/splunk/etc/system/default/props.conf maxDist = 100
/opt/splunk/etc/system/default/props.conf priority =
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf pulldown_type = 1
/opt/splunk/etc/system/default/props.conf sourcetype =
/opt/splunk/etc/system/default/props.conf termFrequencyWeightedDist = false
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf [rapid7:insightvm:asset:vulnerability_finding]
/opt/splunk/etc/system/default/props.conf ADD_EXTRA_TIME_FIELDS = True
/opt/splunk/etc/system/default/props.conf ANNOTATE_PUNCT = True
/opt/splunk/etc/system/default/props.conf AUTO_KV_JSON = true
/opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE =
/opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE_DATE = True
/opt/splunk/etc/system/default/props.conf CHARSET = UTF-8
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf DATETIME_CONFIG = CURRENT
/opt/splunk/etc/system/default/props.conf DEPTH_LIMIT = 1000
/opt/splunk/etc/system/default/props.conf DETERMINE_TIMESTAMP_DATE_WITH_SYSTEM_TIME = false
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf FIELDALIAS-aob_gen_rapid7:insightvm:asset:vulnerability_finding_alias_1 = asset_hostname AS name
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf FIELDALIAS-aob_gen_rapid7:insightvm:asset:vulnerability_finding_alias_2 = asset_ip AS ip
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf FIELDALIAS-aob_gen_rapid7:insightvm:asset:vulnerability_finding_alias_3 = asset_ip AS dest
/opt/splunk/etc/system/default/props.conf HEADER_MODE =
/opt/splunk/etc/system/default/props.conf LB_CHUNK_BREAKER_TRUNCATE = 2000000
/opt/splunk/etc/system/default/props.conf LEARN_MODEL = true
/opt/splunk/etc/system/default/props.conf LEARN_SOURCETYPE = true
/opt/splunk/etc/system/default/props.conf LINE_BREAKER_LOOKBEHIND = 100
/opt/splunk/etc/system/default/props.conf MATCH_LIMIT = 100000
/opt/splunk/etc/system/default/props.conf MAX_DAYS_AGO = 2000
/opt/splunk/etc/system/default/props.conf MAX_DAYS_HENCE = 2
/opt/splunk/etc/system/default/props.conf MAX_DIFF_SECS_AGO = 3600
/opt/splunk/etc/system/default/props.conf MAX_DIFF_SECS_HENCE = 604800
/opt/splunk/etc/system/default/props.conf MAX_EVENTS = 256
/opt/splunk/etc/system/default/props.conf MAX_TIMESTAMP_LOOKAHEAD = 128
/opt/splunk/etc/system/default/props.conf MUST_BREAK_AFTER =
/opt/splunk/etc/system/default/props.conf MUST_NOT_BREAK_AFTER =
/opt/splunk/etc/system/default/props.conf MUST_NOT_BREAK_BEFORE =
/opt/splunk/etc/system/default/props.conf SEGMENTATION = indexing
/opt/splunk/etc/system/default/props.conf SEGMENTATION-all = full
/opt/splunk/etc/system/default/props.conf SEGMENTATION-inner = inner
/opt/splunk/etc/system/default/props.conf SEGMENTATION-outer = outer
/opt/splunk/etc/system/default/props.conf SEGMENTATION-raw = none
/opt/splunk/etc/system/default/props.conf SEGMENTATION-standard = standard
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf SHOULD_LINEMERGE = 0
/opt/splunk/etc/system/default/props.conf TRANSFORMS =
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf TRUNCATE = 50000
/opt/splunk/etc/system/default/props.conf detect_trailing_nulls = false
/opt/splunk/etc/system/default/props.conf maxDist = 100
/opt/splunk/etc/system/default/props.conf priority =
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf pulldown_type = 1
/opt/splunk/etc/system/default/props.conf sourcetype =
/opt/splunk/etc/system/default/props.conf termFrequencyWeightedDist = false

This appears to only be happening with the rapid7:insightvm:asset sourcetype, however that is the type that I have primarily been focused on while trying to get things working. It may be happening in the other sourcetypes however I have not yet seen it in my limited time experimenting with that data.

Any help is much appreciated, thank you!

topic Why is Splunk not extracting Rapid7 JSON data correctly? in Getting Data In

Why is Splunk not extracting Rapid7 JSON data correctly?