Getting Data In

Splunk not extracting JSON from RAPID7 InsightVM Properly

st1cky
Loves-to-Learn Lots

Hello,

 

I am running into a bit of a challenge getting the data from the Rapid7 InsightVM TA to extract properly. The data in question is a nested JSON structure-- when I run a search on the data, the object appears to visualize properly, however the extracted fields are a mess and result in the data being unsearchable. I would expect that the nested fields get extracted into their own keys and values, however instead they get extracted into field* and in most cases those fields are comprised of a value from another key and the key itself. 

st1cky_0-1683637616783.pngst1cky_1-1683637726512.png

 

 

Environment is distributed-- the TA is deployed on the HF ( which forwards to a UF forwarding tier, no TA there), Indexer Cluster, and Search Head cluster. The App is deployed on the SH cluster only.  

Props.conf (HF, Indexer Cluster, SHC):

[source::...ta-rapid7-insightvm*.log*]
SHOULD_LINEMERGE = true
sourcetype = tarapid7insightvm:log

[source::...ta_rapid7_insightvm*.log*]
SHOULD_LINEMERGE = true
sourcetype = tarapid7insightvm:log

[rapid7:insightvm:asset]
FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_1 = host_name AS name
FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_2 = os_description AS version
FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_3 = os_description AS os
FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_4 = os_architecture AS family
FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_5 = os_system_name AS vendor_product
SHOULD_LINEMERGE = 0
pulldown_type = 1
DATETIME_CONFIG = CURRENT

[rapid7:insightvm:asset:vulnerability_finding]
FIELDALIAS-aob_gen_rapid7:insightvm:asset:vulnerability_finding_alias_1 = asset_hostname AS name
FIELDALIAS-aob_gen_rapid7:insightvm:asset:vulnerability_finding_alias_2 = asset_ip AS ip
FIELDALIAS-aob_gen_rapid7:insightvm:asset:vulnerability_finding_alias_3 = asset_ip AS dest
SHOULD_LINEMERGE = 0
pulldown_type = 1
TRUNCATE = 50000
DATETIME_CONFIG = CURRENT

[rapid7:insightvm:vulnerability_definition]
FIELDALIAS-aob_gen_rapid7:insightvm:vulnerability_definition_alias_1 = categories AS category
FIELDALIAS-aob_gen_rapid7:insightvm:vulnerability_definition_alias_2 = cves AS cve
SHOULD_LINEMERGE = 0
pulldown_type = 1
TRUNCATE = 50000


 

btool output from SH:

/opt/splunk/bin/splunk btool props list rapid7:insightvm:asset --debug
/opt/splunk/etc/apps/search/local/props.conf [rapid7:insightvm:asset]
/opt/splunk/etc/system/default/props.conf ADD_EXTRA_TIME_FIELDS = True
/opt/splunk/etc/system/default/props.conf ANNOTATE_PUNCT = True
/opt/splunk/etc/system/default/props.conf AUTO_KV_JSON = true
/opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE =
/opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE_DATE = True
/opt/splunk/etc/system/default/props.conf CHARSET = UTF-8
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf DATETIME_CONFIG = CURRENT
/opt/splunk/etc/system/default/props.conf DEPTH_LIMIT = 1000
/opt/splunk/etc/system/default/props.conf DETERMINE_TIMESTAMP_DATE_WITH_SYSTEM_TIME = false
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_1 = host_name AS name
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_2 = os_description AS version
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_3 = os_description AS os
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_4 = os_architecture AS family
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_5 = os_system_name AS vendor_product
/opt/splunk/etc/system/default/props.conf HEADER_MODE =
/opt/splunk/etc/system/default/props.conf LB_CHUNK_BREAKER_TRUNCATE = 2000000
/opt/splunk/etc/system/default/props.conf LEARN_MODEL = true
/opt/splunk/etc/system/default/props.conf LEARN_SOURCETYPE = true
/opt/splunk/etc/system/default/props.conf LINE_BREAKER_LOOKBEHIND = 100
/opt/splunk/etc/system/default/props.conf MATCH_LIMIT = 100000
/opt/splunk/etc/system/default/props.conf MAX_DAYS_AGO = 2000
/opt/splunk/etc/system/default/props.conf MAX_DAYS_HENCE = 2
/opt/splunk/etc/system/default/props.conf MAX_DIFF_SECS_AGO = 3600
/opt/splunk/etc/system/default/props.conf MAX_DIFF_SECS_HENCE = 604800
/opt/splunk/etc/system/default/props.conf MAX_EVENTS = 256
/opt/splunk/etc/system/default/props.conf MAX_TIMESTAMP_LOOKAHEAD = 128
/opt/splunk/etc/system/default/props.conf MUST_BREAK_AFTER =
/opt/splunk/etc/system/default/props.conf MUST_NOT_BREAK_AFTER =
/opt/splunk/etc/system/default/props.conf MUST_NOT_BREAK_BEFORE =
/opt/splunk/etc/apps/search/local/props.conf REPORT-rapid7 = REPORT-rapid7
/opt/splunk/etc/apps/search/local/props.conf REPORT-vm-cred-check = REPORT-vm-cred-check
/opt/splunk/etc/system/default/props.conf SEGMENTATION = indexing
/opt/splunk/etc/system/default/props.conf SEGMENTATION-all = full
/opt/splunk/etc/system/default/props.conf SEGMENTATION-inner = inner
/opt/splunk/etc/system/default/props.conf SEGMENTATION-outer = outer
/opt/splunk/etc/system/default/props.conf SEGMENTATION-raw = none
/opt/splunk/etc/system/default/props.conf SEGMENTATION-standard = standard
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf SHOULD_LINEMERGE = 0
/opt/splunk/etc/system/default/props.conf TRANSFORMS =
/opt/splunk/etc/system/default/props.conf TRUNCATE = 10000
/opt/splunk/etc/system/default/props.conf detect_trailing_nulls = false
/opt/splunk/etc/system/default/props.conf maxDist = 100
/opt/splunk/etc/system/default/props.conf priority =
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf pulldown_type = 1
/opt/splunk/etc/system/default/props.conf sourcetype =
/opt/splunk/etc/system/default/props.conf termFrequencyWeightedDist = false
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf [rapid7:insightvm:asset:vulnerability_finding]
/opt/splunk/etc/system/default/props.conf ADD_EXTRA_TIME_FIELDS = True
/opt/splunk/etc/system/default/props.conf ANNOTATE_PUNCT = True
/opt/splunk/etc/system/default/props.conf AUTO_KV_JSON = true
/opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE =
/opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE_DATE = True
/opt/splunk/etc/system/default/props.conf CHARSET = UTF-8
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf DATETIME_CONFIG = CURRENT
/opt/splunk/etc/system/default/props.conf DEPTH_LIMIT = 1000
/opt/splunk/etc/system/default/props.conf DETERMINE_TIMESTAMP_DATE_WITH_SYSTEM_TIME = false
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf FIELDALIAS-aob_gen_rapid7:insightvm:asset:vulnerability_finding_alias_1 = asset_hostname AS name
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf FIELDALIAS-aob_gen_rapid7:insightvm:asset:vulnerability_finding_alias_2 = asset_ip AS ip
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf FIELDALIAS-aob_gen_rapid7:insightvm:asset:vulnerability_finding_alias_3 = asset_ip AS dest
/opt/splunk/etc/system/default/props.conf HEADER_MODE =
/opt/splunk/etc/system/default/props.conf LB_CHUNK_BREAKER_TRUNCATE = 2000000
/opt/splunk/etc/system/default/props.conf LEARN_MODEL = true
/opt/splunk/etc/system/default/props.conf LEARN_SOURCETYPE = true
/opt/splunk/etc/system/default/props.conf LINE_BREAKER_LOOKBEHIND = 100
/opt/splunk/etc/system/default/props.conf MATCH_LIMIT = 100000
/opt/splunk/etc/system/default/props.conf MAX_DAYS_AGO = 2000
/opt/splunk/etc/system/default/props.conf MAX_DAYS_HENCE = 2
/opt/splunk/etc/system/default/props.conf MAX_DIFF_SECS_AGO = 3600
/opt/splunk/etc/system/default/props.conf MAX_DIFF_SECS_HENCE = 604800
/opt/splunk/etc/system/default/props.conf MAX_EVENTS = 256
/opt/splunk/etc/system/default/props.conf MAX_TIMESTAMP_LOOKAHEAD = 128
/opt/splunk/etc/system/default/props.conf MUST_BREAK_AFTER =
/opt/splunk/etc/system/default/props.conf MUST_NOT_BREAK_AFTER =
/opt/splunk/etc/system/default/props.conf MUST_NOT_BREAK_BEFORE =
/opt/splunk/etc/system/default/props.conf SEGMENTATION = indexing
/opt/splunk/etc/system/default/props.conf SEGMENTATION-all = full
/opt/splunk/etc/system/default/props.conf SEGMENTATION-inner = inner
/opt/splunk/etc/system/default/props.conf SEGMENTATION-outer = outer
/opt/splunk/etc/system/default/props.conf SEGMENTATION-raw = none
/opt/splunk/etc/system/default/props.conf SEGMENTATION-standard = standard
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf SHOULD_LINEMERGE = 0
/opt/splunk/etc/system/default/props.conf TRANSFORMS =
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf TRUNCATE = 50000
/opt/splunk/etc/system/default/props.conf detect_trailing_nulls = false
/opt/splunk/etc/system/default/props.conf maxDist = 100
/opt/splunk/etc/system/default/props.conf priority =
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf pulldown_type = 1
/opt/splunk/etc/system/default/props.conf sourcetype =
/opt/splunk/etc/system/default/props.conf termFrequencyWeightedDist = false


This appears to only be happening with the rapid7:insightvm:asset sourcetype, however that is the type that I have primarily been focused on while trying to get things working. It may be happening in the other sourcetypes however I have not yet seen it in my limited time experimenting with that data. 

 

Any help is much appreciated, thank you!

Labels (1)
0 Karma

st1cky
Loves-to-Learn Lots

Also, the raw event text:

st1cky_0-1683638561675.png

 

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...