Getting Data In
×

Are you a member of the Splunk Community?

Sign in or Register with your Splunk account to get your questions answered, access valuable resources and connect with experts!
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Ask a Question

Splunk not extracting JSON from RAPID7 InsightVM Properly

st1cky
Loves-to-Learn Lots
‎05-09-2023 06:19 AM

Hello,

 

I am running into a bit of a challenge getting the data from the Rapid7 InsightVM TA to extract properly. The data in question is a nested JSON structure-- when I run a search on the data, the object appears to visualize properly, however the extracted fields are a mess and result in the data being unsearchable. I would expect that the nested fields get extracted into their own keys and values, however instead they get extracted into field* and in most cases those fields are comprised of a value from another key and the key itself. 

st1cky_0-1683637616783.pngst1cky_1-1683637726512.png

 

 

Environment is distributed-- the TA is deployed on the HF ( which forwards to a UF forwarding tier, no TA there), Indexer Cluster, and Search Head cluster. The App is deployed on the SH cluster only.  

Props.conf (HF, Indexer Cluster, SHC):

[source::...ta-rapid7-insightvm*.log*]
SHOULD_LINEMERGE = true
sourcetype = tarapid7insightvm:log

[source::...ta_rapid7_insightvm*.log*]
SHOULD_LINEMERGE = true
sourcetype = tarapid7insightvm:log

[rapid7:insightvm:asset]
FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_1 = host_name AS name
FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_2 = os_description AS version
FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_3 = os_description AS os
FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_4 = os_architecture AS family
FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_5 = os_system_name AS vendor_product
SHOULD_LINEMERGE = 0
pulldown_type = 1
DATETIME_CONFIG = CURRENT

[rapid7:insightvm:asset:vulnerability_finding]
FIELDALIAS-aob_gen_rapid7:insightvm:asset:vulnerability_finding_alias_1 = asset_hostname AS name
FIELDALIAS-aob_gen_rapid7:insightvm:asset:vulnerability_finding_alias_2 = asset_ip AS ip
FIELDALIAS-aob_gen_rapid7:insightvm:asset:vulnerability_finding_alias_3 = asset_ip AS dest
SHOULD_LINEMERGE = 0
pulldown_type = 1
TRUNCATE = 50000
DATETIME_CONFIG = CURRENT

[rapid7:insightvm:vulnerability_definition]
FIELDALIAS-aob_gen_rapid7:insightvm:vulnerability_definition_alias_1 = categories AS category
FIELDALIAS-aob_gen_rapid7:insightvm:vulnerability_definition_alias_2 = cves AS cve
SHOULD_LINEMERGE = 0
pulldown_type = 1
TRUNCATE = 50000


 

btool output from SH:

/opt/splunk/bin/splunk btool props list rapid7:insightvm:asset --debug
/opt/splunk/etc/apps/search/local/props.conf [rapid7:insightvm:asset]
/opt/splunk/etc/system/default/props.conf ADD_EXTRA_TIME_FIELDS = True
/opt/splunk/etc/system/default/props.conf ANNOTATE_PUNCT = True
/opt/splunk/etc/system/default/props.conf AUTO_KV_JSON = true
/opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE =
/opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE_DATE = True
/opt/splunk/etc/system/default/props.conf CHARSET = UTF-8
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf DATETIME_CONFIG = CURRENT
/opt/splunk/etc/system/default/props.conf DEPTH_LIMIT = 1000
/opt/splunk/etc/system/default/props.conf DETERMINE_TIMESTAMP_DATE_WITH_SYSTEM_TIME = false
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_1 = host_name AS name
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_2 = os_description AS version
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_3 = os_description AS os
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_4 = os_architecture AS family
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_5 = os_system_name AS vendor_product
/opt/splunk/etc/system/default/props.conf HEADER_MODE =
/opt/splunk/etc/system/default/props.conf LB_CHUNK_BREAKER_TRUNCATE = 2000000
/opt/splunk/etc/system/default/props.conf LEARN_MODEL = true
/opt/splunk/etc/system/default/props.conf LEARN_SOURCETYPE = true
/opt/splunk/etc/system/default/props.conf LINE_BREAKER_LOOKBEHIND = 100
/opt/splunk/etc/system/default/props.conf MATCH_LIMIT = 100000
/opt/splunk/etc/system/default/props.conf MAX_DAYS_AGO = 2000
/opt/splunk/etc/system/default/props.conf MAX_DAYS_HENCE = 2
/opt/splunk/etc/system/default/props.conf MAX_DIFF_SECS_AGO = 3600
/opt/splunk/etc/system/default/props.conf MAX_DIFF_SECS_HENCE = 604800
/opt/splunk/etc/system/default/props.conf MAX_EVENTS = 256
/opt/splunk/etc/system/default/props.conf MAX_TIMESTAMP_LOOKAHEAD = 128
/opt/splunk/etc/system/default/props.conf MUST_BREAK_AFTER =
/opt/splunk/etc/system/default/props.conf MUST_NOT_BREAK_AFTER =
/opt/splunk/etc/system/default/props.conf MUST_NOT_BREAK_BEFORE =
/opt/splunk/etc/apps/search/local/props.conf REPORT-rapid7 = REPORT-rapid7
/opt/splunk/etc/apps/search/local/props.conf REPORT-vm-cred-check = REPORT-vm-cred-check
/opt/splunk/etc/system/default/props.conf SEGMENTATION = indexing
/opt/splunk/etc/system/default/props.conf SEGMENTATION-all = full
/opt/splunk/etc/system/default/props.conf SEGMENTATION-inner = inner
/opt/splunk/etc/system/default/props.conf SEGMENTATION-outer = outer
/opt/splunk/etc/system/default/props.conf SEGMENTATION-raw = none
/opt/splunk/etc/system/default/props.conf SEGMENTATION-standard = standard
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf SHOULD_LINEMERGE = 0
/opt/splunk/etc/system/default/props.conf TRANSFORMS =
/opt/splunk/etc/system/default/props.conf TRUNCATE = 10000
/opt/splunk/etc/system/default/props.conf detect_trailing_nulls = false
/opt/splunk/etc/system/default/props.conf maxDist = 100
/opt/splunk/etc/system/default/props.conf priority =
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf pulldown_type = 1
/opt/splunk/etc/system/default/props.conf sourcetype =
/opt/splunk/etc/system/default/props.conf termFrequencyWeightedDist = false
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf [rapid7:insightvm:asset:vulnerability_finding]
/opt/splunk/etc/system/default/props.conf ADD_EXTRA_TIME_FIELDS = True
/opt/splunk/etc/system/default/props.conf ANNOTATE_PUNCT = True
/opt/splunk/etc/system/default/props.conf AUTO_KV_JSON = true
/opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE =
/opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE_DATE = True
/opt/splunk/etc/system/default/props.conf CHARSET = UTF-8
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf DATETIME_CONFIG = CURRENT
/opt/splunk/etc/system/default/props.conf DEPTH_LIMIT = 1000
/opt/splunk/etc/system/default/props.conf DETERMINE_TIMESTAMP_DATE_WITH_SYSTEM_TIME = false
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf FIELDALIAS-aob_gen_rapid7:insightvm:asset:vulnerability_finding_alias_1 = asset_hostname AS name
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf FIELDALIAS-aob_gen_rapid7:insightvm:asset:vulnerability_finding_alias_2 = asset_ip AS ip
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf FIELDALIAS-aob_gen_rapid7:insightvm:asset:vulnerability_finding_alias_3 = asset_ip AS dest
/opt/splunk/etc/system/default/props.conf HEADER_MODE =
/opt/splunk/etc/system/default/props.conf LB_CHUNK_BREAKER_TRUNCATE = 2000000
/opt/splunk/etc/system/default/props.conf LEARN_MODEL = true
/opt/splunk/etc/system/default/props.conf LEARN_SOURCETYPE = true
/opt/splunk/etc/system/default/props.conf LINE_BREAKER_LOOKBEHIND = 100
/opt/splunk/etc/system/default/props.conf MATCH_LIMIT = 100000
/opt/splunk/etc/system/default/props.conf MAX_DAYS_AGO = 2000
/opt/splunk/etc/system/default/props.conf MAX_DAYS_HENCE = 2
/opt/splunk/etc/system/default/props.conf MAX_DIFF_SECS_AGO = 3600
/opt/splunk/etc/system/default/props.conf MAX_DIFF_SECS_HENCE = 604800
/opt/splunk/etc/system/default/props.conf MAX_EVENTS = 256
/opt/splunk/etc/system/default/props.conf MAX_TIMESTAMP_LOOKAHEAD = 128
/opt/splunk/etc/system/default/props.conf MUST_BREAK_AFTER =
/opt/splunk/etc/system/default/props.conf MUST_NOT_BREAK_AFTER =
/opt/splunk/etc/system/default/props.conf MUST_NOT_BREAK_BEFORE =
/opt/splunk/etc/system/default/props.conf SEGMENTATION = indexing
/opt/splunk/etc/system/default/props.conf SEGMENTATION-all = full
/opt/splunk/etc/system/default/props.conf SEGMENTATION-inner = inner
/opt/splunk/etc/system/default/props.conf SEGMENTATION-outer = outer
/opt/splunk/etc/system/default/props.conf SEGMENTATION-raw = none
/opt/splunk/etc/system/default/props.conf SEGMENTATION-standard = standard
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf SHOULD_LINEMERGE = 0
/opt/splunk/etc/system/default/props.conf TRANSFORMS =
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf TRUNCATE = 50000
/opt/splunk/etc/system/default/props.conf detect_trailing_nulls = false
/opt/splunk/etc/system/default/props.conf maxDist = 100
/opt/splunk/etc/system/default/props.conf priority =
/opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf pulldown_type = 1
/opt/splunk/etc/system/default/props.conf sourcetype =
/opt/splunk/etc/system/default/props.conf termFrequencyWeightedDist = false


This appears to only be happening with the rapid7:insightvm:asset sourcetype, however that is the type that I have primarily been focused on while trying to get things working. It may be happening in the other sourcetypes however I have not yet seen it in my limited time experimenting with that data. 

 

Any help is much appreciated, thank you!

Labels (1)
Labels
0 Karma
Reply

st1cky
Loves-to-Learn Lots
‎05-09-2023 06:22 AM

Also, the raw event text:

st1cky_0-1683638561675.png

 

0 Karma
Reply
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.
Take the Survey

Can’t make it to .conf25? Join us online!

Watch Global Broadcast
Get Updates on the Splunk Community!

Can’t Make It to Boston? Stream .conf25 and Learn with Haya Husain

Boston may be buzzing this September with Splunk University and .conf25, but you don’t have to pack a bag to ...

Splunk Lantern’s Guide to The Most Popular .conf25 Sessions

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Unlock What’s Next: The Splunk Cloud Platform at .conf25

In just a few days, Boston will be buzzing as the Splunk team and thousands of community members come together ...