Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
- About st1cky
st1cky
Loves-to-Learn Lots
Member since:
11-10-2020
05-09-2023
Community Statistics
| Posts | 3 |
| Solutions | 0 |
| Karma Given | 0 |
| Karma Received | 0 |
| Member Since | 11-10-2020 |
Activity Feed
- Posted Why is Splunk not extracting Rapid7 JSON data correctly? on Getting Data In. 05-09-2023 07:34 AM
- Posted Re: Splunk ont extracting JSON from RAPID7 InsightVM Properly on Getting Data In. 05-09-2023 06:22 AM
- Posted Splunk not extracting JSON from RAPID7 InsightVM Properly on Getting Data In. 05-09-2023 06:19 AM
Topics I've Started
| Subject | Karma | Author | Latest Post |
|---|---|---|---|
| 0 | |||
| 0 |
05-09-2023
07:34 AM
Hello,
I am running into a bit of a challenge getting the data from the Rapid7 InsightVM TA to extract properly. The data in question is a nested JSON structure-- when I run a search on the data, the object appears to visualize properly, however the extracted fields are a mess and result in the data being unsearchable. I would expect that the nested fields get extracted into their own keys and values, however instead they get extracted into field* and in most cases those fields are comprised of a value from another key and the key itself.
Environment is distributed-- the TA is deployed on the HF ( which forwards to a UF forwarding tier, no TA there), Indexer Cluster, and Search Head cluster. The App is deployed on the SH cluster only. Props.conf (HF, Indexer Cluster, SHC):
[source::...ta-rapid7-insightvm*.log*] SHOULD_LINEMERGE = true sourcetype = tarapid7insightvm:log
[source::...ta_rapid7_insightvm*.log*] SHOULD_LINEMERGE = true sourcetype = tarapid7insightvm:log
[rapid7:insightvm:asset] FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_1 = host_name AS name FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_2 = os_description AS version FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_3 = os_description AS os FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_4 = os_architecture AS family FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_5 = os_system_name AS vendor_product SHOULD_LINEMERGE = 0 pulldown_type = 1 DATETIME_CONFIG = CURRENT
[rapid7:insightvm:asset:vulnerability_finding] FIELDALIAS-aob_gen_rapid7:insightvm:asset:vulnerability_finding_alias_1 = asset_hostname AS name FIELDALIAS-aob_gen_rapid7:insightvm:asset:vulnerability_finding_alias_2 = asset_ip AS ip FIELDALIAS-aob_gen_rapid7:insightvm:asset:vulnerability_finding_alias_3 = asset_ip AS dest SHOULD_LINEMERGE = 0 pulldown_type = 1 TRUNCATE = 50000 DATETIME_CONFIG = CURRENT
[rapid7:insightvm:vulnerability_definition] FIELDALIAS-aob_gen_rapid7:insightvm:vulnerability_definition_alias_1 = categories AS category FIELDALIAS-aob_gen_rapid7:insightvm:vulnerability_definition_alias_2 = cves AS cve SHOULD_LINEMERGE = 0 pulldown_type = 1 TRUNCATE = 50000
btool output from SH:
/opt/splunk/bin/splunk btool props list rapid7:insightvm:asset --debug /opt/splunk/etc/apps/search/local/props.conf [rapid7:insightvm:asset] /opt/splunk/etc/system/default/props.conf ADD_EXTRA_TIME_FIELDS = True /opt/splunk/etc/system/default/props.conf ANNOTATE_PUNCT = True /opt/splunk/etc/system/default/props.conf AUTO_KV_JSON = true /opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE = /opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE_DATE = True /opt/splunk/etc/system/default/props.conf CHARSET = UTF-8 /opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf DATETIME_CONFIG = CURRENT /opt/splunk/etc/system/default/props.conf DEPTH_LIMIT = 1000 /opt/splunk/etc/system/default/props.conf DETERMINE_TIMESTAMP_DATE_WITH_SYSTEM_TIME = false /opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_1 = host_name AS name /opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_2 = os_description AS version /opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_3 = os_description AS os /opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_4 = os_architecture AS family /opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_5 = os_system_name AS vendor_product /opt/splunk/etc/system/default/props.conf HEADER_MODE = /opt/splunk/etc/system/default/props.conf LB_CHUNK_BREAKER_TRUNCATE = 2000000 /opt/splunk/etc/system/default/props.conf LEARN_MODEL = true /opt/splunk/etc/system/default/props.conf LEARN_SOURCETYPE = true /opt/splunk/etc/system/default/props.conf LINE_BREAKER_LOOKBEHIND = 100 /opt/splunk/etc/system/default/props.conf MATCH_LIMIT = 100000 /opt/splunk/etc/system/default/props.conf MAX_DAYS_AGO = 2000 /opt/splunk/etc/system/default/props.conf MAX_DAYS_HENCE = 2 /opt/splunk/etc/system/default/props.conf MAX_DIFF_SECS_AGO = 3600 /opt/splunk/etc/system/default/props.conf MAX_DIFF_SECS_HENCE = 604800 /opt/splunk/etc/system/default/props.conf MAX_EVENTS = 256 /opt/splunk/etc/system/default/props.conf MAX_TIMESTAMP_LOOKAHEAD = 128 /opt/splunk/etc/system/default/props.conf MUST_BREAK_AFTER = /opt/splunk/etc/system/default/props.conf MUST_NOT_BREAK_AFTER = /opt/splunk/etc/system/default/props.conf MUST_NOT_BREAK_BEFORE = /opt/splunk/etc/apps/search/local/props.conf REPORT-rapid7 = REPORT-rapid7 /opt/splunk/etc/apps/search/local/props.conf REPORT-vm-cred-check = REPORT-vm-cred-check /opt/splunk/etc/system/default/props.conf SEGMENTATION = indexing /opt/splunk/etc/system/default/props.conf SEGMENTATION-all = full /opt/splunk/etc/system/default/props.conf SEGMENTATION-inner = inner /opt/splunk/etc/system/default/props.conf SEGMENTATION-outer = outer /opt/splunk/etc/system/default/props.conf SEGMENTATION-raw = none /opt/splunk/etc/system/default/props.conf SEGMENTATION-standard = standard /opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf SHOULD_LINEMERGE = 0 /opt/splunk/etc/system/default/props.conf TRANSFORMS = /opt/splunk/etc/system/default/props.conf TRUNCATE = 10000 /opt/splunk/etc/system/default/props.conf detect_trailing_nulls = false /opt/splunk/etc/system/default/props.conf maxDist = 100 /opt/splunk/etc/system/default/props.conf priority = /opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf pulldown_type = 1 /opt/splunk/etc/system/default/props.conf sourcetype = /opt/splunk/etc/system/default/props.conf termFrequencyWeightedDist = false /opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf [rapid7:insightvm:asset:vulnerability_finding] /opt/splunk/etc/system/default/props.conf ADD_EXTRA_TIME_FIELDS = True /opt/splunk/etc/system/default/props.conf ANNOTATE_PUNCT = True /opt/splunk/etc/system/default/props.conf AUTO_KV_JSON = true /opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE = /opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE_DATE = True /opt/splunk/etc/system/default/props.conf CHARSET = UTF-8 /opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf DATETIME_CONFIG = CURRENT /opt/splunk/etc/system/default/props.conf DEPTH_LIMIT = 1000 /opt/splunk/etc/system/default/props.conf DETERMINE_TIMESTAMP_DATE_WITH_SYSTEM_TIME = false /opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf FIELDALIAS-aob_gen_rapid7:insightvm:asset:vulnerability_finding_alias_1 = asset_hostname AS name /opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf FIELDALIAS-aob_gen_rapid7:insightvm:asset:vulnerability_finding_alias_2 = asset_ip AS ip /opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf FIELDALIAS-aob_gen_rapid7:insightvm:asset:vulnerability_finding_alias_3 = asset_ip AS dest /opt/splunk/etc/system/default/props.conf HEADER_MODE = /opt/splunk/etc/system/default/props.conf LB_CHUNK_BREAKER_TRUNCATE = 2000000 /opt/splunk/etc/system/default/props.conf LEARN_MODEL = true /opt/splunk/etc/system/default/props.conf LEARN_SOURCETYPE = true /opt/splunk/etc/system/default/props.conf LINE_BREAKER_LOOKBEHIND = 100 /opt/splunk/etc/system/default/props.conf MATCH_LIMIT = 100000 /opt/splunk/etc/system/default/props.conf MAX_DAYS_AGO = 2000 /opt/splunk/etc/system/default/props.conf MAX_DAYS_HENCE = 2 /opt/splunk/etc/system/default/props.conf MAX_DIFF_SECS_AGO = 3600 /opt/splunk/etc/system/default/props.conf MAX_DIFF_SECS_HENCE = 604800 /opt/splunk/etc/system/default/props.conf MAX_EVENTS = 256 /opt/splunk/etc/system/default/props.conf MAX_TIMESTAMP_LOOKAHEAD = 128 /opt/splunk/etc/system/default/props.conf MUST_BREAK_AFTER = /opt/splunk/etc/system/default/props.conf MUST_NOT_BREAK_AFTER = /opt/splunk/etc/system/default/props.conf MUST_NOT_BREAK_BEFORE = /opt/splunk/etc/system/default/props.conf SEGMENTATION = indexing /opt/splunk/etc/system/default/props.conf SEGMENTATION-all = full /opt/splunk/etc/system/default/props.conf SEGMENTATION-inner = inner /opt/splunk/etc/system/default/props.conf SEGMENTATION-outer = outer /opt/splunk/etc/system/default/props.conf SEGMENTATION-raw = none /opt/splunk/etc/system/default/props.conf SEGMENTATION-standard = standard /opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf SHOULD_LINEMERGE = 0 /opt/splunk/etc/system/default/props.conf TRANSFORMS = /opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf TRUNCATE = 50000 /opt/splunk/etc/system/default/props.conf detect_trailing_nulls = false /opt/splunk/etc/system/default/props.conf maxDist = 100 /opt/splunk/etc/system/default/props.conf priority = /opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf pulldown_type = 1 /opt/splunk/etc/system/default/props.conf sourcetype = /opt/splunk/etc/system/default/props.conf termFrequencyWeightedDist = false
This appears to only be happening with the rapid7:insightvm:asset sourcetype, however that is the type that I have primarily been focused on while trying to get things working. It may be happening in the other sourcetypes however I have not yet seen it in my limited time experimenting with that data.
Any help is much appreciated, thank you!
... View more
Labels
- Labels:
-
JSON
-
props.conf
05-09-2023
06:22 AM
Also, the raw event text:
... View more
05-09-2023
06:19 AM
Hello, I am running into a bit of a challenge getting the data from the Rapid7 InsightVM TA to extract properly. The data in question is a nested JSON structure-- when I run a search on the data, the object appears to visualize properly, however the extracted fields are a mess and result in the data being unsearchable. I would expect that the nested fields get extracted into their own keys and values, however instead they get extracted into field* and in most cases those fields are comprised of a value from another key and the key itself. Environment is distributed-- the TA is deployed on the HF ( which forwards to a UF forwarding tier, no TA there), Indexer Cluster, and Search Head cluster. The App is deployed on the SH cluster only. Props.conf (HF, Indexer Cluster, SHC): [source::...ta-rapid7-insightvm*.log*] SHOULD_LINEMERGE = true sourcetype = tarapid7insightvm:log [source::...ta_rapid7_insightvm*.log*] SHOULD_LINEMERGE = true sourcetype = tarapid7insightvm:log [rapid7:insightvm:asset] FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_1 = host_name AS name FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_2 = os_description AS version FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_3 = os_description AS os FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_4 = os_architecture AS family FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_5 = os_system_name AS vendor_product SHOULD_LINEMERGE = 0 pulldown_type = 1 DATETIME_CONFIG = CURRENT [rapid7:insightvm:asset:vulnerability_finding] FIELDALIAS-aob_gen_rapid7:insightvm:asset:vulnerability_finding_alias_1 = asset_hostname AS name FIELDALIAS-aob_gen_rapid7:insightvm:asset:vulnerability_finding_alias_2 = asset_ip AS ip FIELDALIAS-aob_gen_rapid7:insightvm:asset:vulnerability_finding_alias_3 = asset_ip AS dest SHOULD_LINEMERGE = 0 pulldown_type = 1 TRUNCATE = 50000 DATETIME_CONFIG = CURRENT [rapid7:insightvm:vulnerability_definition] FIELDALIAS-aob_gen_rapid7:insightvm:vulnerability_definition_alias_1 = categories AS category FIELDALIAS-aob_gen_rapid7:insightvm:vulnerability_definition_alias_2 = cves AS cve SHOULD_LINEMERGE = 0 pulldown_type = 1 TRUNCATE = 50000 btool output from SH: /opt/splunk/bin/splunk btool props list rapid7:insightvm:asset --debug /opt/splunk/etc/apps/search/local/props.conf [rapid7:insightvm:asset] /opt/splunk/etc/system/default/props.conf ADD_EXTRA_TIME_FIELDS = True /opt/splunk/etc/system/default/props.conf ANNOTATE_PUNCT = True /opt/splunk/etc/system/default/props.conf AUTO_KV_JSON = true /opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE = /opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE_DATE = True /opt/splunk/etc/system/default/props.conf CHARSET = UTF-8 /opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf DATETIME_CONFIG = CURRENT /opt/splunk/etc/system/default/props.conf DEPTH_LIMIT = 1000 /opt/splunk/etc/system/default/props.conf DETERMINE_TIMESTAMP_DATE_WITH_SYSTEM_TIME = false /opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_1 = host_name AS name /opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_2 = os_description AS version /opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_3 = os_description AS os /opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_4 = os_architecture AS family /opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf FIELDALIAS-aob_gen_rapid7:insightvm:asset_alias_5 = os_system_name AS vendor_product /opt/splunk/etc/system/default/props.conf HEADER_MODE = /opt/splunk/etc/system/default/props.conf LB_CHUNK_BREAKER_TRUNCATE = 2000000 /opt/splunk/etc/system/default/props.conf LEARN_MODEL = true /opt/splunk/etc/system/default/props.conf LEARN_SOURCETYPE = true /opt/splunk/etc/system/default/props.conf LINE_BREAKER_LOOKBEHIND = 100 /opt/splunk/etc/system/default/props.conf MATCH_LIMIT = 100000 /opt/splunk/etc/system/default/props.conf MAX_DAYS_AGO = 2000 /opt/splunk/etc/system/default/props.conf MAX_DAYS_HENCE = 2 /opt/splunk/etc/system/default/props.conf MAX_DIFF_SECS_AGO = 3600 /opt/splunk/etc/system/default/props.conf MAX_DIFF_SECS_HENCE = 604800 /opt/splunk/etc/system/default/props.conf MAX_EVENTS = 256 /opt/splunk/etc/system/default/props.conf MAX_TIMESTAMP_LOOKAHEAD = 128 /opt/splunk/etc/system/default/props.conf MUST_BREAK_AFTER = /opt/splunk/etc/system/default/props.conf MUST_NOT_BREAK_AFTER = /opt/splunk/etc/system/default/props.conf MUST_NOT_BREAK_BEFORE = /opt/splunk/etc/apps/search/local/props.conf REPORT-rapid7 = REPORT-rapid7 /opt/splunk/etc/apps/search/local/props.conf REPORT-vm-cred-check = REPORT-vm-cred-check /opt/splunk/etc/system/default/props.conf SEGMENTATION = indexing /opt/splunk/etc/system/default/props.conf SEGMENTATION-all = full /opt/splunk/etc/system/default/props.conf SEGMENTATION-inner = inner /opt/splunk/etc/system/default/props.conf SEGMENTATION-outer = outer /opt/splunk/etc/system/default/props.conf SEGMENTATION-raw = none /opt/splunk/etc/system/default/props.conf SEGMENTATION-standard = standard /opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf SHOULD_LINEMERGE = 0 /opt/splunk/etc/system/default/props.conf TRANSFORMS = /opt/splunk/etc/system/default/props.conf TRUNCATE = 10000 /opt/splunk/etc/system/default/props.conf detect_trailing_nulls = false /opt/splunk/etc/system/default/props.conf maxDist = 100 /opt/splunk/etc/system/default/props.conf priority = /opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf pulldown_type = 1 /opt/splunk/etc/system/default/props.conf sourcetype = /opt/splunk/etc/system/default/props.conf termFrequencyWeightedDist = false /opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf [rapid7:insightvm:asset:vulnerability_finding] /opt/splunk/etc/system/default/props.conf ADD_EXTRA_TIME_FIELDS = True /opt/splunk/etc/system/default/props.conf ANNOTATE_PUNCT = True /opt/splunk/etc/system/default/props.conf AUTO_KV_JSON = true /opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE = /opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE_DATE = True /opt/splunk/etc/system/default/props.conf CHARSET = UTF-8 /opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf DATETIME_CONFIG = CURRENT /opt/splunk/etc/system/default/props.conf DEPTH_LIMIT = 1000 /opt/splunk/etc/system/default/props.conf DETERMINE_TIMESTAMP_DATE_WITH_SYSTEM_TIME = false /opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf FIELDALIAS-aob_gen_rapid7:insightvm:asset:vulnerability_finding_alias_1 = asset_hostname AS name /opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf FIELDALIAS-aob_gen_rapid7:insightvm:asset:vulnerability_finding_alias_2 = asset_ip AS ip /opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf FIELDALIAS-aob_gen_rapid7:insightvm:asset:vulnerability_finding_alias_3 = asset_ip AS dest /opt/splunk/etc/system/default/props.conf HEADER_MODE = /opt/splunk/etc/system/default/props.conf LB_CHUNK_BREAKER_TRUNCATE = 2000000 /opt/splunk/etc/system/default/props.conf LEARN_MODEL = true /opt/splunk/etc/system/default/props.conf LEARN_SOURCETYPE = true /opt/splunk/etc/system/default/props.conf LINE_BREAKER_LOOKBEHIND = 100 /opt/splunk/etc/system/default/props.conf MATCH_LIMIT = 100000 /opt/splunk/etc/system/default/props.conf MAX_DAYS_AGO = 2000 /opt/splunk/etc/system/default/props.conf MAX_DAYS_HENCE = 2 /opt/splunk/etc/system/default/props.conf MAX_DIFF_SECS_AGO = 3600 /opt/splunk/etc/system/default/props.conf MAX_DIFF_SECS_HENCE = 604800 /opt/splunk/etc/system/default/props.conf MAX_EVENTS = 256 /opt/splunk/etc/system/default/props.conf MAX_TIMESTAMP_LOOKAHEAD = 128 /opt/splunk/etc/system/default/props.conf MUST_BREAK_AFTER = /opt/splunk/etc/system/default/props.conf MUST_NOT_BREAK_AFTER = /opt/splunk/etc/system/default/props.conf MUST_NOT_BREAK_BEFORE = /opt/splunk/etc/system/default/props.conf SEGMENTATION = indexing /opt/splunk/etc/system/default/props.conf SEGMENTATION-all = full /opt/splunk/etc/system/default/props.conf SEGMENTATION-inner = inner /opt/splunk/etc/system/default/props.conf SEGMENTATION-outer = outer /opt/splunk/etc/system/default/props.conf SEGMENTATION-raw = none /opt/splunk/etc/system/default/props.conf SEGMENTATION-standard = standard /opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf SHOULD_LINEMERGE = 0 /opt/splunk/etc/system/default/props.conf TRANSFORMS = /opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf TRUNCATE = 50000 /opt/splunk/etc/system/default/props.conf detect_trailing_nulls = false /opt/splunk/etc/system/default/props.conf maxDist = 100 /opt/splunk/etc/system/default/props.conf priority = /opt/splunk/etc/apps/TA-rapid7-insightvm/default/props.conf pulldown_type = 1 /opt/splunk/etc/system/default/props.conf sourcetype = /opt/splunk/etc/system/default/props.conf termFrequencyWeightedDist = false This appears to only be happening with the rapid7:insightvm:asset sourcetype, however that is the type that I have primarily been focused on while trying to get things working. It may be happening in the other sourcetypes however I have not yet seen it in my limited time experimenting with that data. Any help is much appreciated, thank you!
... View more
Labels
- Labels:
-
props.conf
Contact Me
| Online Status |
Offline
|
| Date Last Visited |
05-09-2023
06:20 PM
|
