All Apps and Add-ons

aws:cloudtrail field extractiong not working on SHC env.

sylim_splunk
Splunk Employee
Splunk Employee

One item in particular that I'm seeing is the aws:cloudtrail sourcetype which is that the aws:cloudtrail sourcetype is not extracting fields as it is for other aws-related sourcetypes (e.g. aws:description).
This also appears to only occur on our SHC instead of the our standalone search head which does show the field extractions as expected.
Both search head environments are using the same TA content (being sourced by our Git repository) so the only difference is that the standalone search head is using the deployment server to get the TA while the search head cluster is getting the TA via deployer. I will attach screenshots of both environments as well as the correctly parsed aws:description data to demonstrate the issue.

1 Solution

sylim_splunk
Splunk Employee
Splunk Employee

Typically this can happen due to the precedence between the apps & add-ons. As the different apps/add-ons are installed on SHC SH and the standalone SH the first thing is to try to find the differences.

Use find . -name "props.conf" -o -name "transforms.conf" | xargs grep "aws:cloudtrail" this will tell us what files have configurations for the sourcetype in question.
This time, no other "aws:cloudtrail" related configuration in the standalone search head. However, in the SHC there is another props.conf in SAI, which has many other aws:XYZ sourcetypes, also it has a suspicious one, i.e, kv_mode = none - this is basically disabling automatic key-value field extractions.

[aws:cloudtrail] in splunk_app_infrastructure/default/props.conf
KV_MODE = none

The props.conf in SAI has higher precedence than what is defined in splunk_app_aws, "kv_mode=json" .

https://docs.splunk.com/Documentation/Splunk/8.0.3/Admin/Wheretofindtheconfigurationfiles#How_app_di...
" In the app/user context, precedence is determined instead by reverse-lexicographical order. "
To solve this problem, comment out the kv_mode in SAI.

View solution in original post

sylim_splunk
Splunk Employee
Splunk Employee

Typically this can happen due to the precedence between the apps & add-ons. As the different apps/add-ons are installed on SHC SH and the standalone SH the first thing is to try to find the differences.

Use find . -name "props.conf" -o -name "transforms.conf" | xargs grep "aws:cloudtrail" this will tell us what files have configurations for the sourcetype in question.
This time, no other "aws:cloudtrail" related configuration in the standalone search head. However, in the SHC there is another props.conf in SAI, which has many other aws:XYZ sourcetypes, also it has a suspicious one, i.e, kv_mode = none - this is basically disabling automatic key-value field extractions.

[aws:cloudtrail] in splunk_app_infrastructure/default/props.conf
KV_MODE = none

The props.conf in SAI has higher precedence than what is defined in splunk_app_aws, "kv_mode=json" .

https://docs.splunk.com/Documentation/Splunk/8.0.3/Admin/Wheretofindtheconfigurationfiles#How_app_di...
" In the app/user context, precedence is determined instead by reverse-lexicographical order. "
To solve this problem, comment out the kv_mode in SAI.

johnansett
Communicator

Thanks, this solved that exact issue.  Only thing I would say it's better to create a props.conf in local in the SAI app and then added the stanza

[aws:cloudtrail]
KV_MODE = json

That way when you upgrade the app it won't revert back.

Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...