All Apps and Add-ons

aws:cloudtrail field extractiong not working on SHC env.

sylim_splunk
Splunk Employee
Splunk Employee

One item in particular that I'm seeing is the aws:cloudtrail sourcetype which is that the aws:cloudtrail sourcetype is not extracting fields as it is for other aws-related sourcetypes (e.g. aws:description).
This also appears to only occur on our SHC instead of the our standalone search head which does show the field extractions as expected.
Both search head environments are using the same TA content (being sourced by our Git repository) so the only difference is that the standalone search head is using the deployment server to get the TA while the search head cluster is getting the TA via deployer. I will attach screenshots of both environments as well as the correctly parsed aws:description data to demonstrate the issue.

1 Solution

sylim_splunk
Splunk Employee
Splunk Employee

Typically this can happen due to the precedence between the apps & add-ons. As the different apps/add-ons are installed on SHC SH and the standalone SH the first thing is to try to find the differences.

Use find . -name "props.conf" -o -name "transforms.conf" | xargs grep "aws:cloudtrail" this will tell us what files have configurations for the sourcetype in question.
This time, no other "aws:cloudtrail" related configuration in the standalone search head. However, in the SHC there is another props.conf in SAI, which has many other aws:XYZ sourcetypes, also it has a suspicious one, i.e, kv_mode = none - this is basically disabling automatic key-value field extractions.

[aws:cloudtrail] in splunk_app_infrastructure/default/props.conf
KV_MODE = none

The props.conf in SAI has higher precedence than what is defined in splunk_app_aws, "kv_mode=json" .

https://docs.splunk.com/Documentation/Splunk/8.0.3/Admin/Wheretofindtheconfigurationfiles#How_app_di...
" In the app/user context, precedence is determined instead by reverse-lexicographical order. "
To solve this problem, comment out the kv_mode in SAI.

View solution in original post

sylim_splunk
Splunk Employee
Splunk Employee

Typically this can happen due to the precedence between the apps & add-ons. As the different apps/add-ons are installed on SHC SH and the standalone SH the first thing is to try to find the differences.

Use find . -name "props.conf" -o -name "transforms.conf" | xargs grep "aws:cloudtrail" this will tell us what files have configurations for the sourcetype in question.
This time, no other "aws:cloudtrail" related configuration in the standalone search head. However, in the SHC there is another props.conf in SAI, which has many other aws:XYZ sourcetypes, also it has a suspicious one, i.e, kv_mode = none - this is basically disabling automatic key-value field extractions.

[aws:cloudtrail] in splunk_app_infrastructure/default/props.conf
KV_MODE = none

The props.conf in SAI has higher precedence than what is defined in splunk_app_aws, "kv_mode=json" .

https://docs.splunk.com/Documentation/Splunk/8.0.3/Admin/Wheretofindtheconfigurationfiles#How_app_di...
" In the app/user context, precedence is determined instead by reverse-lexicographical order. "
To solve this problem, comment out the kv_mode in SAI.

johnansett
Communicator

Thanks, this solved that exact issue.  Only thing I would say it's better to create a props.conf in local in the SAI app and then added the stanza

[aws:cloudtrail]
KV_MODE = json

That way when you upgrade the app it won't revert back.

Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...