Are there any ways to whitelist specific fields from json format logs during data-onboarding?

[vincentwhn]

Due to privacy concerns, I would like to modify the _raw content during the data onboarding phase in order to implement a whitelist for JSON fields. Through testing, I found that I can achieve this using SEDCMD. However, SEDCMD applies to the entire sourcetype.

Is there any way to use SEDCMD while applying it only to specific events within same sourcetype?

[PickleRick]

Don't use regexes to manipulate structured data. It will cause pain sooner or later.

Use a tool which can understand the json structure. If you're on Splunk 10 you can use Edge Processor.

[vincentwhn]

I am still using Splunk 9.3.......

Can you share more about Edge Processor? Thanks.

[PickleRick]

https://www.splunk.com/en_us/blog/platform/introducing-edge-processor-for-splunk-enterprise-data-man...

[vincentwhn]

Thanks a lot. Will take a look at it.

[vjdev]

Hello,

Mask using regex , Filter using regex and EVAL can be achievable. 

This can help you lot. 
https://help.splunk.com/en/splunk-cloud-platform/get-started/splunk-validated-architectures/getting-...

• Mask, redact, remove, or otherwise change raw data before indexing
• Tag, add, lookup, or otherwise augment raw data before indexing
  
  Thank You!

[vjdev]

Hello,

Hope below example help for you!

1st Option Transform.

[session-anonymizer]
REGEX = (?m)^(.*)SessionId=\w+(\w{4}[&"].*)$
FORMAT = $1SessionId=########$2
DEST_KEY = _raw

2nd option is Ingest actions.

If both not helpful, explain more about the requirement . 

Thank you!

[vincentwhn]

Thanks for your assist.

"Could you share more about Ingest actions?