Deployment Architecture

Forward Truncated Events

flucman
Explorer

I'm looking to forward only the first portion of a log file event to the indexer to be indexed. The remaining data I cannot send due to possible PCI reasons. I have installed the full splunk instance to use the heavy forwarder and have tried updating props.conf with an EXTRACT statement to pull out what I want forwarded. Portion below:

[myindex]
EXTRACT-myindexEXT = (?<A>\d+\:\d+\.\d+\:\d+\.\d+) (?<B>\d+) (?<C>\w+)

But the full logged message is getting sent to the indexer. Is there a way to do this?

Tags (1)
0 Karma

kristian_kolb
Ultra Champion

You have misunderstood what the EXTRACT will do. That is for creating fields at search time. It has nothing to do with masking data, or limit the amount of each message being indexed.

According to this document ,if you have a heavy forwarder, you can use the SEDCMD function to mask out data, since the parsing phase takes place on a heavy forwarder.

from the docs on props.conf

SEDCMD-<name> = <sed script>
* Only used at index time.
* Commonly used to anonymize incoming data at index time, such as credit card or social
  security numbers. For more information, search the online documentation for "anonymize
  data."

TRUNCATE = n (also in props.conf) may be an option.

For more info on anonymization, please see;

http://docs.splunk.com/Documentation/Splunk/5.0.2/Data/Anonymizedatausingconfigurationfiles

Hope this helps,

K

Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In January, the Splunk Threat Research Team had one release of new security content via the Splunk ES Content ...

Expert Tips from Splunk Professional Services, Ensuring Compliance, and More New ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Observability Release Update: AI Assistant, AppD + Observability Cloud Integrations & ...

This month’s releases across the Splunk Observability portfolio deliver earlier detection and faster ...