topic Re: Limiting ingested fields in Azure Event Hubs in All Apps and Add-ons

Limiting ingested fields in Azure Event Hubs

zippo706 — Mon, 10 Aug 2020 19:38:38 GMT

I"d like to send audit data through an event hub. However, i want my heavy fwd'r to not send all fields to splunk as 75% of is will be useless and take up all my ingesting quota.

Is there an easy way to do this? The data coming in is Azure SQL where i don't beleive i can change data going into the hub.

Re: Limiting ingested fields in Azure Event Hubs

richgalloway — Mon, 10 Aug 2020 20:38:57 GMT

If you want to discard entire events, see https://docs.splunk.com/Documentation/Splunk/8.0.5/Forwarding/Routeandfilterdatad#Filter_event_data_and_send_to_queues

If you want to discard parts of events, use SEDCMD in props.conf.

[mysourcetype] SEDCMD-winevent = s/This event is generated.*//

Re: Limiting ingested fields in Azure Event Hubs

zippo706 — Mon, 10 Aug 2020 20:47:29 GMT

Thanks for the info. can i discard or manipulate fields in an event. I'm going to speak logstash here and mutate to delete "reallybigfieldIdon'tcareabout"

Re: Limiting ingested fields in Azure Event Hubs

richgalloway — Tue, 11 Aug 2020 00:29:08 GMT

Yes, you can do that with SEDCMD. It will be on the raw event, however, since fields haven't been extracted when SEDCMD runs.

Re: Limiting ingested fields in Azure Event Hubs

zippo706 — Wed, 12 Aug 2020 16:02:27 GMT

Thank you, much appreciated.