Best Practices for Streaming Logs from Splunk Cloud to External Platforms

NavS — Sat, 16 Nov 2024 22:12:26 GMT

Hi Splunk Community,

I need advice on the best approach for streaming logs from Splunk Cloud Platform to an external platform. The logs are already being ingested into Splunk Cloud from various applications used by my client's organization. Now, the requirement is to forward or stream these logs to an external system for additional processing and analytics.

#Splunk cloud

Thank you

Nav

Re: Best Practices for Streaming Logs from Splunk Cloud to External Platforms

tscroggins — Sat, 16 Nov 2024 23:05:34 GMT

Hi @NavS,

Refer to https://docs.splunk.com/Documentation/SplunkCloud/latest/Service/SplunkCloudservice for supported data egress methods:

Data Egress Dynamic Data Self-Storage export of aged data per index from Splunk Cloud Platform to Amazon S3 or Google Cloud Storage No limit to the amount of data that can be exported from your indexes to your Amazon S3 or Google Cloud Storage account in the same region. Dynamic Data Self-Storage is designed to export 1 TB of data per hour.
Data Egress Search results via UI or REST API Recommend no more than 10% of ingested data For optimal performance, no single query, or all queries in aggregate over the day from the UI or REST API, should return full results of more than 10% of ingested daily volume. To route data to multiple locations, consider solutions like Ingest Actions, Ingest Processor, or the Edge Processor solution.
Data Egress Search results to Splunk User Behavior Analytics (UBA) No limit Data as a result of search queries to feed into Splunk User Behavior Analytics (UBA).

To stream events to both Splunk Cloud and another destination, an intermediate forwarding solution is required.

You should contact your client's Splunk account team for confirmation, but your Splunk Cloud native options are likely limited to the table above.

Re: Best Practices for Streaming Logs from Splunk Cloud to External Platforms

NavS — Sat, 16 Nov 2024 23:08:58 GMT

Thank you @tscroggins

Data Egress	Dynamic Data Self-Storage export of aged data per index from Splunk Cloud Platform to Amazon S3 or Google Cloud Storage	No limit to the amount of data that can be exported from your indexes to your Amazon S3 or Google Cloud Storage account in the same region.	Dynamic Data Self-Storage is designed to export 1 TB of data per hour.
Data Egress	Search results via UI or REST API	Recommend no more than 10% of ingested data	For optimal performance, no single query, or all queries in aggregate over the day from the UI or REST API, should return full results of more than 10% of ingested daily volume. To route data to multiple locations, consider solutions like Ingest Actions, Ingest Processor, or the Edge Processor solution.
Data Egress	Search results to Splunk User Behavior Analytics (UBA)	No limit	Data as a result of search queries to feed into Splunk User Behavior Analytics (UBA).

topic Best Practices for Streaming Logs from Splunk Cloud to External Platforms in Splunk Cloud Platform

Best Practices for Streaming Logs from Splunk Cloud to External Platforms

Re: Best Practices for Streaming Logs from Splunk Cloud to External Platforms

Re: Best Practices for Streaming Logs from Splunk Cloud to External Platforms