topic Re: Fetching data from Splunk Cloud every 5 min over the API in Splunk Cloud Platform

Fetching data from Splunk Cloud every 5 min over the API

Jonas951 — Tue, 28 Jul 2020 15:26:41 GMT

According to

https://docs.splunk.com/Documentation/SplunkCloud/latest/Service/SplunkCloudservice

Data extracted as a result of search query, whether from the UI or REST API is limited to 5% of daily ingest for optimal performance.

And

Scheduled search is not supported from a hybrid search head.

Let's say I want to fetch over the API (not from a hybrid search head, instead of from a third-party system) 5 min worth of data and I schedule that search to run every minute.

I cannot see that that kind of set up would violate the agreement, but I want to make sure.

- 5 min worth of data every min will never equal 5% of daily ingest...

Anyone who has done a similar setup successfully?

Many Thanks

Jonas

Re: Fetching data from Splunk Cloud every 5 min over the API

richgalloway — Tue, 28 Jul 2020 18:57:07 GMT

By my reckoning, pulling 5 minutes of data every minute equals 600% of daily ingest.

Re: Fetching data from Splunk Cloud every 5 min over the API

Jonas951 — Tue, 28 Jul 2020 20:23:40 GMT

Hi, you are indeed right, 600% as a total.

That is what I cannot get my head around since it says "Data extracted as a result of search query"

My take on that is that every individual search query is not allowed to bring back a dataset larger than 5% of daily ingest.

Splitting hairs, I know 🙂

PS, otherwise it should be "Data extracted as a result of total search queries per day are not allowed to bring back a dataset larger than 5% of daily ingest."

Are you with me?

Small different in language, but a huge difference in terms of what I can do with my data in Splunk Cloud

Many thanks for answering my post 🙂

Best

Re: Fetching data from Splunk Cloud every 5 min over the API

richgalloway — Wed, 29 Jul 2020 12:35:47 GMT

I suppose it comes down to how many results your searches find. I search over 5 minutes every minute that ends up with a single results could be fine. You probably should contact Splunk for a definitive answer.