Splunk IT Service Intelligence

ITSI Service Template too big to sync!

ewan000
Path Finder

I created a KPI base search with 197 metrics

I made a service template with each of these metrics as a KPI

When I try to sync the service template, or create a new service using the template I get the error:

"Error while updating services linked to service_template="241ed73e-000c-4635-a9b0-3ea04577e5dd". Exception="'Object you are trying to save is too large (55209132 bytes). KV store only supports documents within 16MB sizes.'""

Checking the documentation on base searches I find:

"Most of the KPI base searches delivered with ITSI are configured to run every minute. Based on testing on a system with 32 cores and 16 GB of memory, a single KPI base search can support up to 5,000 KPIs with 15 entities matched by service entity rules reasonably well."

I'm well below the 5000 KPIs with 15 entities.

If I retrieve the Service Template via the API it is rather large at 6Mb mainly due to the long base search being repeated in each KPI. But still nothing like the 55Mb reported in the error

What limit am i running into here and how can i get around it?

Labels (1)
0 Karma

ewan000
Path Finder

I moved the base search's search to a saved search and made the KPI base search run the saved search, massively reducing the length of the search string in the base KPI.

After that the Service Template had no trouble syncing, so it does seem to be purely related to the length of the search string.

0 Karma

ewan000
Path Finder

deleting a few KPIs drastically starts to reduce the size. I got rid of 10 or so and it went down to 44Mb

0 Karma
Get Updates on the Splunk Community!

Updated Team Landing Page in Splunk Observability

We’re making some changes to the team landing page in Splunk Observability, based on your feedback. The ...

New! Splunk Observability Search Enhancements for Splunk APM Services/Traces and ...

Regardless of where you are in Splunk Observability, you can search for relevant APM targets including service ...

Webinar Recap | Revolutionizing IT Operations: The Transformative Power of AI and ML ...

The Transformative Power of AI and ML in Enhancing Observability   In the realm of IT operations, the ...