I have several service templates which are in the status of "Sync Scheduled" I've left them some time with no change. What trouble shooting steps can I take? ie. How can I find out when they are scheduled to sync? How can I force a sync? How can I see any sync errors? ( think I know this, but for completeness...) What are the common causes of service templates not syncing promptly? What is the expected time to wait before they sync? ... View more

I created a KPI base search with 197 metrics I made a service template with each of these metrics as a KPI When I try to sync the service template, or create a new service using the template I get the error: "Error while updating services linked to service_template="241ed73e-000c-4635-a9b0-3ea04577e5dd". Exception="'Object you are trying to save is too large (55209132 bytes). KV store only supports documents within 16MB sizes.'"" Checking the documentation on base searches I find: "Most of the KPI base searches delivered with ITSI are configured to run every minute. Based on testing on a system with 32 cores and 16 GB of memory, a single KPI base search can support up to 5,000 KPIs with 15 entities matched by service entity rules reasonably well." I'm well below the 5000 KPIs with 15 entities. If I retrieve the Service Template via the API it is rather large at 6Mb mainly due to the long base search being repeated in each KPI. But still nothing like the 55Mb reported in the error What limit am i running into here and how can i get around it? ... View more

Hi, We are ingesting syslog and statsd data directly on a Heavy Forwarder via upd inputs This seems to work fine for small volumes of data, but heavy load causes all the ingestion on these inputs to drop off. even ones not receiving large amounts of data. What limits are there in splunk or linux which might cause this behaviour and does anyone know of a fix? PS. I know the recommendation is to use a separate syslog box and then read the files. But the reason for this are always given as : "you lose data when you restart splunk" which is not a problem for us. Is there a secret reason?! ... View more

I am attempting to make a trellis visualization off the sample data : * clientip=* | iplocation clientip | lookup prod product_id output product_name | top product_name limit=5 by Country This works fine on a historical search. However, if I switch to real-time search the visualization does not display as expected. Instead of being split by country, the only available "split by" option in the trellis formatting UI is "Aggregations (4)" and 4 bar charts are displayed: product_name, country, count and percent with no y-axis. The doc page for the trellis visualization seems to suggest that there is something special about the by clause. it returns a list of possible values which the visualization needs to make its charts I guess. And you can see why that might not work with real-time streamed matches. But it is not explicitly called out as being incompatible. Am I doing something wrong, or is it impossible to make a trellis chart with real-time searches? ... View more

I have 264 services I can't delete. If you delete them through the UI or API they recreate themselves after a matter of seconds. If you edit them the changes are undone. ... View more

The Service Analyser seems to be unable to filter with the square brackets characters. In Configure Services I am able to escape the characters with \[ or \] but this doesn't work for the Service Analyser Can anyone point me at a list of allowed characters and escape sequences for service names? or is this a known bug? ... View more

When we are editing base searches we get the warning: “Entity Alias Filtering will be removed in a future release” The help link doesn’t take you anywhere with any information on this and we are relying heavily on entity filtering. What’s the plans in the space and how should we prepare for them? ITSI v4.1.1 ... View more

I created (via csv import) and then deleted (via Configure Services UI) a bunch of services which were dependent services of other services. The expected behaviour for me was that this would remove the nodes from the Service Analyzer diagram. However, I am left with a blank node to joining the services which depended on the deleted service. Is this a bug? How can i get rid of the blank node? we are on ITSI 4.1.1 Splunk Enterprise 7.1.3 got some erros in the log, not sure if they are related [itsi.consumer] [init] [exception] [48538] Saved search "Indicator - ecd48ed9-7874-4650-a885-36ff77817683 - ITSI Search" was not found, ignoring delete ... View more

Is there an official c# client for the ITSI rest interface? https://docs.splunk.com/Documentation/ITSI/4.1.1/RESTAPI/ITSIRESTAPIreference ... View more

Is it possible to have multiple service analyzers? I was under the impression that "Create Service Analyzer" would create a new blank dependency diagram, to which i would add services etc. But now I have actually had an occasion to use it, I see that it merely saves search options on the default tree. The problem I am facing is that we have a product with separate infrastructure per tennant. Each tenant will have a separate (almost)identical deployment of ServiceX which depends on ServiceY etc I am prevented from making two services called 'ServiceX' as duplicate service titles are not allowed I could have a single ServiceX and use entities to split the data but this would turn red if the other tenants ServiceY failed. Which would not be an accurate representation of the reality I could prefix each service with the tenant name "TenantA.ServiceX" but this would lead to everything in the tree having very long Environment1.MarketAlpha.TennantA.SubProductQ.ServiceX names I could presumably, have a seperate ITSI installation per tenant. But this will only reduce the problem and maybe introduce new problems where we have shared components I was hoping to make a service analyser template and have a seperate dependency map for each tenant based on it. Does anyone have any good solutions? ... View more