We have not been able to deploy Predictive and anomaly detection capabilities that can leverage MLTK as current ML algorithms in ITSI cannot account for business hour/seasonality. The question that I have is... is there a query example that you can point me to that would show how to exclude specific date ranges such as day of the week and hour for example exclude sat and Sunday. We are trying to include in glass tables predictive score, but the thing is with the current capabilities of ITSI it's not possible to exclude time ranges.  What we want to do is leverage MLTK for that.. to create custom prediction health score model... but with exclusion of time ranges.... then publish that model back into ITSI and show health score.. I guess the question is... how do I exclude time ranges from considerations... I guess the better question is... when ITSI predicts the health score does it take into consideration time data?  If so how can I use the same algorithm, but by excluding time ranges...?   If you can point me to appropriate resource, this would greatly help me out. ... View more

The problem we are running into is, the entity import can not keep up with the transient nature of AWS cloud, where we have auto scaling AWS EC2 instances and containers which get terminated and new ones gets created, but the new entities are not added automatically to services. How can I solve this issue in ITSI?  Any ideas? Here is a reference link for what auto scaling is.  Basically we are getting data from CloudWatch and passing it to Splunk, but the things, because thousands of containers are spinning up and down, Splunk somehow doesn't add dynamically such entities into services...  Is there anyway to overcome this? Or should I be using some other approach to monitor such types of microservices where containers spinning up and down, where one moment they can be in one node the next in another.... yet the entities do not get updated.... every time that happens...    How do other customers monitor micro-services in such case?    Any ideas?  https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-instance-monitoring.html ... View more

Hi we are using different types of tools to monitor our infrastructure and i am trying to get the data from multiple tools and feed into Splunk ITSI.  I am using Zenoss and AppDynamics to send logs and metrics to Splunk. Unfortunately there are some challenges I am running into. It's very difficult to prep the Zenoss and AppD logs/metrics to be consumed by Splunk in ITSI. We have yet not been able to define KPI for server availability (Device PING up/down) and Service/Process down, Port down, individual utilization etc….as without getting and normalizing all Zenoss and AppD data into Splunk, ITSI cannot be successful. In Splunk, we are missing KPI’s for PING/Availability, Processes, Port, Interface utilization, SWAP utilization in % Only meaningful KPI we are getting in Splunk is LoadAvg, CPU utilization in % and Memory utilization in%. Is there anyway to monitor this other data in Splunk ITSI? Note I know for example in Solar Winds and in New Relic there is a way to monitor availability..  Has anyone ran into this similar issues?  Or should I be using different tools for ping/availability, interface utilization, etc...  If that's the case is anyone using ITSI to link back to other tools?      ... View more

Question, we are trying to monitor disk space usage in Splunk ITSI. We are trying to use templates as much as possible in our environment.  What I am trying to understand is how doe we monitor drive space when it comes to each individual server having multiple file systems. Do we have to write multi KPI searches for each and every server/entity if we want to identify issues like a Disk space becoming full if a particular log file was writing debug messages? Do we have to write multi KIP searches if we want to identify  CPU /Memory was 100% since a runaway process or service was consuming very high CPU /Memory resources? For these types of root cause correlation what would be a good way of representing this visually?  It   looks like the deep dive does not provide this level of visibility and it seems to me that it would require manual correlation.  Is my understanding correct on this one? I guess what i am trying to say is... i want to see for example... which log file caused the disk space to go high and be able to see the log entry.. for that somehow from the same view...   Similar for the CPU... how do we see which CPU process ended up causing CPU to spike..  We see the metric based data from deep dive view, but it would be also nice to see the actual process... or drill into the metric somehow... and for it to show... why the spike has happened... Is this something that can be done from deep dive view? Or we need to create some individual manual type of correlations, and jump outside of the native ITSI deep dive functionality?           ... View more

How can we create templates for OS type like – Linux, AIX and Windows so that it can account for KPI’s around filesystems as each server can have different volumes or number of filesystems like one Windows server can have C,D and E drive and the other server can have C, E, T drive so how can one template handle these differences. Is it possible to visualize multiple KPIs  in dynamic template? We are running Zenoss and inside of Zenoss what we have is literally a server that shows... disk space usage across different types of... data volumes...   Splunk is only showing one single KPI for “disk_percentUsed” while this server has 11 total filesystems.  Has anyone run into these limitations of being able to present multiple KPIs in a single dynamic template? Or ITSI is the wrong place for us to do this and instead perhaps some other Splunk tool should be used? Or do we need to create multiple dynamic templates to tackle this? We just want to visualize file systems... as part of dynamic template, and leverage dynamic templates as much as possible.   Has anyone else ran into such limitations of not being able to display file systems from Zenoss in ITSI or ran into similar type of challenges and if so.. which solution did you come up with to over come such challenges?   Or maybe ITSI is not exactly the tool that should be used for this and perhaps some other dashboard visualization inside of Splunk should be used?   So i guess if you look at the KPI you will see it shows disk_percentUsed for a specific Linux server... for example.. but the thing is Linux server has multiple drives...  in this case how do we represent all of these multiple drives as part of single disk_percentUsed KPI?  Or that's not really possible and we would need to show for each file system different KPI?  Any ideas?  ... View more

Hi question, I am running into an issue with Splunk ITSI. We have a lot of different types of entities shown in service analyzer. The problem we are running into is that entity names we are coming in as not user friendly.. showing up as instance id, we would like to have it as a user friendly name,.. and the other issues is value shown as number 9 instead of number 9 we would like to show it as either good or bad (9 is good, and not 9 is bad) In order to modify  entity name in ITSI from non friendly name to friendly name do i need to pull these values directly from AWS tag or there is other ways of doing this? ... View more

Hi I am trying to evaluate Splunk in context of all the features that it has to offer.... I downloaded the demo, but the thing is there is no data for me to test out the features to see how unified dashboard looks like, is there a good reference link where you can point me to where I can get some sample data to infuse into Splunk to test it's ITSI capabilities? ... View more