Hi, We are running ITSI on a dedicated Search head running 16 cores cpu , 23 gb RAM and about 150 gb disk. It's searching an Index Cluster serving a SH Cluster and two stand alone search heads. We have about 400 services with 3-5 kpis each. The majority of these services have been created by import through search and mass produced. They have all worked fine. However, when we try to add more services, from a specific search, we are getting NaN for each KPI and ServiceHealthScore. We have removed backfilling on the service KPIs and set static thresholds. The skipped search rate is very low,( 1,34%, 66 searches) But the new services (138) all are missing values. When the service is opened up in deep dive and summary index turned off, we get data for the entirety of kpis. The Kpis use stats sum and count. We turned up the [kvstore] max_size_per_batch_save_mb = 100 In limits.conf Our ITSI_backfill is at 49 mb. we are not using Backfill on these services/kpis so that should not be the problem. So I am wondering two things, How does ITSI create the services and Kpis from search? How do I reload a service and its kpis? BR / Daniel ... View more

Hi, I am building a clustered Splunk environment for one of our customers. The Environment is built as follows: - cluster master the cluster master also acts as license master, deployment server, SHC deployer, and DMC - indexer1 acts as search peer, license slave - indexer2 acts as search peer, license slave - search head 1 Search head captain - search head 2 search head member - search head 3 search head member Now the cluster master acts as both a deployment server and a cluster master. My question is, what is the best way to copy configuration from deployment server, ie sourcetypes and indexes to the search peers, ie in master_apps. I am thinking about using symbolic link to copy conf from deployment apps to the master apps directory OR using the deployment server to deploy the changes straight to the indexers. But I am looking for a "best practice" and I'm not sure if the symlinking could cause problems. Any suggestions on how to go about this problem? Thanks! / Daniel ... View more

I am running Splunk DB Connect 2 on the only search head in a clustered Environment. I have a Connection to an MS SQL Server Using MS generic driver with Windows authentication. I also have an Identity up and running. I don't have any problems using either db inputs getting data into Splunk or db outputs. My issue is that when I want to add a db lookup from the Operations tab, I get the error with db lookups. I have tried to search the internal index for any dbx errors, but I have not found anything out of order. I'm running Splunk DB Connect 2.1.3 on Splunk 6.2.2 running on a Linux server. Is there any way I can set up the lookup through inputs.conf, or is there a simpler way to solve this problem? ... View more

Hi, My issue is I have two different searches, first: index=test user=test document=* second: index=test2 user=test src=home action=view What I would like to do is gather the timestamps from the first search and add them as a condition for the second search. I would also like to shorten the timestamp to the current hour so I can get the view actions that happen before and after there was a document value. Is there any way of doing this in Splunk? ... View more