I want to add some monitoring of data model acceleration to inform us when it is falling behind (as this can have quite serious effects on our dashboard performance). However, so far I have not been able to find a way to do this programmatically (ie. other than by physically looking at the details of the data model in the manager list. I found the following REST API endpoint URI "datamodel/acceleration/{name}", but this is now deprecated (and seemed to not actually have a value for progress anyway). http://docs.splunk.com/Documentation/Splunk/6.3.1/RESTREF/RESTknowledge#datamodel.2Facceleration_.28DEPRECATED.29 The completely undocumented "| datamodelinfo" command exists but only returns results for 2 of our approx. 100 data models for some unknown reason. So three questions... 1) How can I programmatically monitor data model acceleration progress? 2) If the above REST API has been deprecated, what is the functionality replacement? 3) Has anyone successfully used the "| datamodelinfo" command for all datamodels ie. is the incomplete results issue affecting others or just me? Cheers, Glenn ... View more

What are people's experiences and expectations of the impact caused by a multisite indexer cluster rolling restart (on version 6.4.3)? By "impact", I mean that a complete set of events is not returned in searches. I can identify two classes of data: "historical" data that was sent to the cluster before it was put into maintenance mode, and "recent" data sent to the cluster after it was put into maintenance mode. For the purposes of example let's assume: - A two site cluster with search factor of 2, and 1 copy forced in each site - The rolling restart will never restart more than 1 indexer at a time I had expected that there would be no search impact to "historical" data, as there should always be a second searchable copy of the data within the cluster. This would require search heads to break site affinity and search across site, but according to the docs on cluster maintenance mode, this should happen as the cluster master will still attempt to reassign primaries. In testing, I have found that historical data availability is severely and unpredictably affected. Often a search head will not search across sites at all towards the beginning of a rolling restart. Later in the rolling restart, it starts to search across sites, but data is still not complete. I can see that potentially there could be an impact to "recent" data while the rolling restart is in progress. An event could have been written to a certain indexer, not replicated since the cluster was in maintenance mode, and then the indexer that holds it goes down for its restart and the data becomes unavailable until that indexer returns. Does this mean that it is the case that it is not possible to restart an indexer cluster without severely impacting data searchability, and so it becomes necessary to prevent user access throughout, as well as disabling alerting and anything else that relies on search? The docs seem to say that indexer clustering provides high availability where the data is always available for searching, but this appears to be a false claim. If this impact is real and I haven't stuffed it up somehow, how can it be mitigated? ... View more

Is there any functionality (built-in to Splunk, or that someone has created custom) to do lookups to an external RESTful service from within a Splunk search, which functions similarly to a static file lookup or DB lookup with the DB connect app? This mean the rest query run from Splunk would return a table of some kind, which could be used for doing a lookup. Our organisation maintains a Service Oriented Architecture, where we prefer to expose data via API rather than query DBs directly. Using a service layer allows us to manage performance indexing and caching, and avoids issues caused by db structural change etc. A search command which can query a REST endpoint and lookup on the table which is returned would be a very useful piece of functionality. The REST modular input is not quite right, as it indexes data from REST rather than doing lookups on it. ... View more

Hi, I have read in a number of other Splunk Answers that the tsidxstats files generated by the tscollect command need to be manually managed. What I couldn't find was how to manually manage them. We have a large volume of tsidxstats files created on our search head which I need to clear. In this case I would be happy enough to just delete them every now and again. In our case, they are being generated by searches in the NetApp OnTap app. So, is it OK to just delete the .tsidx files within the namespace, while Splunk is still up and running? Cheers, Glenn ... View more

Say, I have three events. 2014/04/16 23:54:00,000 id=aaaaa doing thing A 2014/04/16 23:54:00,021 id=aaaaa doing thing B 2014/04/16 23:54:00,046 id=aaaaa doing thing C Is there a way to join these all up into a process flow, and calculate the latencies between each stage? I know I could use transaction to join them into a single event, where duration is calculated automatically as total time for the flow, but is not sufficient for situations where you need durations/latencies between each event. An example of where this would be useful, is for flow analysis of a software system, and identifying which parts of the flow are causing delays. The solution would ideally be able to run over large datasets, flow can be monitored statistically for trend/historical analysis. Ie. not just for tracing the flow of a single ID. Bonus question! What if the same ID is not present in all events (although can be correlated via some events): 2014/04/16 23:54:00,000 id=aaaaa doing thing A 2014/04/16 23:54:00,021 id=aaaaa Xid=bbbbb doing thing B 2014/04/16 23:54:00,046 Xid=bbbbb doing thing C How can you do the common field correlation over large data sets? ... View more