On the Splunk server, take a look in the directory for the AWS app under $SPLUNK_HOME/etc/apps . Under default/data/ui/views you should be able to see the original dashboard XML. If that is the case, go to the app's local/data/ui/views and copy the XML for that view from the local directory to some temporary location. Then, in your browser go to http://yoursplunkserver:8000/en-US/debug/refresh to reload. Getting rid of the version in the local directory should set what displays back to the default dashboard. ... View more

OK so it sounds like part 1 on the discrepancy is due to the difference in the _internal index retention which affects the license master logs that will show and the retention on the index shown here. The disk utilization on the indexer represents a much longer time period than what's reflected in _internal. To address the second part of the question on reducing disk usage without changing the retention period, here's a couple options: Check your cluster master to see if there are excess bucket copies that you can remove to free up some space. If the data is infrequently accessed past a certain age or if slower searches beyond a certain age aren't a concern, look into tsidx reduction. Make sure your replication and search factors aren't too high as this will require additional space for the extra copies. Use something like |dbinspect index=index1 | chart count by guId to make sure you didn't happen to catch much higher than average space usage for that index due to a bucket imbalance. If the bucket counts aren't reasonably close to even, first make sure you don't have an imbalance on incoming data then consider doing a cluster rebalance for just that index (or include others if you notice the problem on other indexes too). ... View more

I'm not sure how search quotas would apply to scheduled searches owned by "nobody", if there's a lot then it's possible that they could all be treated as one user from the scheduler's perspective, but I haven't actually seen that problem happen. I prefer to change them to a new user so there is clear ownership, especially if an app is being used by multiple teams and I need to have some idea of who owns which content within it. Changing ownership to a new owner also forces the teams who should be responsible for the content to evaluate whether they still need it and avoid duplicating it rather than just having it sit in the environment forever after the original owner leaves because no one takes responsibility for it. I have heard mentions of performance problems caused by large local.meta files due to entries for old/unused content, so forcing cleanup and clear content ownership is probably a good thing. You should also note that if you're looking specifically at cases where knowledge objects were orphaned due to the user being removed, there is now an option to reassign ownership through Splunk Web which seems a lot less painful than doing it in local.meta, but I haven't tried it myself yet. If you're changing local.meta directly, the debug/refresh REST endpoint should work, but it wouldn't be a bad idea to check after hitting it that the ownership did actually change. ... View more

I'm going to assume that you are going to be building new indexers rather than clustering your existing ones. If that is the case, you would need to start by building your cluster master and new indexers. For your search heads you will need to setup the new search head cluster with the deployer and get it searching both the new indexers and the old indexers, assuming your current 2 aren't going to become part of your cluster. It's especially important to move the search heads first if you are also upgrading Splunk versions on this move since the search heads have to be either the same or a higher version than the indexers. Then, you'll want to copy over the default app directories from your old search head to the deployer and push them out, but be very careful not to include out of the box apps like "search" in that push. The local app directories are best setup by copying them to each cluster member so that users will be able to delete their own content. If you push them from the deployer, the cluster members will see them merged into default from the deployer, so users wouldn't be able to delete their own content and some don't like that. For the user level directories on the search head, I just copied them to each SHC member because we're using LDAP, but if you have locally defined users you'd likely want to treat that migration similarly to the app migration in which you define default content on the deployer and local on each cluster member. Next, make sure your new indexers are setup with the same input and parsing configurations as your old ones. Then, you have 2 options, and the best one for you will depend on your comfort level and the tolerance for a complete outage of indexers in your environment. Option 1: Have your forwarders start sending new data to your new indexers. After you flip your forwarders, take a good look at your old indexers to make sure they are not still taking in data. Once you're sure they aren't, restart all of them to roll any hot buckets to warm. Then you should be able to use the guide at https://docs.splunk.com/Documentation/Splunk/7.0.0/Installation/MigrateaSplunkinstance to actually move the data. Note that the legacy buckets won't replicate, if that is a huge concern you might want to consult with professional services since they do have some methods of making those buckets replicate anyway. As the data is activated on the new servers, you should take down the old servers that the data came from to prevent duplicates from being returned in the search results. Option 2: Take your old indexers offline, follow the migration instructions linked above, bring up your new indexers, then have your forwarders redirect to the new ones. This is simpler to implement but results in more downtime. You could theoretically reduce the downtime by doing an rsync in advance, then a second rsync during the change window just for the updates. After that, since you are going from 2 indexers to 3, you would want to rebalance your cluster so that the buckets are being searched evenly across the indexers. Otherwise the 2 that got the migrated data would be getting most of the load. Now, if you are looking to migrate your existing indexers into a cluster rather than building new ones, you'll want to check https://docs.splunk.com/Documentation/Splunk/6.6.3/Indexer/Migratenon-clusteredindexerstoaclusteredenvironment for guidelines on how to do that. You'll still need to address your search heads first if you are upgrading to a newer version. ... View more

What OS is your Splunk instance running on? If you installed it by untarring the app in etc/apps rather than a UI upload, I'd double check the file permissions and ownership within that app. ... View more