Greetings fellow delvers of the deep data.... We recently made some changes to indexes.conf because we were not sure the config was doing what we wanted it to do. The result of that poorly considered decision moved 2.6 TB of data out of hot/warm into cold, and another 400 GB into frozen that probably was prematurely aged out. Let me preface this by saying for our size, we have a lot of SSD on our new indexers. Current disk footprint /dev/mapper/splunkdatacold 8.8T 4.4T 4.5T 49% /splunkdatacold /dev/mapper/splunkdatafrozen 15T 2.5T 13T 17% /splunkdatafrozen /dev/mapper/vg02-lv01 650G 414G 237G 64% /splunkdatamodels /dev/mapper/vg02-lv00 11T 933G 9.1T 10% /splunkdatahot The result of that grand data shuffle is that I have a 11 TB SSD volume for hot with 9.1 TB unused space. I also have 5 TB unallocated in the volume group that I could add. I'm told that amount of SSD is somewhat uncommon, but we were going for a reasonably future proof configuration. I am about to cluster another indexer with the same space footprint into this, so I want to make sure we are utilizing the SSD effectively. For the purpose of this question, I will stick to one of our big indexes. We had these two settings and turning them off is what caused the multiple terabyte shift out of hot. maxHotIdleSecs = 86400 maxWarmDBCount = 6800 One of our bigger and typical indexes definitions. [networks] homePath = volume:hot/networks/db coldPath = volume:cold/networks/colddb thawedPath = $SPLUNK_DB/networks/thaweddb maxTotalDataSizeMB = 5083636 homePath.maxDataSizeMB = 3389260 coldPath.maxDataSizeMB = 1694376 #explicit path to frozen directory coldToFrozenDir = /splunkdatafrozen/networks We had thought that getting a 2/3 to 1/3 ratio between hot+warm/cold would be done by specifying the max Total size, and then the homepath.maxDataSizeMB and coldPath.maxDataSizeMB so that those two values added up to the maxTotal value. So the question is, how do you go about engineering a 2/3 to 1/3 split between hot/cold? We want to utilize the SSD's as much as possible, which is why we have been playing with the maxwarmDBcount parm and maxHotIdleSecs Today we turned both of those back on and change the networks sizing a bit. Just to see what actually happens, because the previous configuration never pushed hot past 30% Current parms in place: maxHotIdleSecs = 86400 maxWarmDBCount = 6800 [networks] homePath = volume:hot/networks/db coldPath = volume:cold/networks/colddb thawedPath = $SPLUNK_DB/networks/thaweddb maxTotalDataSizeMB = 5083636 #explicit path to frozen directory coldToFrozenDir = /splunkdatafrozen/networks maxTotalDataSizeMB = 5083636 homePath.maxDataSizeMB = 5083636 ... View more

Good morning those more knowledgeable than myself 🙂 The index usage default panel which shows such useful information as earliest event, is not quite giving me what I need. Trying to manage Hot/warm, cold and frozen in such a way as 60% of the data is on hot/warm, 40% of the data is on cold and anything older than 115 days (we promise 90 searchable) goes to frozen. The frozen data is included in the earliest event calculation, and I'd like to either see my footprint in size for only hot/cold or for all three so I can calculate the hot/cold ratio without the blur introduced by including frozen. Ideally I'd like to have the ability to control hot/cold retention entirely by the date range of the data, not the size of it but that seems to be impossible to do directly hence calculating it. Now having turned frozen on, the data we were using to size the indexes is being made fuzzy by including the frozen data in the earliest event count. So what particular Splunk incantation is needed to parse the index footprint data out like that? -J ... View more

Hey gang. This is somewhat urgent. Moved the indexer to a physical box from a virtual. In our situation we use an alias 'splunkprodindex' to point to the indexer. We did this knowing we would have to move the indexer off virtual. This was to theoretically make it easier to do this. Shut everything down. rsynced the data. Changed the alias to point to the new indexer in DNS. Updated serverclass to whitelist the indexer in the deployment server. Updated forwarder outputs with a comment field as it contained the DNS alias and we wanted to make sure the deployment server had a conversation with the forwarders immediately I have 55 missing forwarders in the deployment monitor screen. When we check a number of these, we see in the log they are still trying to talk to the old server name. Active Directory DNS was flushed. Name resolution on the box (Windows AD servers) was confirmed as to be pointing to the right IP address. We removed the old indexer from the indexer role, and started the new indexer and deployment server and we could see that serverclass and forwarder outputs had a recent timestamp that was after the changes made in the deployment server. We've had several things report in with no trouble. We have stopped and restated forwarders. We have in one case uninstalled and reinstalled the forwarder on one Active Directory server. When we installed it, we only supplied the deployment server name, no indexer name specified. One AD server in a different domain has since checked in to the new server. We made no changes to it. What do we do next? We're going to lose AD data as the logs roll over shortly if we have not already. ... View more

Following the steps in this document: http://docs.splunk.com/Documentation/Splunk/6.2.5/Installation/MigrateaSplunkinstance This is Linux to Linux - Prior to doing this on the new hardware, we are attempting to do this in the lab environment in which we built and configured an identical machine to use as a new indexer. The above document does not appear to cover everything needed to do this. We shut down the lab/test system completely and rsynced the data. We have some indexes pointing to /splunkdatahot /splundatacold... frozen... etc. And a couple still sitting in main in the /opt/splunk/var/lib/splunk file system. So /opt/splunk, /opt/splunk/var/lib/splunk, /splunkdatahot /spunkdatacold, /splundatafrozen, and splundatamodels were copied from the old VM to the new VM to test the process. While this lab environment is V to V, the process should be identical for V to P if I understand things correctly. When things first came up, the new indexer needed the --accept-license flag as expected. Had to change some of the configuration files (add the new name to the indexer whitelist in the server class and chase down where the machine name for http://i[machine]:8000/en-US/app/launcher/home#en-US/app/launcher/home was stored. We are able to search the copied data. We have our custom forwarder outputs file where the indexer name is stored (in prod this will use the alias and we will repoint the alias and not have to do this) and we changed the name of the name of the tcpout:primary_indexers stanza to the new machine name. Deployment server pushed that out to the new indexer (the only machines running at this point are the indexer and the deployment server). We are getting an error message on the indexer console: Forwarding to indexer group primary_indexers blocked for 100 seconds. Since this machine is the only machine in the that group so we don't know why it's saying that. forwarder outputs looks like this: [tcpout] defaultGroup = primary_indexers [tcpout:primary_indexers] server = [servername]006.[fqdn]:9997 We see the forwarder outputs being pushed out to the universal forwarders with that name in them but we have no new data coming into the index. The forwarder logs are also showing the forwarder still attempting to talk to the old indexer name even though forwarder_outputs contains the new indexer name. In short, we cannot do this in prod unless we have a much clearer 'how to' guide. If anyone can help us fill in the blanks we would greatly appreciate it. ... View more

Right now I am tasked with creating a report for a department showing who is using elevated privileges in Linux and for what commands. That search looks somewhat like this (I've anonymized the server names because of policy, but it will give you the idea. index=* sudo host=[LOGHOST] ( ("[serverytype1]dl002" ) OR ("[serverytype1]dl004" ) OR ("[serverytype1]pl007" ) OR ("[serverytype1]pl008" ) OR ("[serverytype1]pl009" ) OR ("[serverytype1]pl010" ) OR ("[serverytype1]pl011" ) OR ("[serverytype1]pl012" ) OR ("[serverytype1]pl013" ) OR ("[serverytype1]pl014" ) OR ("[serverytype1]tl002" ) OR ("[serverytype1]tl005" ) OR ("[serverytype1]tl006" ) OR ("[serverytype1]tl009" ) OR ("[serverytype1]tl010" ) OR ("[serverytype1]tl011" ) OR ("[serverytype1]tl012" ) OR ("[serverytype1]tl013" ) OR ("[serverytype3]pl001" ) OR ("[serverytype3]sl001" ) OR ("[serverytype4]pl001" ) OR ("[serverytype4]pl002" ) OR ("[serverytype4]pl003" ) OR ("[serverytype4]pl004" ) OR ("[serverytype4]pl005" ) OR ("[serverytype4]pl006" ) OR ("[serverytype4]pl007" ) OR ("[serverytype4]pl008" ) OR ("[serverytype4]tl001" ) OR ("[serverytype2]pl003" ) OR ("[serverytype2]pl004" ) OR ("[serverytype2]pl005" ) OR ("[serverytype2]pl006" ) OR ("[serverytype2]pl007" ) OR ("[serverytype2]pl008" ) OR ("[serverytype2]pl009" ) OR ("[serverytype2]tl001" ) OR ("[serverytype2]tl002" ) OR ("[serverytype2]tl005" ) OR ("[serverytype1]pl016" ) OR ("[serverytype1]pl015" ) ) ( NOT scomact ) (NOT USER=root) So I have a lookup table with all of the server names that are listed above. The problem with doing either method, is that either the list in the search needs to be updated every time (Today alone 12 server decommissions for this group and 6 server build request last week) or the lookup table needs to be updated. The syslog data is coming in, in most cases via remote syslog to a loghost which then sends it in via a universal forwarder. This makes all of the data come in as hostname '[LOGHOST]' so I cannot do a host=name. Sourcetype=sudo is too limiting as I have some sudo log stuff coming in on auth-priv sourctype and a couple others. Could we normalize that stuff? Yeah probably. Just haven't. The server names are split between a naming convention that would allow pattern matching, and an older naming convention where servers were named after Minnesota lakes and thus not able to be pattern matched for all of the hosts. If this was a scripting language I'd just loop through the list of servers and grab the data that met the date ranges and stuff it into a file, salt to taste and serve. How do I create a less cumbersome search than the above example? ... View more