Here is our documentation that discusses migrating: http://docs.splunk.com/Documentation/Splunk/6.5.2/Installation/MigrateaSplunkinstance As you will see in this section, you will need to stop Splunk: 1. Stop Splunk Enterprise on the host from which you want to migrate. 2. Copy the entire contents of the $SPLUNK_HOME directory from the old host to the new host. 3. Install the appropriate version of Splunk Enterprise for the target platform. 4. Confirm that index configuration files (indexes.conf) contain the correct location and path specification for any non-default indexes. 5. Start Splunk Enterprise on the new instance. 6. Log into Splunk Enterprise with your existing credentials. 7. After you log in, confirm that your data is intact by searching it. While you can make an index read-only, in order to migrate properly, you will need to stop Splunk. ... View more

As you mentioned this is a bug. This message should be benign, particularly on a forwarder. However, if you need the two keys that are missing to utilize the signedAudit = true stanzas in inputs.conf, create the audit keys by issuing the following command from $SPLUNK_HOME/bin/: ./splunk createssl audit-keys OR if you don't need to sign any events, remove the [auditTrail] stanza from $SPLUNK_HOME/etc/system/default/audit.conf. ... View more

I am not sure what the cause is. I had seen this on a few cases recently where we saw those types of errors and had speculated on what may be happening. As it wasn't the reason for the case, we did not deep dive and chalked it up to a fluke. When I saw your answers post it made me consider that it may be a larger problem. Going through the current cases, I am seeing a few with the same local.meta errors. As I have data in hand, I am going to file a bug with our dev team to see if they understand what is happening. As soon as I get more information, I will share that with you. ## EDIT ## This is fixed in: 6.6.11 (SPL-136970) 7.0.6 (SPL-158503) 7.1.3 (SPL-158504) ... View more

This can happen if the cert used by Splunkd to talk to Mongod has expired. Verify your certs are valid. For example, to validate the expiration date for server.pem you can run: From $SPLUNK_HOME/etc/auth/ openssl x509 -enddate -noout -in ./server.pem Results: notAfter=Dec 10 14:017:25 2015 GMT In the example above, the cert is expired. If you want to create a new cert you can look at splunk createssl: $SPLUNK_HOME/bin/splunk help createssl An example: $SPLUNK_HOME/bin/splunk createssl server-cert -d $SPLUNK_HOME/etc/auth -n server -c cn.domain.com -l 2048 Simply adjust for your environment requirements/settings. Once the new cert is in place, you can test to confirm it is valid: From $SPLUNK_HOME/etc/auth/ openssl x509 -enddate -noout -in ./server.pem Results: notAfter=Aug 22 15:30:45 2019 GMT If it is now valid, restart Splunk and validate if KVStore is running: ps -ax | grep mongod 26108 ? Ssl 62:11 mongod --dbpath=/opt/splunk/var/lib/splunk/kvstore/mongo --port=8191 --timeStampFormat=iso8601-utc --smallfiles --oplogSize=200 --keyFile=/opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key --setParameter=enableLocalhostAuthBypass=0 --replSet=50D25A40-7DD2-4017-A223-732705AD4A96 --sslAllowInvalidHostnames --sslMode=preferSSL --sslPEMKeyFile=/opt/splunk/etc/auth/server.pem --sslPEMKeyPassword=xxxxxxxx --nounixsocket And also: $SPLUNK_HOME/bin/splunk _internal call /services/server/info |grep -i kvstore <s:key name="kvStoreStatus">ready</s:key> A third way to check is to run the following search from command line on the instance where you have the "Deployment Management Console / Monitor Console" Configured. $SPLUNK_HOME/bin/splunk search "| rest /services/server/info splunk_server=* | fields splunk_server, kvStoreStatus" splunk_server kvStoreStatus ------------------------- ------------- indexer01.domain.com ready indexer02.domain.com ready indexer03.domain.com ready indexer04.domain.com ready indexer05.domain.com ready ... View more

The DMC or "Distributed Management Console" was replaced by the "Monitoring Console" in 6.5. Similar functionality, with new additions/improvements in 6.5, but the app was renamed. ... View more

NT4 EOL was on Dec 31st 2004 which is before "Splunk" was founded. Going back through our available documentation, I am unable to find a UF version which supported that platform, but its possible that a very early version could have. Most definitely, none of our currently available products support it. ... View more