Currently we do not have this capability. In the upcoming major release this capability may be added . ... View more

That is not the case today. ... View more

Starting Splunk Version 7.1 splunk has made changes to way the default admin account is created. Once of the key change is when you When you start the forwarder for the first time under most conditions, it prompts you to create an admin password. Like This appears to be your first time running this version of Splunk. An Admin password must be set before installation proceeds. Password must contain at least: * 8 total printable ASCII character(s). Please enter a new password: See http://docs.splunk.com/Documentation/Splunk/7.1.1/Security/Secureyouradminaccount in Securing Splunk for additional information about creating a secure password. Also refer : http://docs.splunk.com/Documentation/Forwarder/7.1.1/Forwarder/InstallaWindowsuniversalforwarderfromaZIPfile When you install the universal forwarder, you must create a password for the Splunk admin user. The installer does not create a password for the user. Think of a password and be ready to supply it when you perform the installation. If you do not supply a password during a silent installation, the universal forwarder can install without any users defined, which prevents login. You must then create a user-seed.conf file to fix the problem and restart the forwarder. ... View more

Here are few things to check: 1) Check if the delegation failed due to quota issues ? i.e with in the same minute we re-try twice.. 2) Also, check if you see connectivity issue between search head cluster and Captain. It could be because the SH's running the alerts are not able to connect to captain 3)Check the health of the members during that 5 alert fired period, like is the captain switching too fast. 4) Check if the members are all part of the show shcluster-status and the RT jobs have duplicate among members.When a captain receives information about searches in the cluster and it finds duplicate RT searches they should be cancelled. ... View more

Looks like Azure SNAT process will drop the connection entries once the connections is idle for 4 minutes. This termination is not graceful, as it is not sending a TCP Reset to the connection end points. This behavior matches with the problematic TCP streams in the tcpdumps reviewed for the mongodb port 8191, across multiple SHs. It seems, issue is possible if and only if the VM kernel's TCP keep alive timeout is higher than the Azure TCP Idle timeout. By the time VM stack starts sending keep alive, the SNAT already deleted the connection context. From the packet captures, we do see that the Splunk server instances are set with TCP keep-alive timeout value of 300 seconds. First keep-alive starts after 300 seconds. Azure VM public IP tcp out settings is set to default 4 mins VM kernel and the settings are as below [root@jetprodeastsh1 ~]# sysctl -a | grep net.ipv4.tcp_keepalive_time net.ipv4.tcp_keepalive_time = 7200 [root@jetprodeastsh1 ~]# sysctl -a | grep net.ipv4.tcp_keepalive_intvl net.ipv4.tcp_keepalive_intvl = 75 [root@jetprodeastsh1 ~]# sysctl -a | grep net.ipv4.tcp_keepalive_probes net.ipv4.tcp_keepalive_probes = 9 This states that the VM’s kernel has the higher tcp timeout than at the VM public IP. To resolve this issue we changed Change the net.ipv4.tcp_keepalive_time = 180 ... View more

Here are some observation and resolution for this issue We noticed errors in mongod.log and splunkd.log for Kvstore change indicated that log so we enabled DEBUG shows delete in Delete of ACCELERATE searches. To debug this issue we enable DEBUG for $HOME_SPLUNK/splunk set log-level KVStorageProvider -level DEBUG $HOME_SPLUNK/splunk set log-level KVStoreConfigurationProvider -level DEBUG To increase debugging for mongod we specified [kvstore] verbose = true With the debug enabled logs we were able to trace timeouts to their originating actions, which as expected were deletions of ACCELERATE searches from mongo via SavedSearchHistory 06-06-2018 09:46:30.264 +0000 DEBUG KVStorageProvider - Delete data for key='nobody;search;_ACCELERATE_EFE4643E-A7D0-4EB5-B688-624724C9CD14_search_xxt-admin_f11e1bbde0afa60a_ACCELERATE_' 06-06-2018 09:46:30.264 +0000 DEBUG KVStorageProvider - Internal read query='{ "_key" : "nobody;search;_ACCELERATE_EFE4643E-A7D0-4EB5-B688-624724C9CD14_search_xxt-admin_f11e1bbde0afa60a_ACCELERATE_", "_user" : "nobody" }' limit='1' skip='0' fields='{ "_id" : 0 }' sort='{ }' 06-06-2018 09:46:30.285 +0000 DEBUG KVStorageProvider - Internal read for query='{ "_key" : "nobody;search;_ACCELERATE_EFE4643E-A7D0-4EB5-B688-624724C9CD14_search_xxt-admin_f11e1bbde0afa60a_ACCELERATE_", "_user" : "nobody" }' found data='[ { "runtimes" : [ 1.291 ], "sched_time" : 1528277400, "_user" : "nobody", "_key" : "nobody;search;_ACCELERATE_EFE4643E-A7D0-4EB5-B688-624724C9CD14_search_xxt-admin_f11e1bbde0afa60a_ACCELERATE_" } ]' 06-06-2018 09:46:30.285 +0000 DEBUG KVStorageProvider - Delete data with query='{ "_key" : "nobody;search;_ACCELERATE_EFE4643E-A7D0-4EB5-B688-624724C9CD14_search_xxt-admin_f11e1bbde0afa60a_ACCELERATE_", "_user" : "nobody" }' 2018-06-06T09:46:30.286Z D QUERY [conn1249] Running query: query: { _key: "nobody;search;_ACCELERATE_EFE4643E-A7D0-4EB5-B688-624724C9CD14_search_xxt-admin_f11e1bbde0afa60a_ACCELERATE_", _user: "nobody" } sort: {} projection: { _id: 0 } skip: 0 limit: 1 2018-06-06T09:46:30.286Z D QUERY [conn1249] Only one plan is available; it will be run but will not be cached. query: { _key: "nobody;search;_ACCELERATE_EFE4643E-A7D0-4EB5-B688-624724C9CD14_search_xxt-admin_f11e1bbde0afa60a_ACCELERATE_", _user: "nobody" } sort: {} projection: { _id: 0 } skip: 0 limit: 1, planSummary: IXSCAN { _user: 1, _key: 1 } 2018-06-06T09:46:30.286Z I COMMAND [conn1249] query s_systemMX8edh8vqo2ngaR2K53MLFyt_SavedSrC2W6YtGw4V2Yg7Myvf8S3GC.c query: { $query: { _key: "nobody;search;_ACCELERATE_EFE4643E-A7D0-4EB5-B688-624724C9CD14_search_xxt-admin_f11e1bbde0afa60a_ACCELERATE_", _user: "nobody" }, $orderby: {} } planSummary: IXSCAN { _user: 1, _key: 1 } ntoreturn:1 ntoskip:0 nscanned:1 nscannedObjects:1 keyUpdates:0 writeConflicts:0 numYields:0 nreturned:1 reslen:208 locks:{ Global: { acquireCount: { r: 2 } }, MMAPV1Journal: { acquireCount: { r: 1 } }, Database: { acquireCount: { r: 1 } }, Collection: { acquireCount: { R: 1 } } } 0ms 06-06-2018 09:51:30.286 +0000 ERROR SavedSearchHistory - db_delete( "nobody;search;_ACCELERATE_EFE4643E-A7D0-4EB5-B688-624724C9CD14_search_xxt-admin_f11e1bbde0afa60a_ACCELERATE_" ) failed: An error occurred during the last operation ('deleteData', domain: '2', code: '4'): Failed to send "delete" command with database "s_systemMX8edh8vqo2ngaR2K53MLFyt_SavedSrC2W6YtGw4V2Yg7Myvf8S3GC": Failed to read 36 bytes from socket within 300000 milliseconds. 06-06-2018 09:51:30.286 +0000 ERROR KVStorageProvider - An error occurred during the last operation ('deleteData', domain: '2', code: '4'): Failed to send "delete" command with database "s_systemMX8edh8vqo2ngaR2K53MLFyt_SavedSrC2W6YtGw4V2Yg7Myvf8S3GC": Failed to read 36 bytes from socket within 300000 milliseconds. The mongod.log was showing messages like below 06-06-2018 10:18:13.457 +0000 WARN MongoDriver - stream: Failure to buffer 36 bytes: Failed to buffer 36 bytes within 300000 milliseconds. 06-06-2018 10:18:13.458 +0000 ERROR KVStorageProvider - An error occurred during the last operation ('deleteData', domain: '2', code: '4'): Failed to send "delete" command with database "s_systemMX8edh8vqo2ngaR2K53MLFyt_SavedSrC2W6YtGw4V2Yg7Myvf8S3GC": Failed to read 36 bytes from socket within 300000 milliseconds. The network trace showed Certain tcp streams which had many retransmissions and repeated duplicate acks though there seem to be missing packets. We can correlate these directly to mongo errors seen in splunkd.log ... View more

0001 , 002 .. is the log entries commit ids metadata1, metadata2 is a rolling record of current captaincy ... View more

Issue turned out to be - Search Head bind to two cluster Master , one was upgraded to 7.1 and other was not. Once the CM that was not upgraded was removed- all searches worked fine. @server.conf [clustering] master_uri = CM1:8089,CM2:8089 mode = searchhead ... View more

Additional Points to consider: Data rebalancing attempts to achieve an optimal balance, not a perfect balance rebalance per-index data based on specific index buckets number rather than total buckets number ... View more

1) Check for excess buckets and remove them. Rebalance does not run until we are done with removing excess buckets to prevent unnecessary rebalancing. https://CLUSTER_MASTER:8000/en-US/manager/system/clustering_bucket_details?tab=excess-buckets-tab index=_internal sourcetype=metrics group=subtask_counts name=cmmaster_service to_fix_excess 2) Check that if CMRepJob, CMChangeBucketjob_build, CMTruncJob jobs are running. If there are no buckets and no jobs running related to data rebalance, then something is blocking the rebalance process. index=_internal source=splunkd.log host=CM CMRepJob running job | timechart count by job 3) Next check this endpoint to see if any jobs are stuck here. If you see a bucket stuck here with the initial reason as "rebalance cluster buckets" and the latest reason as "bucket has a pending discard peer..." then you will need to resync the buckets against the discarding peer. curl -k -u ADMIN https://CLUSTER_MASTER:8089/services/cluster/master/fixup?level=remove_excess So now find the buckets under pending discard and look for the peer that lists state=PendingDiscard. curl -k -u ADMIN https://CLUSTER_MASTER:8089//services/cluster/master/buckets?filter=status=PendingDiscard Now resync the buckets against the discarding peers. curl -k -u ADMIN https://CLUSTER_MASTER:8089/services/cluster/master/control/control/resync_bucket_from_peer -d bucket_id=customer_metrics~260~0496D8DF-7666-48F1-8E98-2F5355493040 -d peer=C449637C-3731-4498-BA9F-DEB0A05B347B 4) If no buckets are in stuck in pending discard, check "latest reason" at this endpoint for clues. curl -k -u ADMIN https://CLUSTER_MASTER:8089//services/cluster/master/fixup?level=rebalance 5) Also check for stale replications that could be blocking current replications. curl -k -u ADMIN https://CLUSTER_MASTER:8089//services/cluster/master/replications 6) If all else fails, try to trigger rebalance again. ... View more

Here is a test I did. Created a sample test.log file with data for 2019 64.242.88.10 - - [07/Mar/2019:16:58:54 -0800] "GET /mailman/listinfo/administration HTTP/1.1" 200 6459 lordgun.org - - [07/Mar/2019:17:01:53 -0800] "GET /razor.html HTTP/1.1" 200 2869 64.242.88.10 - - [07/Mar/2019:17:09:01 -0800] "GET /twiki/bin/search/Main/SearchResult?scope=text®ex=on&search=Joris%20*Benschop[^A-Za-z] HTTP/1.1" 200 4284 64.242.88.10 - - [07/Mar/2019:17:10:20 -0800] "GET /twiki/bin/oops/TWiki/TextFormattingRules?template=oopsmore¶m1=1.37¶m2=1.37 HTTP/1.1" 200 11400 64.242.88.10 - - [07/Mar/2019:17:13:50 -0800] "GET /twiki/bin/edit/TWiki/DefaultPlugin?t=1078688936 HTTP/1.1" 401 12846 64.242.88.10 - - [07/Mar/2019:17:16:00 -0800] "GET /twiki/bin/search/Main/?scope=topic®ex=on&search=^g HTTP/1.1" 200 3675 64.242.88.10 - - [07/Mar/2019:17:17:27 -0800] "GET /twiki/bin/search/TWiki/?scope=topic®ex=on&search=^d HTTP/1.1" 200 5773 lj1036.inktomisearch.com - - [07/Mar/2019:17:18:36 -0800] "GET /robots.txt HTTP/1.0" 200 68 lj1090.inktomisearch.com - - [07/Mar/2019:17:18:41 -0800] "GET /twiki/bin/view/Main/LondonOffice HTTP/1.0" 200 3860 64.242.88.10 - - [07/Mar/2019:17:21:44 -0800] "GET /twiki/bin/attach/TWiki/TablePlugin HTTP/1.1" 401 12846 64.242.88.10 - - [07/Mar/2019:17:22:49 -0800] "GET /twiki/bin/view/TWiki/ManagingWebs?rev=1.22 HTTP/1.1" 200 9310 64.242.88.10 - - [07/Mar/2019:17:23:54 -0800] "GET /twiki/bin/statistics/Main HTTP/1.1" 200 808 64.242.88.10 - - [07/Mar/2019:17:26:30 -0800] "GET /twiki/bin/view/TWiki/WikiCulture HTTP/1.1" 200 5935 Today is 03/29/2018 17:26:11 After that data is indexed we need to check if the data is in future. source="*test.log.log" host="test" sourcetype="apache" | eval delay=_indextime-_time | convert ctime(_time) AS xtime |convert ctime(_indextime) AS indextime | table _raw delay _time indextime xtime It shows that data is early by 773981. We can see same information in metrics.log 03-29-2018 17:26:12.099 -0700 INFO Metrics - group=per_sourcetype_thruput, series="test", kbps=0.1403387022431233, eps=1.258039598828591, kb=4.3505859375, ev=39, avg_age=776259.9487179487, max_age=778822 03-29-2018 17:26:12.098 -0700 INFO Metrics - group=per_host_thruput, series="test", kbps=0.1403387022431233, eps=1.258039598828591, kb=4.3505859375, ev=39, avg_age=776259.9487179487, max_age=778822 ... View more

Here is a situation when this may happen let’s say you push a Indexer Cluster bundle from Cluster Master. When the bundle is pushed from the Cluster Master, it compiled the list of peers that are up and where the bundle has to be pushed. After that Cluster master wait g for all peers to return the status after applying the bundle, if it does not happen the Cluster master will not get out of the Maintenance mode. To confirm if this is your situation run (splunk show cluster-bundle-status) and check which of the client is not on the latest bundle and fix the issue with ot , and restart ... View more