i) Try to work out what error it hits, use online xml validator. ii) If it's not something you can't fix due to the mal-formatted xml then try to use csv format in the request.. for example in kwargs_export add "output_mode":"csv" . kwargs_export = {"earliest_time": "-7d", "latest_time": "now", "search_mode": "normal", "output_mode": "csv" } ... View more

It was caused by the 404 error captured in red on the bottom of the screen, which tells us not able to find the lookup file you used during the test. It happens as the new app that was created doesn't allow system access to its objects. The error message is a bit misleading or not accurate enough for you to take any remediation actions. You can fix it by exporting the newly created app you used in the step #5.b, such as in metadata/local.meta, [] export=system Or [managed_configurations/lookup%3ALOOKUPName] export = system ... View more

The error was generated by the receiving SMTP server complaining that the email hit the size limit on the server - it could have been set on server level or per user or team, role basis, or sender/receiver basis it could vary. Please contact your mail system admin to discuss it. Before that you can go and check the size of attachment which can be found in the path below ; results_file="/opt/splunk/var/run/splunk/dispatch/scheduler_username_aAAAAAAAAQ_RMD1234556ABCDEFG_at_1567108320_86782_0972A208-ABCD-4000-98B2-ABCD046F3B4A/results.csv.gz Emails get encrypted which could increase the size by 30% or so. Or you can try to reduce the size by changing the search commands or time-ranges. ... View more

There are a few ways to clean up the cache; i) Restart of splunk, ii) Or run comand below; ./splunk _internal call /authentication/providers/services/_reload -auth admin:changeme iii) Or hit the rest endpoint; "| rest splunk_server=* /services/authentication/providers/services/_reload " ... View more

To fix the issue temporarily you need to restart manually the rest that haven't been restarted, that will allow them to pick up the new shcluster bundle successfully. For versions available as of this writing the captain changes in the middle of rolling restart can happen but you need to find out why this is happening and get it fixed. Enhancement work is going on currently. To understand the happenings between deployer, captain and members during the rolling restart check the below; i) Deployer pushes bundle; index=_internal host="SHC members" source=*/splunkd.log* "does not support reload" | dedup host| sort _time|table host _time ii) Captain initiates rolling-restart of peers index=_internal source=*/splunkd.log* "Starting a rolling restart of the peers" host=CaptainName iii) SHC peers are instructed to restart index=_internal host="SHC members" source=*/splunkd.log* | table _time host | sort _time iv) Visit then captain and the new captain to find any abnormal activities. On the search head, searchhead11 05-12-2019 23:18:25.740 -0400 INFO SHCSlave - event=SHPSlave::handleHeartbeatDone master has instructed peer to restart … 05-12-2019 23:36:37.546 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_DFM" … 05-12-2019 23:41:03.191 -0400 INFO IndexProcessor - handleSignal : Disabling streaming searches. 05-12-2019 23:41:03.191 -0400 INFO SHClusterMgr - Starting to Signal shutdown RAFT 05-12-2019 23:41:03.191 -0400 INFO SHCRaftConsensus - Shutdown signal received. 05-12-2019 23:41:03.191 -0400 INFO SHClusterMgr - Signal shutdown RAFT completed 05-12-2019 23:46:32.401 -0400 ERROR AdminHandler:ServerControl - forcing shutdown since it did not complete in 600.000 seconds 05-12-2019 23:46:50.716 -0400 WARN ConfMetrics - single_action=BASE_INITIALIZE took wallclock_ms=1905 05-12-2019 23:46:50.729 -0400 INFO ServerConfig - My GUID is ABCDEFGH-1234-ABCD-A0A7-E8DA24D123456 v) On the new captain, check the messages from the component "SHCRaftConsensus" to find the raft voting activities. Hopefully that would tell us why the captaincy change happened. Then captain might have trouble to talk to some members, like below - find the below as an example. In splunkd.log 05-12-2019 22:36:30.573 -0500 ERROR SHCRaftConsensus - failed appendEntriesRequest err: uri=https://searchhead11:8089/services/shcluster/member/consensus/pseudoid/raft_append_entries?output_mode=json, socket_error=Connect Timeout to https://searchhead11:8089 05-12-2019 22:41:47.666 -0500 ERROR SHCRaftConsensus - 70 consecutive appendEntriesFailures to https://searchhead11:8089 Once you understand what happened during the bundle push then you may be able to work out what triggered the issue. In general the captain changes happens by either captaincy transfer command or by timeouts by raft layer. If you see this kind of issue frequently check "election_timeout_ms" under "shclustering" stanza and increase it to higher than cxn_timeout, rcv_timeout and send_timeout. ... View more

It wants to say "Cache is full and cache space couldn't be reserved". And the log message is a bit misleading not accurate enough - The error messaging sent to search incorrectly states we've hit the 2GB minfreespace when we've actually just unable to reserve space before download. With version 7.3.x it should be "Cache was full and space could not be reserved" When the Cache manager is not able to evict buckets check the below; 1) is the disk full ? 2) is max_cache_size too low ? 3) is eviction_padding is too high ? 4) are buckets being uploaded to remote storage? Buckets that haven't been uploaded cannot be evicted. 5) is disk full of hot buckets? Try a restart for rolling to warm. 6) is the cache large enough to support the data volume that is searched. 7) is the data balanced and are the primaries balanced (assuming clustered) If none of them helps please open a support ticket with the details as well as diags. ... View more

This could have been caused by some corrupted buckets when searches run against them. You may want to fix the buckets and try the same search to see if it fixes it. Follow the steps below to get list of buckets suspected corrupted. *** How to get the list of corrupt buckets *** 1. @the indexer, cd to $SPLUNK_HOME/var/log/splunk 2. Run below $ grep "MAP:" crash-2019-07-31*.log |grep "/opt/splunk/storage" "/opt/splunk/storage" varies according to your deployment set up and is taken from the line below in crash log. crash-2019-07-31-00:15:17.log: MAP: 7f00e9cdb000-7f00ea000000 r--s 00000000 fd:03 563872524 /opt/splunk/storage/hot/myindex1/rb_1560184689_1559942722_7530_AAAAAAAA-BBBB-1111-8C82-ABAD1EDD033D/1560184689-1560184620-11473276039248555956.tsidx 3. It will return the problematic buckets. From the above example, the bucket location is /opt/splunk/storage/hot/myindex1/rb_1560184689_1559942722_7530_AAAAAAAA-BBBB-1111-8C82-ABAD1EDD033D *** How to fix the corrupted buckets *** Rebuilding the bucket using fsck should fix the problem. Follow the steps to rebuild buckets: 0. @CM, splunk enable maintenance-mode 1. @Anonymous, splunk offline 2. @Anonymous, for all the buckets from above, run splunk fsck repair --one-bucket --bucket-path="path_from_above" i.e: splunk fsck repair --one-bucket --bucket-path=/opt/splunk/storage/hot/myindex1/rb_1560184689_1559942722_7530_AAAAAAAA-BBBB-1111-8C82-ABAD1EDD033D 3. @Anonymous, splunk start 4. @CM, splunk disable maintenance-mode If this is not helping improve the situation please contact Splunk Support with details of deployment architecture and a drag from the indexer. ... View more

This appears happening when it is copied over from other deployment or edited manually. Many cases of this kind have been resolved by editing it again via the Splunkweb GUI. It always has to be through the GUI. ... View more

Don't panic! Splunk is not actually deleting buckets but it is quite misleading. The bucket information in the CM will be removed while the buckets are there physically on the indexers file system. This is typically happening when the indexers get rolling restarted. ... View more

We have a plan to improve the behaviour of the modInput which will be done on version 5. Until then you can consider to tune it by modifying the timer value as below; ---- file: etc/apps/Splunk_TA_aws/bin/splunksdc/config.py 74 def has_expired(self): 75 now = time.time() 76 if now - self._last_check > 30: ### This is 30seconds timer you may want to try like, 20000 considering the previous credentials is valid for the next 6 hours ( 6 X 3600) ### 77 eelf._last_check = now 78 self._has_expired = self._check() ---- file: etc/apps/Splunk_TA_aws/bin/splunksdc/config.py This has reduced the chances of duplicate data by 95+ %. ... View more

As of this writing, it appears autoLBVolume is found not working as it is supposed to. You may want to remove it and instead add forceTimebasedAutoLB=true with smaller value for autoLBFrequency, like 10s - as small as. you can find better load-balancing across the indexers. ... View more

We have seen this issues from time to time and have been working on it to get to the bottom of issue. This can be worked around by using forcedTimebasedAutoLB = true until the fix is found and released. ... View more

It turns out to have been caused by debug option, debug_metrics=true - the symptoms disappeared after debug_metrics reverted back to false. 'debug_metrics=true' expands the amount of perf data in info.csv collected by the search process because it breaks down the perf data by indexers - have hundreds of indexers in multiple clusters. The aggregation does not occur and the data for each indexer is preserved. The perf data becomes hundred times greater and the search head loads data for analysis. Until any fix is available please monitor the memory usage of splunkd whenever you use 'debug_metrics = true' in limits.conf. ... View more