We have the following architecture
1 SearchHead
1 Cluster Master
8 Indexers
1 deployment server
I am now added 2 new indexers , I see its syncing but very slow
[splunk@ilissplidx10 local]$ cat server.conf
[general]
parallelIngestionPipelines=2
[queue=typingQueue]
maxSize = 20MB
[queue=indexQueue]
maxSize = 30MB
[queue=aggQueue]
maxSize = 30MB
[queue=parsingQueue]
maxSize = 30MB
[clustering]
cxn_timeout = 600
send_timeout = 600
rcv_timeout = 600
heartbeat_period = 10
[kvstore]
disabled = true
[splunk@ilissplidx10 local]$ cat limits.conf
[default]
max_mem_usage_mb = 600
#
[search]
#dispatch_dir_warning_size = 3500
base_max_searches = 60
# # ERROR: Events may not be returned in sub-second order due to memory pressure.
max_rawsize_perchunk = 200000000
#
[pdf]
max_rows_per_table = 10000
#
[scheduler]
max_searches_perc = 100
#
[join]
subsearch_maxout = 500000
#
[realtime]
indexed_realtime_use_by_default = true
[splunk@ilissplidx10 local]$ cat distsearch.conf
[distributedSearch]
statusTimeout = 20
[splunk@ilissplidx10 local]$
Only side effects which I have realised is some missing/duplicate events when you are doing searches when rebalancing is running.
r. Ismo
Please define "very slow". How long had the resync been underway when the screen shot was taken? How much data is syncing (we can see bucket counts, but not the sizes of the buckets)?
hi today i see [rayar@ilissplidx10 ~]$ df -h /splunk-hot
Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_splunk-lv_splunk 9.0T 260G 8.8T 3% /splunk-hot [rayar@ilissplidx10 ~]$
and only 1500 buckets
You have single site cluster?
And you have started rebalancing with
splunk rebalance cluster-data -action start
And what is your rebalancing target % ?
splunk list cluster-config | egrep rebalance_threshold
And which kind of load and memory sage you have on those nodes?
https://docs.splunk.com/Documentation/Splunk/8.1.1/Indexer/Rebalancethecluster
r. Ismo
Should I run the commend on the new indexers after the installation ?
splunk rebalance cluster-data -action start
[splunk@ilissplmstr01 bin]$ splunk list cluster-config | egrep rebalance_threshold
Your session is invalid. Please login.
Splunk username: rayar
Password:
rebalance_threshold:1
[splunk@ilissplmstr01 bin]$
top - 10:33:41 up 19:13, 1 user, load average: 0.98, 1.25, 1.54
Tasks: 936 total, 1 running, 935 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.5 us, 0.1 sy, 0.0 ni, 99.2 id, 0.2 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 10562880+total, 93134956+free, 8376572 used, 11656192+buff/cache
KiB Swap: 3145724 total, 3145724 free, 0 used. 10461928+avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
11722 splunk 20 0 1876928 93572 11052 S 25.2 0.0 0:00.83 splunkd
11687 splunk 20 0 3200000 88324 11416 S 15.2 0.0 0:00.60 splunkd
40912 splunk 20 0 11.6g 870976 25620 S 8.6 0.1 367:39.00 splunkd
11693 splunk 20 0 1519856 69820 10488 S 3.0 0.0 0:00.21 splunkd
11690 splunk 20 0 209136 70184 10476 S 2.6 0.0 0:00.21 splunkd
Rebalance must start on CM as check for threshold.
Load should check on those indexers also. You probably have MC (monitoring console) in use where you can check resource usage etc.?
Which kind of disks you have on those indexers? Those must deliver at least 800 IOPS by Splunk volumes (currently recommendations is 1200, if I recall right).
should I run
splunk rebalance cluster-data -action start on the cluster master ?
strange but I see now that in the monitoring console I don't see the new indexers
what configuration file holds this data ?
You can see and estimate that by the next query:
index=_internal host=<YOUR CM> sourcetype=splunkd component=CMMaster "Starting rebalance" OR completion
| rex "percent=(?<pcnt>\d+.\d+)"
| convert num(pcnt) as x
| timechart minspan=30s max(x) as max_prc min(x) as min_prc | fields - min_prc
| predict max_prc as "Rebalance % forecast" future_timespan=200
| rename max_prc as "Rebalance % now"
To get those indexers to your CM dashboards you must first go to MC's general settings and apply those nodes to use. That updated needed information to add those to correct views.
The search is failing
index=_internal host=illinissplnkmaster.corp.amdocs.com sourcetype=splunkd component=CMMaster "Starting rebalance" OR completion
| rex "percent=(?<pcnt>\d+.\d+)"
| convert num(pcnt) as x
| timechart minspan=30s max(x) as max_prc min(x) as min_prc | fields - min_prc
| predict max_prc as "Rebalance % forecast" future_timespan=200
| rename max_prc as "Rebalance % now"
with command="predict", No data
I do see the new indexers in the generic settings
You must apply those new indexers here to update those to correct dashboards/selections.
Only side effects which I have realised is some missing/duplicate events when you are doing searches when rebalancing is running.
r. Ismo
thanks a lot
I was able to sync