Installation

New indexer sync is very slow

rayar
Path Finder

We have the following architecture

1 SearchHead
1 Cluster Master

8 Indexers 

1 deployment server  

I am now added 2 new indexers , I see its syncing but very slow 

2021-02-03_10-00-01.png


[splunk@ilissplidx10 local]$ cat server.conf
[general]
parallelIngestionPipelines=2

[queue=typingQueue]
maxSize = 20MB

[queue=indexQueue]
maxSize = 30MB

[queue=aggQueue]
maxSize = 30MB

[queue=parsingQueue]
maxSize = 30MB

[clustering]
cxn_timeout = 600
send_timeout = 600
rcv_timeout = 600
heartbeat_period = 10


[kvstore]
disabled = true
[splunk@ilissplidx10 local]$ cat limits.conf
[default]
max_mem_usage_mb = 600
#
[search]
#dispatch_dir_warning_size = 3500
base_max_searches = 60
# # ERROR: Events may not be returned in sub-second order due to memory pressure.
max_rawsize_perchunk = 200000000
#
[pdf]
max_rows_per_table = 10000
#
[scheduler]
max_searches_perc = 100
#
[join]
subsearch_maxout = 500000
#
[realtime]
indexed_realtime_use_by_default = true
[splunk@ilissplidx10 local]$ cat distsearch.conf
[distributedSearch]

statusTimeout = 20

[splunk@ilissplidx10 local]$

Labels (1)
0 Karma
1 Solution

soutamo
SplunkTrust
SplunkTrust

Only side effects which  I have realised is some missing/duplicate events when you are doing searches when rebalancing is running.

r. Ismo

View solution in original post

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Please define "very slow".  How long had the resync been underway when the screen shot was taken?  How much data is syncing (we can see bucket counts, but not the sizes of the buckets)?

---
If this reply helps you, an upvote would be appreciated.
0 Karma

rayar
Path Finder

hi today i see [rayar@ilissplidx10 ~]$ df -h /splunk-hot

Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_splunk-lv_splunk 9.0T 260G 8.8T 3% /splunk-hot [rayar@ilissplidx10 ~]$

and only 1500 buckets 

0 Karma

soutamo
SplunkTrust
SplunkTrust

You have single site cluster?

And you have started rebalancing with 

splunk rebalance cluster-data -action start 

And what is your rebalancing target % ?

splunk list cluster-config | egrep rebalance_threshold

 

And which kind of load and memory sage you have on those nodes?

https://docs.splunk.com/Documentation/Splunk/8.1.1/Indexer/Rebalancethecluster

r. Ismo

0 Karma

rayar
Path Finder

Should I run the commend on the new indexers after the installation  ? 

splunk rebalance cluster-data -action start 

 

[splunk@ilissplmstr01 bin]$ splunk list cluster-config | egrep rebalance_threshold
Your session is invalid. Please login.
Splunk username: rayar
Password:
rebalance_threshold:1
[splunk@ilissplmstr01 bin]$

 

top - 10:33:41 up 19:13, 1 user, load average: 0.98, 1.25, 1.54
Tasks: 936 total, 1 running, 935 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.5 us, 0.1 sy, 0.0 ni, 99.2 id, 0.2 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 10562880+total, 93134956+free, 8376572 used, 11656192+buff/cache
KiB Swap: 3145724 total, 3145724 free, 0 used. 10461928+avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
11722 splunk 20 0 1876928 93572 11052 S 25.2 0.0 0:00.83 splunkd
11687 splunk 20 0 3200000 88324 11416 S 15.2 0.0 0:00.60 splunkd
40912 splunk 20 0 11.6g 870976 25620 S 8.6 0.1 367:39.00 splunkd
11693 splunk 20 0 1519856 69820 10488 S 3.0 0.0 0:00.21 splunkd
11690 splunk 20 0 209136 70184 10476 S 2.6 0.0 0:00.21 splunkd

 

0 Karma

soutamo
SplunkTrust
SplunkTrust

Rebalance must start on CM as check for threshold.

Load should check on those indexers also. You probably have MC (monitoring console) in use where you can check resource usage etc.?

Which kind of disks you have on those indexers? Those must deliver at least 800 IOPS by Splunk volumes (currently recommendations is 1200, if I recall right).

0 Karma

rayar
Path Finder

should I run 

splunk rebalance cluster-data -action start  on the cluster master  ? 
strange but I see now that in the monitoring console I don't see the new indexers
what configuration file holds this data ?
 

2021-02-04_14-17-02.png

Tags (1)
0 Karma

soutamo
SplunkTrust
SplunkTrust

You can see and estimate that by the next query:

index=_internal host=<YOUR CM> sourcetype=splunkd component=CMMaster "Starting rebalance" OR completion 
        | rex "percent=(?<pcnt>\d+.\d+)" 
        | convert num(pcnt) as x 
        | timechart minspan=30s max(x) as max_prc min(x) as min_prc |  fields - min_prc 
| predict max_prc as "Rebalance % forecast" future_timespan=200
| rename max_prc as "Rebalance % now"

 

To get those indexers to your CM dashboards you must first go to MC's general settings and apply those nodes to use. That updated needed information to add those to correct views.

0 Karma

rayar
Path Finder

 The search is failing 

index=_internal host=illinissplnkmaster.corp.amdocs.com sourcetype=splunkd component=CMMaster "Starting rebalance" OR completion
| rex "percent=(?<pcnt>\d+.\d+)"
| convert num(pcnt) as x
| timechart minspan=30s max(x) as max_prc min(x) as min_prc | fields - min_prc
| predict max_prc as "Rebalance % forecast" future_timespan=200
| rename max_prc as "Rebalance % now"

with command="predict", No data

I do see the new indexers in the generic settings 

 

2021-02-04_15-43-54.png

0 Karma

soutamo
SplunkTrust
SplunkTrust

You must apply those new indexers here to update those to correct dashboards/selections. 

0 Karma

rayar
Path Finder

Is there any risk in running 

splunk rebalance cluster-data -action start  

 

Tags (1)
0 Karma

soutamo
SplunkTrust
SplunkTrust

Only side effects which  I have realised is some missing/duplicate events when you are doing searches when rebalancing is running.

r. Ismo

View solution in original post

0 Karma

rayar
Path Finder

thanks a lot 

I was able to sync 

0 Karma
Take the 2021 Splunk Career Survey

Help us learn about how Splunk has
impacted your career by taking the 2021 Splunk Career Survey.

Earn $50 in Amazon cash!