Installation

New indexer sync is very slow

rayar
Contributor

We have the following architecture

1 SearchHead
1 Cluster Master

8 Indexers 

1 deployment server  

I am now added 2 new indexers , I see its syncing but very slow 

2021-02-03_10-00-01.png


[splunk@ilissplidx10 local]$ cat server.conf
[general]
parallelIngestionPipelines=2

[queue=typingQueue]
maxSize = 20MB

[queue=indexQueue]
maxSize = 30MB

[queue=aggQueue]
maxSize = 30MB

[queue=parsingQueue]
maxSize = 30MB

[clustering]
cxn_timeout = 600
send_timeout = 600
rcv_timeout = 600
heartbeat_period = 10


[kvstore]
disabled = true
[splunk@ilissplidx10 local]$ cat limits.conf
[default]
max_mem_usage_mb = 600
#
[search]
#dispatch_dir_warning_size = 3500
base_max_searches = 60
# # ERROR: Events may not be returned in sub-second order due to memory pressure.
max_rawsize_perchunk = 200000000
#
[pdf]
max_rows_per_table = 10000
#
[scheduler]
max_searches_perc = 100
#
[join]
subsearch_maxout = 500000
#
[realtime]
indexed_realtime_use_by_default = true
[splunk@ilissplidx10 local]$ cat distsearch.conf
[distributedSearch]

statusTimeout = 20

[splunk@ilissplidx10 local]$

Labels (1)
0 Karma
1 Solution

isoutamo
SplunkTrust
SplunkTrust

Only side effects which  I have realised is some missing/duplicate events when you are doing searches when rebalancing is running.

r. Ismo

View solution in original post

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Please define "very slow".  How long had the resync been underway when the screen shot was taken?  How much data is syncing (we can see bucket counts, but not the sizes of the buckets)?

---
If this reply helps you, Karma would be appreciated.
0 Karma

rayar
Contributor

hi today i see [rayar@ilissplidx10 ~]$ df -h /splunk-hot

Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_splunk-lv_splunk 9.0T 260G 8.8T 3% /splunk-hot [rayar@ilissplidx10 ~]$

and only 1500 buckets 

0 Karma

isoutamo
SplunkTrust
SplunkTrust

You have single site cluster?

And you have started rebalancing with 

splunk rebalance cluster-data -action start 

And what is your rebalancing target % ?

splunk list cluster-config | egrep rebalance_threshold

 

And which kind of load and memory sage you have on those nodes?

https://docs.splunk.com/Documentation/Splunk/8.1.1/Indexer/Rebalancethecluster

r. Ismo

0 Karma

rayar
Contributor

Should I run the commend on the new indexers after the installation  ? 

splunk rebalance cluster-data -action start 

 

[splunk@ilissplmstr01 bin]$ splunk list cluster-config | egrep rebalance_threshold
Your session is invalid. Please login.
Splunk username: rayar
Password:
rebalance_threshold:1
[splunk@ilissplmstr01 bin]$

 

top - 10:33:41 up 19:13, 1 user, load average: 0.98, 1.25, 1.54
Tasks: 936 total, 1 running, 935 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.5 us, 0.1 sy, 0.0 ni, 99.2 id, 0.2 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 10562880+total, 93134956+free, 8376572 used, 11656192+buff/cache
KiB Swap: 3145724 total, 3145724 free, 0 used. 10461928+avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
11722 splunk 20 0 1876928 93572 11052 S 25.2 0.0 0:00.83 splunkd
11687 splunk 20 0 3200000 88324 11416 S 15.2 0.0 0:00.60 splunkd
40912 splunk 20 0 11.6g 870976 25620 S 8.6 0.1 367:39.00 splunkd
11693 splunk 20 0 1519856 69820 10488 S 3.0 0.0 0:00.21 splunkd
11690 splunk 20 0 209136 70184 10476 S 2.6 0.0 0:00.21 splunkd

 

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Rebalance must start on CM as check for threshold.

Load should check on those indexers also. You probably have MC (monitoring console) in use where you can check resource usage etc.?

Which kind of disks you have on those indexers? Those must deliver at least 800 IOPS by Splunk volumes (currently recommendations is 1200, if I recall right).

0 Karma

rayar
Contributor

should I run 

splunk rebalance cluster-data -action start  on the cluster master  ? 
strange but I see now that in the monitoring console I don't see the new indexers
what configuration file holds this data ?
 

2021-02-04_14-17-02.png

Tags (1)
0 Karma

isoutamo
SplunkTrust
SplunkTrust

You can see and estimate that by the next query:

index=_internal host=<YOUR CM> sourcetype=splunkd component=CMMaster "Starting rebalance" OR completion 
        | rex "percent=(?<pcnt>\d+.\d+)" 
        | convert num(pcnt) as x 
        | timechart minspan=30s max(x) as max_prc min(x) as min_prc |  fields - min_prc 
| predict max_prc as "Rebalance % forecast" future_timespan=200
| rename max_prc as "Rebalance % now"

 

To get those indexers to your CM dashboards you must first go to MC's general settings and apply those nodes to use. That updated needed information to add those to correct views.

0 Karma

rayar
Contributor

 The search is failing 

index=_internal host=illinissplnkmaster.corp.amdocs.com sourcetype=splunkd component=CMMaster "Starting rebalance" OR completion
| rex "percent=(?<pcnt>\d+.\d+)"
| convert num(pcnt) as x
| timechart minspan=30s max(x) as max_prc min(x) as min_prc | fields - min_prc
| predict max_prc as "Rebalance % forecast" future_timespan=200
| rename max_prc as "Rebalance % now"

with command="predict", No data

I do see the new indexers in the generic settings 

 

2021-02-04_15-43-54.png

0 Karma

isoutamo
SplunkTrust
SplunkTrust

You must apply those new indexers here to update those to correct dashboards/selections. 

0 Karma

rayar
Contributor

Is there any risk in running 

splunk rebalance cluster-data -action start  

 

Tags (1)
0 Karma

isoutamo
SplunkTrust
SplunkTrust

Only side effects which  I have realised is some missing/duplicate events when you are doing searches when rebalancing is running.

r. Ismo

0 Karma

rayar
Contributor

thanks a lot 

I was able to sync 

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...