About andynewsoncap

andynewsoncap · ‎06-12-2023

Yes complexity of your architecture. But not if you break it down. Forget about all my Cluster. Simple question. How can I find out the Daily Size and Compression Ratio for an Index. I know you said its (0.15 x RF + 0.35 x SF) but i not sure that's is right. But it may well work given I need to allow fudge factor. Then expand the question. To all the Indexers on a given set of Indexers. Something like. What is the size of each index on a given set of Indexer for the last 30 days. Them Simply take that answer / 30 to give an approximate daily ingestion size in MB. Then take the (0.15 x RF + 0.35 x SF) to work up a number. And times that but a number of days for Hot/Warm and a number of Days for Cold. i.e. homePath.maxDataSizeMB = And coldPath.maxDataSizeMB = And maxTotalDataSizeMB =

andynewsoncap · ‎06-12-2023

In fact we have 9 Index Clusters. 2 in US. 5 node Cluster and 3 node cluster 2 in APAC. 4 node Cluster and 3 node cluster 2 in LATAM. 4 node Cluster and 3 node cluster 2 in EMEA. 5 node Cluster and 3 node cluster All reporting to one 9 node Search head cluster. Which has its own 4 node ITSI Indexing Cluster. 2+ 2 + 2 + 2 + 1 There are about 40,000 devices feeding it Data. lets say 10k / 10k / 10k / 10k All are set to 2 / 2. SF / RF. Each Index Cluster has its own set of Indexes / Index names based on the Data it is collecting for that region. So I can tell if this is EMEA data or USA Data. etc.etc. Based of the source of the Indexer. The customer has a habit of spinning up server in one region but colleting data for another region. Yer dont ask. Customer is always right. Right 🙂 For complex reason i will not go into here. But this is how myself and Splunk PS designed it 3 years ago. And its been running for 3 years now. Now we need to ensure we are keeping the data for an appropriator amount of time. Which we are not.

andynewsoncap · ‎06-12-2023

Thanks. We can't be the only people to ask this. Oh and we designed it with Splunk PS. And there are very good reasons why we have so many indexes. 5 Index Cluster so spread-out so 70-100 per cluster. I worked out I would have to take RF and SF into account. I was hoping someone may know how to pull size and compression factor from an SPL. If not next stop Splunk on-demand i guess. Thanks.

andynewsoncap · ‎06-12-2023

Hello, as far as I can understand and please correct me if I am wrong. How an index behaves is based on it’s conf. We have 5 IDX cluster and over 300 IDXes at this stage. Our AIM is to keep 9 days (or there about) in Hot/Warm and 31days (or there about in Cold) One day maybe keep data in Frozen (but we are not there yet). So as far as I understand in Splunk this is controlled by working out the Size per day of the data. Then Take that number and x by 9 and x by 31. To create homePath.maxDataSizeMB = And coldPath.maxDataSizeMB = And maxTotalDataSizeMB = Then finally frozenTimePeriodInSecs = To further complicate matters there is a compress ration for RAW and the compression ratio for TSIDX and the number of Indexers in the cluster and the replication factor. All of which make for some fun and complicate calculation. My question is this. Have anyone come up with a way to do this? Or at has someone worked out how to extract a list of IDX per INDEX Cluster and the current daily Data Rate. And doing the same to extract per IDX per INDEX Cluster the compression Ratio of the RAX Data and the TSIDX Then maybe its possible to do some magic in Excel to build out Per IDX homePath.maxDataSizeMB = coldPath.maxDataSizeMB = maxTotalDataSizeMB = Thanks.

andynewsoncap · ‎09-30-2022

Ok so what about asking is it possible to pull back the real time from one Splunk infrastructure component.

andynewsoncap · ‎09-29-2022

I just smile at this point 🙂

andynewsoncap · ‎09-29-2022

Hello, Anyone know if its possible to pull back the time from all the Splunk infrastructure. I have over 200 IDX / SHD / DEP etc etc server. In 4 Regions around the world. And I think my NTP is failing / drifting. And I what to show my IT Dept the problem if we have one. So is it possible to ask, all the Splunk infrastructure the time. So I can see / show at a glance oh that IDX server is 5 mins out from its cluster buddy's? Thanks.

andynewsoncap · ‎08-15-2022

Hello, I have data being gather one per min. FYI its disk usage %. Is it possible to create an SPL that output simple time from _time and UsePct every time UsePct changes. Not dedup it well yes but only when it (UsePct) changes. So if on a give date / hour / min it goes up or down. I can track the change. i.e. 2022-08-15 07:54:29 100% 2022-08-15 07:55:29 100% 2022-08-15 07:56:29 100% 2022-08-15 07:57:29 100% 2022-08-15 07:58:29 99% 2022-08-15 08:00:29 100% 2022-08-15 08:01:29 100% 2022-08-15 08:02:29 100% For this i would see 2022-08-15 07:57:29 100% 2022-08-15 07:58:29 99% 2022-08-15 07:59:29 100% Close as i can get it this ((index=windows OR index=perfmon OR index=os*) tag=oshost tag=performance tag=storage) host=by0saq Filesystem="/dev/mapper/vgappl-_u01_app" | eval date=strftime(_time,"%x") | sort _time | table date UsePct | dedup date Thanks.

andynewsoncap · ‎02-03-2022

I not saying this fixed it. but this made it go away for now. I set on the CM ./splunk edit cluster-config -max_peer_build_load 8 ./splunk edit cluster-config -max_peer_rep_load 20 I put the CM into maint mode. And on the indexer I set max_replication_errors to 10 from 3 I then rebooted each indexer just to get them as clean as possible. Once I had rebooted each Index and all where up and working. I restart the CM and took it out of Maint mode. After about 3 hours is sorted itself. then i reset back to the original values..

andynewsoncap · ‎01-28-2022

Platform has been live for close to two year. Firewall ports are still open. MTU is still 1500 has not been changed. No error in the OS logs. 3 of my nine clusters started doing this. In fact 7 our of 9 have search factor not meet errors. Which where fixed by putting CM into main. manual rolling restart of the Indexer and restarting CM and taking it out of Main mode. But I ma left with three with 1000's of bucket fix ups and this error now.

Posts	10
Solutions	0
Karma Given	0
Karma Received	0
Member Since	‎06-08-2021

Online Status	Offline
Date Last Visited	‎06-12-2023 01:10 PM

How to tune index retention?

Anyone know if its possible to pull back the time ...

How to only show when a value changes?

Too many bucket replication errors to target peer

Re: How to tune index retention?

Re: How to tune index retention?

Re: How to tune index retention?

How to tune index retention?

Re: Time on Splunk infrastructure

Re: Time on Splunk infrastructure

Anyone know if its possible to pull back the time ...

How to only show when a value changes?

Re: Too many bucket replication errors to target p...

Too many bucket replication errors to target peer