Hi Blake,
This is an interesting algorithm!
I am proposing two solutions for you:
MLTK/SPL only:
Single-linkage clustering is not yet a supported algorithm in the current release of MLTK. But MLTK offers some other clustering algorithms such as kmeans, spectral clustering, DBSCAN and Birch. So if you don't mind trying one of the supported clustering algorithms, then here is one possible solution: (1) transpose your raw data to flip events/fields, (2) calculate differences per pair of columns and generate the M * N^2/2 matrix (you may need SPL commands such as map, join, and/or foreach), (3) perform binary classification via fit command, (4) since step (1)-(3) can be done using SPL and ML-SPL, you can copy/paste the SPL into the search bar in the Clustering dashboard of MLTK and try out different supported clustering methods.
Via ML-SPL API:
If you already have your custom script ready to perform the transformation and clustering as you described, you can wire it up with ML-SPL and it could be more convenient than the first method. A reference script will be SpectralClustering.py and DBSCAN.py in the Splunk_ML_Toolkit/bin/algos directory, where you can follow the way fit_predict is implemented and replace it with your own script.
Hope it's useful.
... View more