Splunk Search

How to prevent Splunk from streaming / chunking data to custom search commands?

pmunaret
Explorer

Hi all,

I encountered the problem in MLTK that the data from the search is passed in multiple chunks to my custom classifier (using the apply command). Interestingly enough, the fit command passes the entire dataframe to the apply method of the custom classifier as shown below.

Search with apply
index=test
| apply model

PID 10489 2022-06-12 22:35:13,604 INFO [mlspl.ECOD Logger] [apply] Length of dataframe: 50
PID 10489 2022-06-12 22:35:13,730 INFO [mlspl.ECOD Logger] [apply] Length of dataframe: 205
PID 10489 2022-06-12 22:35:13,821 INFO [mlspl.ECOD Logger] [apply] Length of dataframe: 41
Search with fit command
index=test
| fit ECOD date_hour into model

PID 8345 2022-06-12 22:27:50,867 INFO [mlspl.ECOD Logger] [apply] Length of dataframe: 296

The second one is the behavior I want since I need the data as a single batch. Setting "chunked=false" in the commands.conf to use the legacy protocol does not work because MLTK is not compatible with v1. Setting "streaming=false"  also has no effect. Does anyone know how I can prevent Splunk from splitting the data in multiple chunks?

Any help is appreciated! Thanks. 

Labels (1)
Tags (1)

rlalle
Engager

Were you able to come up with a better solution for this?

0 Karma

pmunaret
Explorer
I just tested it with the collapse command and it kinda works, however, I'm not a fan of using an internal command that is designed for debugging / testing.
 
Search with apply
index=test
| collapse
| apply model

PID 10489 2022-06-13 19:00:02,152 INFO [mlspl.ECOD Logger] [apply] Length of dataframe: 296
If anyone has a better way to do this, let me know. 🙂
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Using the Splunk Threat Research Team’s Latest Security Content

REGISTER HERE Tech Talk | Security Edition Did you know the Splunk Threat Research Team regularly releases ...

SplunkTrust | 2024 SplunkTrust Application Period is Open!

It's that time again, folks! That's right, the application/nomination period for the 2024 SplunkTrust is ...