All Apps and Add-ons

Apply command on a large field

KrithikaRamakri
Explorer

Hi everyone, I am trying to apply logistic regression to predict phishing based on a baseline of phishing emails data. But, the issue I am facing is that, the apply command execution inside Splunk is not consistent, it was working fine, but now, the job is stuck at Finalizing.
When I inspected the job, it has these 2 errors.

  1. With the python csv module - Error: field larger than field limit splunk
  2. With the apply command - Error in 'apply' command: Failed to load model

I have tried clearing the cache, recreating the fit and apply model, nothing works. Not sure how to resolve this issue. Can someone please help me on this?

grana_splunk
Splunk Employee
Splunk Employee

Can you check the mlspl.log? Its under $SPLUNK_HOME/var/log/splunk and share the error message here.

0 Karma

skoelpin
SplunkTrust
SplunkTrust

This is most likely a hardware limitation. Have you attempted to predict then do a partial fit on new data coming in? How big is your data set? How far back are you looking? Have you attempted to normalize your data through pre-processing? Whats your hardware look like?

https://docs.splunk.com/Documentation/MLApp/3.3.0/API/Methodcallingconvention

0 Karma

jkat54
SplunkTrust
SplunkTrust

I think this can be resolved by increasing the default kv limit value as described here:

http://docs.splunk.com/Documentation/Splunk/6.5.2/Data/Extractfieldsfromfileswithstructureddata#Stru...

Can you try increasing kv in limits.conf, restarting splunk and then trying again?

Thanks,
Jkat

0 Karma

jkat54
SplunkTrust
SplunkTrust

I think this can be resolved by increasing the default kv limit value as described here:

http://docs.splunk.com/Documentation/Splunk/6.5.2/Data/Extractfieldsfromfileswithstructureddata#Stru...

Can you try increasing kv in limits.conf, restarting splunk and then trying again?

Thanks,
Jkat

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!