Splunk Search

When searching a CSV import via a lookup, what should I do to work around the line limit?

christianubeda
Path Finder

Hi team!

I import a CSV file via lookup and use this search.

index=cesa_paloalto sourcetype="pan:traffic" type=TRAFFIC vendor_action=allow | join src_ip [| inputlookup append=t ipsmalware2.csv | eval src_ip=Ip]
 | stats values(src_ip)

It should match IPs, but if I write an IP in the end of the file, it never matches!

The file is 25MB and 222916 lines

Splunk has a limit? It can't read all lines, I think.

What should I do?

Thank you!

0 Karma

DanielAmlung
Explorer

I would like to push this Question back to top. I have a similiar Problem, with importing a csv file. Splunk also "cuts" the Import after 20.000 lines. Is there some sort of max supported lines withing an csv file?

0 Karma

vishaltaneja070
Motivator

Hello @christiabubedu

There is limit of 10000 records or 60 sec on Subsearch. Thats why the events are not able to match for you.

So to avoid this situation, the best way is to use dedup or filter you data as per your need. try to use dedup for Ip in subsearch which will help you.

index=cesa_paloalto sourcetype="pan:traffic" type=TRAFFIC vendor_action=allow | join src_ip [| inputlookup append=t ipsmalware2.csv fields Ip | dedup Ip | eval src_ip= Ip]
| stats values(src_ip)
0 Karma

mayurr98
Super Champion

can you try this :

index=cesa_paloalto sourcetype="pan:traffic" type=TRAFFIC vendor_action=allow  [| inputlookup ipsmalware2.csv | table Ip | dedup Ip | rename Ip as src_ip] | stats values(src_ip)

OR

 index=cesa_paloalto sourcetype="pan:traffic" type=TRAFFIC vendor_action=allow | search   [| inputlookup ipsmalware2.csv | table Ip | dedup Ip | rename Ip as src_ip] | stats values(src_ip)

let me know if this helps!

0 Karma

christianubeda
Path Finder

Hi sir!

It didn't work...

I introduced 5 IP's in my csv

Line 1 OK
Line 9999 OK
Line 10004 Fail
Line 12000 FAIL
Line 222333(last one) FAIL

It only read first 10000 lines I think...

I need a csv with 9 millions lines... what can I do now?

0 Karma
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...