When searching a CSV import via a lookup, what should I do to work around the line limit?

[christianubeda]

Hi team!

I import a CSV file via lookup and use this search.

index=cesa_paloalto sourcetype="pan:traffic" type=TRAFFIC vendor_action=allow | join src_ip [| inputlookup append=t ipsmalware2.csv | eval src_ip=Ip]
 | stats values(src_ip)

It should match IPs, but if I write an IP in the end of the file, it never matches!

The file is 25MB and 222916 lines

Splunk has a limit? It can't read all lines, I think.

What should I do?

Thank you!

[DanielAmlung]

I would like to push this Question back to top. I have a similiar Problem, with importing a csv file. Splunk also "cuts" the Import after 20.000 lines. Is there some sort of max supported lines withing an csv file?

[vishaltaneja070]

Hello @christiabubedu

There is limit of 10000 records or 60 sec on Subsearch. Thats why the events are not able to match for you.

So to avoid this situation, the best way is to use dedup or filter you data as per your need. try to use dedup for Ip in subsearch which will help you.

index=cesa_paloalto sourcetype="pan:traffic" type=TRAFFIC vendor_action=allow | join src_ip [| inputlookup append=t ipsmalware2.csv fields Ip | dedup Ip | eval src_ip= Ip]
| stats values(src_ip)

[mayurr98]

can you try this :

index=cesa_paloalto sourcetype="pan:traffic" type=TRAFFIC vendor_action=allow  [| inputlookup ipsmalware2.csv | table Ip | dedup Ip | rename Ip as src_ip] | stats values(src_ip)

OR

 index=cesa_paloalto sourcetype="pan:traffic" type=TRAFFIC vendor_action=allow | search   [| inputlookup ipsmalware2.csv | table Ip | dedup Ip | rename Ip as src_ip] | stats values(src_ip)

let me know if this helps!

[christianubeda]

Hi sir!

It didn't work...

I introduced 5 IP's in my csv

Line 1 OK
Line 9999 OK
Line 10004 Fail
Line 12000 FAIL
Line 222333(last one) FAIL

It only read first 10000 lines I think...

I need a csv with 9 millions lines... what can I do now?