topic Re: How come Splunk only can read 10000 lines from my csv? I need 9000000! in Getting Data In https://community.splunk.com/t5/Getting-Data-In/How-come-Splunk-only-can-read-10000-lines-from-my-csv-I-need/m-p/455740#M78829 <P>Hi Vijeta,</P> <P>I only have matches if the IP from my csv is in the first 10000 lines. Actually I have 9000000 lines in my csv. So It didn`t work.</P> <P>If I do that |inputlookup append=t ipsmalware2.csv I see al files. The problem is when I try to match them...</P> Thu, 07 Feb 2019 17:09:20 GMT christianubeda 2019-02-07T17:09:20Z How come Splunk only can read 10000 lines from my csv? I need 9000000! https://community.splunk.com/t5/Getting-Data-In/How-come-Splunk-only-can-read-10000-lines-from-my-csv-I-need/m-p/455733#M78822 <P>Hi team!</P> <P>I have a problem.</P> <P>I want to match two fields. The first one is an src_ip from an indexer(traffic events) the second one is an IP from a CSV.</P> <P>My CSV has 9.000.000 lines and inputlookup only can read the first 10.000 lines... </P> <P>how can I do it??</P> Thu, 07 Feb 2019 15:33:45 GMT https://community.splunk.com/t5/Getting-Data-In/How-come-Splunk-only-can-read-10000-lines-from-my-csv-I-need/m-p/455733#M78822 christianubeda 2019-02-07T15:33:45Z Re: How come Splunk only can read 10000 lines from my csv? I need 9000000! https://community.splunk.com/t5/Getting-Data-In/How-come-Splunk-only-can-read-10000-lines-from-my-csv-I-need/m-p/455734#M78823 <P>Are you using lookup in a subsearch, probably that is limiting it to 10K results as there is a max limit for subserach results. Can you use dedup on IP or avoid subsearch by any means? Also would be better if you can paste your query here.</P> Thu, 07 Feb 2019 16:23:08 GMT https://community.splunk.com/t5/Getting-Data-In/How-come-Splunk-only-can-read-10000-lines-from-my-csv-I-need/m-p/455734#M78823 Vijeta 2019-02-07T16:23:08Z Re: How come Splunk only can read 10000 lines from my csv? I need 9000000! https://community.splunk.com/t5/Getting-Data-In/How-come-Splunk-only-can-read-10000-lines-from-my-csv-I-need/m-p/455735#M78824 <P>This is the query</P> <P>index=cesa_paloalto sourcetype="pan:traffic" type=TRAFFIC vendor_action=allow | join src_ip [| inputlookup append=t ipsmalware2.csv | eval src_ip=Ip]<BR /> | stats values(src_ip)</P> <P>I introduced 5 IP's in my csv</P> <P>Line 1 OK<BR /> Line 9999 OK<BR /> Line 10004 Fail<BR /> Line 12000 FAIL<BR /> Line 222333(last one) FAIL</P> Tue, 29 Sep 2020 23:11:06 GMT https://community.splunk.com/t5/Getting-Data-In/How-come-Splunk-only-can-read-10000-lines-from-my-csv-I-need/m-p/455735#M78824 christianubeda 2020-09-29T23:11:06Z Re: How come Splunk only can read 10000 lines from my csv? I need 9000000! https://community.splunk.com/t5/Getting-Data-In/How-come-Splunk-only-can-read-10000-lines-from-my-csv-I-need/m-p/455736#M78825 <P>Do you have 900000 unique Ip in your lookup? If not can you use dedup ip-</P> <PRE><CODE>index=cesa_paloalto sourcetype="pan:traffic" type=TRAFFIC vendor_action=allow | join src_ip [| inputlookup append=t ipsmalware2.csv | dedup ip|eval src_ip=Ip] | stats values(src_ip) </CODE></PRE> Thu, 07 Feb 2019 16:54:41 GMT https://community.splunk.com/t5/Getting-Data-In/How-come-Splunk-only-can-read-10000-lines-from-my-csv-I-need/m-p/455736#M78825 Vijeta 2019-02-07T16:54:41Z Re: How come Splunk only can read 10000 lines from my csv? I need 9000000! https://community.splunk.com/t5/Getting-Data-In/How-come-Splunk-only-can-read-10000-lines-from-my-csv-I-need/m-p/455737#M78826 <P>Hi,</P> <P>Yes, I have 9000000 unique IP.</P> Thu, 07 Feb 2019 16:56:59 GMT https://community.splunk.com/t5/Getting-Data-In/How-come-Splunk-only-can-read-10000-lines-from-my-csv-I-need/m-p/455737#M78826 christianubeda 2019-02-07T16:56:59Z Re: How come Splunk only can read 10000 lines from my csv? I need 9000000! https://community.splunk.com/t5/Getting-Data-In/How-come-Splunk-only-can-read-10000-lines-from-my-csv-I-need/m-p/455738#M78827 <P>How many unique IP your index returns within the timeframe you are searching?</P> Thu, 07 Feb 2019 16:58:27 GMT https://community.splunk.com/t5/Getting-Data-In/How-come-Splunk-only-can-read-10000-lines-from-my-csv-I-need/m-p/455738#M78827 Vijeta 2019-02-07T16:58:27Z Re: How come Splunk only can read 10000 lines from my csv? I need 9000000! https://community.splunk.com/t5/Getting-Data-In/How-come-Splunk-only-can-read-10000-lines-from-my-csv-I-need/m-p/455739#M78828 <P>If that is less than 10K then you can try below</P> <PRE><CODE>|inputlookup append=t ipsmalware2.csv | eval src_ip=Ip|join src_ip type=inner[|search index=cesa_paloalto sourcetype="pan:traffic" type=TRAFFIC vendor_action=allow ] </CODE></PRE> Thu, 07 Feb 2019 17:00:37 GMT https://community.splunk.com/t5/Getting-Data-In/How-come-Splunk-only-can-read-10000-lines-from-my-csv-I-need/m-p/455739#M78828 Vijeta 2019-02-07T17:00:37Z Re: How come Splunk only can read 10000 lines from my csv? I need 9000000! https://community.splunk.com/t5/Getting-Data-In/How-come-Splunk-only-can-read-10000-lines-from-my-csv-I-need/m-p/455740#M78829 <P>Hi Vijeta,</P> <P>I only have matches if the IP from my csv is in the first 10000 lines. Actually I have 9000000 lines in my csv. So It didn`t work.</P> <P>If I do that |inputlookup append=t ipsmalware2.csv I see al files. The problem is when I try to match them...</P> Thu, 07 Feb 2019 17:09:20 GMT https://community.splunk.com/t5/Getting-Data-In/How-come-Splunk-only-can-read-10000-lines-from-my-csv-I-need/m-p/455740#M78829 christianubeda 2019-02-07T17:09:20Z Re: How come Splunk only can read 10000 lines from my csv? I need 9000000! https://community.splunk.com/t5/Getting-Data-In/How-come-Splunk-only-can-read-10000-lines-from-my-csv-I-need/m-p/455741#M78830 <P>That is not coz of matching , that is limitation of a subsearch. The subsearch returns you only 10K results and that is why rest all appear as not matched.</P> Thu, 07 Feb 2019 17:11:56 GMT https://community.splunk.com/t5/Getting-Data-In/How-come-Splunk-only-can-read-10000-lines-from-my-csv-I-need/m-p/455741#M78830 Vijeta 2019-02-07T17:11:56Z Re: How come Splunk only can read 10000 lines from my csv? I need 9000000! https://community.splunk.com/t5/Getting-Data-In/How-come-Splunk-only-can-read-10000-lines-from-my-csv-I-need/m-p/455742#M78831 <P>DO NOT USE JOIN. It has limits. Try this:</P> <PRE><CODE>index=cesa_paloalto sourcetype="pan:traffic" type=TRAFFIC vendor_action=allow | lookup ipsmalware2.csv ip AS src_ip OUTPUT ip AS keepMeIfNonNull | where isnotnull(keepMeIfNonNull) | stats values(src_ip) </CODE></PRE> Thu, 07 Feb 2019 17:40:23 GMT https://community.splunk.com/t5/Getting-Data-In/How-come-Splunk-only-can-read-10000-lines-from-my-csv-I-need/m-p/455742#M78831 woodcock 2019-02-07T17:40:23Z Re: How come Splunk only can read 10000 lines from my csv? I need 9000000! https://community.splunk.com/t5/Getting-Data-In/How-come-Splunk-only-can-read-10000-lines-from-my-csv-I-need/m-p/455743#M78832 <P>It is hard to answer when you do not show us your search. Why would you not share your SPL?</P> Thu, 07 Feb 2019 17:40:52 GMT https://community.splunk.com/t5/Getting-Data-In/How-come-Splunk-only-can-read-10000-lines-from-my-csv-I-need/m-p/455743#M78832 woodcock 2019-02-07T17:40:52Z