https://community.splunk.com/t5/Splunk-Search/dboutput-with-65-Million-Rows/m-p/199227#M187476 <P>why can't you do at database end? Splunk is not as fast as the database.</P> Wed, 26 Mar 2014 05:20:55 GMT linu1988 2014-03-26T05:20:55Z https://community.splunk.com/t5/Splunk-Search/dboutput-with-65-Million-Rows/m-p/199224#M187473 <P>I need to output 65 Million rows to a database table, I see the default per transaction is 50K. Is there a good way to do this?</P> Wed, 26 Mar 2014 01:40:56 GMT https://community.splunk.com/t5/Splunk-Search/dboutput-with-65-Million-Rows/m-p/199224#M187473 ShaneNewman 2014-03-26T01:40:56Z https://community.splunk.com/t5/Splunk-Search/dboutput-with-65-Million-Rows/m-p/199225#M187474 <P>Just FYI, I tried streaming and it filled up all of the RAM on my server...</P> Wed, 26 Mar 2014 01:53:50 GMT https://community.splunk.com/t5/Splunk-Search/dboutput-with-65-Million-Rows/m-p/199225#M187474 ShaneNewman 2014-03-26T01:53:50Z https://community.splunk.com/t5/Splunk-Search/dboutput-with-65-Million-Rows/m-p/199226#M187475 <P>Streaming has no limit, but if you're not in streaming mode you have a 50k row limit.</P> <P>How much memory? Please open a bug if it's at or over recommended minimum (12gb IIRC)</P> Wed, 26 Mar 2014 03:10:15 GMT https://community.splunk.com/t5/Splunk-Search/dboutput-with-65-Million-Rows/m-p/199226#M187475 jcoates_splunk 2014-03-26T03:10:15Z https://community.splunk.com/t5/Splunk-Search/dboutput-with-65-Million-Rows/m-p/199227#M187476 <P>why can't you do at database end? Splunk is not as fast as the database.</P> Wed, 26 Mar 2014 05:20:55 GMT https://community.splunk.com/t5/Splunk-Search/dboutput-with-65-Million-Rows/m-p/199227#M187476 linu1988 2014-03-26T05:20:55Z https://community.splunk.com/t5/Splunk-Search/dboutput-with-65-Million-Rows/m-p/199228#M187477 <P>I have 24GB allocated to dbx, 48GB overall on the server.</P> Wed, 26 Mar 2014 14:22:20 GMT https://community.splunk.com/t5/Splunk-Search/dboutput-with-65-Million-Rows/m-p/199228#M187477 ShaneNewman 2014-03-26T14:22:20Z https://community.splunk.com/t5/Splunk-Search/dboutput-with-65-Million-Rows/m-p/199229#M187478 <P>The database does not have the data, I am trying to put it in the database.</P> Wed, 26 Mar 2014 14:26:32 GMT https://community.splunk.com/t5/Splunk-Search/dboutput-with-65-Million-Rows/m-p/199229#M187478 ShaneNewman 2014-03-26T14:26:32Z https://community.splunk.com/t5/Splunk-Search/dboutput-with-65-Million-Rows/m-p/199230#M187479 <P>Via splunk create the file into a csv file. (Could split up into 1 million records for better performance.)</P> <P>Then load using native DB commands?</P> <P>DB2 command something like: <BR /> <CODE>db2 import from csvTypeFile of del "insert into table1 (c1, c2, c3,...) "</CODE></P> <P>Oracle use SQL_Loader: <CODE><A href="http://www.orafaq.com/wiki/SQL*Loader_FAQ" target="_blank">http://www.orafaq.com/wiki/SQL*Loader_FAQ</A></CODE></P> Mon, 28 Sep 2020 16:14:46 GMT https://community.splunk.com/t5/Splunk-Search/dboutput-with-65-Million-Rows/m-p/199230#M187479 koshyk 2020-09-28T16:14:46Z