https://community.splunk.com/t5/All-Apps-and-Add-ons/Splunk-MySQL-Connector-gt-Can-do-this/m-p/68453#M70592 <P>I have a table of ~11 million rows and 53 columns. The table looks like:</P> <P>entity_id | week_1 | week_2 | ..... | week_52<BR /> abcd | 35874 | 587489 |.......| 5478</P> <P>While we discover new entities with splunk the table can grow up to ~30 million rows.</P> <P>I was wondering the following:<BR /> -. What is the maximum number of lookup output fields that MySQL connector can support? <BR /> -. Can I have up to 52 or 365 look up output fields?<BR /> -. what is the recommended maximum number of rows a lookup table can have?<BR /> -. If new entities and weekly data are discovered can Splunk MySQL connector insert the new entities and weekly data in the lookup table that is stored in MySQL?<BR /> -. In case we need to update, delete or insert some weekly data can Splunk MySQL connector perform updates, delete, and insert of this magnitude?<BR /> -. Is there any other approach you might recommend?</P> <P>Thanks,<BR /> Lp</P> Mon, 28 Sep 2020 11:22:11 GMT lpolo 2020-09-28T11:22:11Z https://community.splunk.com/t5/All-Apps-and-Add-ons/Splunk-MySQL-Connector-gt-Can-do-this/m-p/68453#M70592 <P>I have a table of ~11 million rows and 53 columns. The table looks like:</P> <P>entity_id | week_1 | week_2 | ..... | week_52<BR /> abcd | 35874 | 587489 |.......| 5478</P> <P>While we discover new entities with splunk the table can grow up to ~30 million rows.</P> <P>I was wondering the following:<BR /> -. What is the maximum number of lookup output fields that MySQL connector can support? <BR /> -. Can I have up to 52 or 365 look up output fields?<BR /> -. what is the recommended maximum number of rows a lookup table can have?<BR /> -. If new entities and weekly data are discovered can Splunk MySQL connector insert the new entities and weekly data in the lookup table that is stored in MySQL?<BR /> -. In case we need to update, delete or insert some weekly data can Splunk MySQL connector perform updates, delete, and insert of this magnitude?<BR /> -. Is there any other approach you might recommend?</P> <P>Thanks,<BR /> Lp</P> Mon, 28 Sep 2020 11:22:11 GMT https://community.splunk.com/t5/All-Apps-and-Add-ons/Splunk-MySQL-Connector-gt-Can-do-this/m-p/68453#M70592 lpolo 2020-09-28T11:22:11Z https://community.splunk.com/t5/All-Apps-and-Add-ons/Splunk-MySQL-Connector-gt-Can-do-this/m-p/68454#M70593 <UL> <LI>What is the maximum number of lookup output fields that MySQLconnector can support? <BR /><FONT color="red"> There is no inherent maximum number of output fields - if MySQL can support it so can the MySQL connector</FONT> </LI> <LI>Can I have up to 52 or 365 look up output fields? <BR /><FONT color="red">Yes, you should be able to</FONT> </LI> <LI>what is the recommended maximum number of rows a lookup table can have? <BR /><FONT color="red">This depends in a lot of things such as the MySQL server spec, MySQL tuning, schema design etc. With a commodity server and some tuning you should be able to get decent performance with hundreds of millions of rows</FONT> </LI> <LI>If new entities and weekly data are discovered can Splunk MySQL connector insert the new entities and weekly data in the lookup table that is stored in MySQL? <BR /><FONT color="red">Yes, you should be able to - look in mysqloutput command</FONT> </LI> <LI>In case we need to update, delete or insert some weekly data can Splunk MySQL connector perform updates, delete, and insert of this magnitude? <BR /><FONT color="red">Yes, you can use scheduled searches in Splunk to execute any table maintenance, look into mysqlquery command</FONT></LI> <LI>Is there any other approach you might recommend? <BR /><FONT color="red">As far as maintenance of the tables you can also use cron jobs to run maintenance scripts</FONT></LI> </UL> Thu, 09 Feb 2012 17:26:30 GMT https://community.splunk.com/t5/All-Apps-and-Add-ons/Splunk-MySQL-Connector-gt-Can-do-this/m-p/68454#M70593 Ledion_Bitincka 2012-02-09T17:26:30Z