Splunk MySQL Connector -> Can do this?

[lpolo]

I have a table of ~11 million rows and 53 columns. The table looks like:

entity_id | week_1 | week_2 | ..... | week_52
abcd | 35874 | 587489 |.......| 5478

While we discover new entities with splunk the table can grow up to ~30 million rows.

I was wondering the following:
-. What is the maximum number of lookup output fields that MySQL connector can support?
-. Can I have up to 52 or 365 look up output fields?
-. what is the recommended maximum number of rows a lookup table can have?
-. If new entities and weekly data are discovered can Splunk MySQL connector insert the new entities and weekly data in the lookup table that is stored in MySQL?
-. In case we need to update, delete or insert some weekly data can Splunk MySQL connector perform updates, delete, and insert of this magnitude?
-. Is there any other approach you might recommend?

Thanks,
Lp

[Ledion_Bitincka]

• What is the maximum number of lookup output fields that MySQLconnector can support?
  There is no inherent maximum number of output fields - if MySQL can support it so can the MySQL connector
• Can I have up to 52 or 365 look up output fields?
  Yes, you should be able to
• what is the recommended maximum number of rows a lookup table can have?
  This depends in a lot of things such as the MySQL server spec, MySQL tuning, schema design etc. With a commodity server and some tuning you should be able to get decent performance with hundreds of millions of rows
• If new entities and weekly data are discovered can Splunk MySQL connector insert the new entities and weekly data in the lookup table that is stored in MySQL?
  Yes, you should be able to - look in mysqloutput command
• In case we need to update, delete or insert some weekly data can Splunk MySQL connector perform updates, delete, and insert of this magnitude?
  Yes, you can use scheduled searches in Splunk to execute any table maintenance, look into mysqlquery command
• Is there any other approach you might recommend?
  As far as maintenance of the tables you can also use cron jobs to run maintenance scripts