Getting Data In

can you remove or archive the original CSV file once it is indexed?

awurster
Contributor

i have a large CSV file / lookup table which i'm writing to via outputlookup.

it's approaching 1G in size and i'm wondering how to best prune it.

can i delete the original data file (using cron for instance) and still use the .idx files for lookups?

or is it best to trim it somehow using a parallel splunk search which removes old entries? and if so - any advice / links on how to do that?

0 Karma

vasanthmss
Motivator

Hi,

If you are exporting something like csv in regular interval (like report) then you can use outputcsv command to export instead of outputlookup.

In other scenario, Assuming that you are comfortable with exported 1G and using output lookup in your search queries without any problem and replication is causing issue means then you can use either blacklist(not to take part of replication) /white list(by default all lookup will take part of replication bundle) to protect your huge lookups take part of replication bundle.

Considering this scenario, if you are using the outputlookup csv to lookup with some events and facing limitation issue then you can try with summary index. it would be easier.

Hope this will help you.

Cheers!

V
0 Karma
Get Updates on the Splunk Community!

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars in April. This post ...

Want to Reduce Costs, Mitigate Risk, Improve Performance, or Increase Efficiencies? ...

Splunk Lantern is Splunk’s customer success center that provides advice from Splunk experts on valuable data ...