Getting Data In

can you remove or archive the original CSV file once it is indexed?


i have a large CSV file / lookup table which i'm writing to via outputlookup.

it's approaching 1G in size and i'm wondering how to best prune it.

can i delete the original data file (using cron for instance) and still use the .idx files for lookups?

or is it best to trim it somehow using a parallel splunk search which removes old entries? and if so - any advice / links on how to do that?

0 Karma



If you are exporting something like csv in regular interval (like report) then you can use outputcsv command to export instead of outputlookup.

In other scenario, Assuming that you are comfortable with exported 1G and using output lookup in your search queries without any problem and replication is causing issue means then you can use either blacklist(not to take part of replication) /white list(by default all lookup will take part of replication bundle) to protect your huge lookups take part of replication bundle.

Considering this scenario, if you are using the outputlookup csv to lookup with some events and facing limitation issue then you can try with summary index. it would be easier.

Hope this will help you.


0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...