Getting Data In

can you remove or archive the original CSV file once it is indexed?

awurster
Contributor

i have a large CSV file / lookup table which i'm writing to via outputlookup.

it's approaching 1G in size and i'm wondering how to best prune it.

can i delete the original data file (using cron for instance) and still use the .idx files for lookups?

or is it best to trim it somehow using a parallel splunk search which removes old entries? and if so - any advice / links on how to do that?

0 Karma

vasanthmss
Motivator

Hi,

If you are exporting something like csv in regular interval (like report) then you can use outputcsv command to export instead of outputlookup.

In other scenario, Assuming that you are comfortable with exported 1G and using output lookup in your search queries without any problem and replication is causing issue means then you can use either blacklist(not to take part of replication) /white list(by default all lookup will take part of replication bundle) to protect your huge lookups take part of replication bundle.

Considering this scenario, if you are using the outputlookup csv to lookup with some events and facing limitation issue then you can try with summary index. it would be easier.

Hope this will help you.

Cheers!

V
0 Karma
Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...