Splunk Search

Avoiding CSV lookup replication errors

Path Finder

I've got a medium-sized (50MB) CSV lookup file with two columns (email address and server name) that I want to use. I tried a straight upload and managed to put down our Splunk instance because replication failed and blocked all searches. Can I dribble the file in 100K lines at a time using outputlookup append=t? Or does the replication just take the whole lookup bundle and try and replicate everything?

Please note: I do not have access to the file system; whatever the solution is, I have to be able to do it from Splunk Web.


0 Karma


The problem isn’t going to be fixed by “dribbling in” the csv one piece at a time.

Depending on the version of splunk you have limits.conf on the search heads and indexers will have a default setting of 800MB or 2GB for search bundle replication (I think it’s 2GB since 6.6).

You’re going over that limit by x MEgabytes when you upload the csv... and causing the issue.

There are several solutions documented for this.

  1. Increase the limits (just know it affects network bandwidth. See search bundle replication settings in limits.conf - you can’t do this via UI as far as I know.
  2. Reduce the size and or number of existing lookups
  3. you can probably do this
  4. Index the data via the web UI and use join, append, etc (using an “OR” condition is better than joins, google how to join without join in splunk)
  5. you can probably do this
0 Karma


Hi There,

That seems a tad bit more than a medium size csv you have there, how many records have you got within it? Have you looked into utilising a KV store instead?

0 Karma
Get Updates on the Splunk Community!

Admin Your Splunk Cloud, Your Way

Join us to maximize different techniques to best tune Splunk Cloud. In this Tech Enablement, you will get ...

Cloud Platform | Discontinuing support for TLS version 1.0 and 1.1

Overview Transport Layer Security (TLS) is a security communications protocol that lets two computers, ...

New Customer Testimonials

Enterprises of all sizes and across different industries are accelerating cloud adoption by migrating ...