Getting Data In

How to efficiently resend data into Splunk with a REST API that will replace old data if it has been updated



I am looking to resend data to Splunk in the most efficient way. I want to resend data into Splunk with a REST API that will replace old data if it has been updated. I don't want to resend all of the data, only anything that has changed.

The goal is to not give Splunk more data than it needs.
I am searching the data based on a by-minute time range so even in the course of 5-10 minutes, resending all of that would be a lot of data if most of it is repeating events.

I'm very new to all of this so I was looking for some guidance on where to start or helpful links to get started.

0 Karma

Esteemed Legend

You can only do that if you store the data in a Lookup File in Splunk. If you do this, you would update it like this:

Some search to pull in new data here (could be dbxquery or something else)
| some SPL to transform the data and ensure that a distinct key field such as "host" exists
| inputlookup append=true YourLookupFIleHere.csv
| dedup host
| outputlookup YourLookupFIleHere.csv
0 Karma


Hi @kdanielsobrien,
maybe my words could seem strange: are you sure that you need Splunk?
Splunk works in a different way than a database:

  • ingested data aren't modifiable (also by REST API),
  • you have always all the data (in the retention period) and not only the data you want, you have all the data you ingested and you cannot delete them,
  • if you use the delete command, data continue to stand in the buckets (in the retention period) but aren't searchable;
  • for this reason Splunk is commonly used for compliance better than a DB!

Anyway, the first question is: way you want to have this approach? to save storage or what else?

Anyway, if you want to do this, you could create a summary index populating it every day with all the correct data you want ( ).


0 Karma


Hi Giuseppe,

I guess I want to filter what data is being sent to Splunk.. For example, I send all the data to Splunk for a 10 minute time span. After I have sent the data to Splunk, a few minutes of data have been replaced with updated new values/data. I only want to resend the new/updated data to Splunk for the few minutes that have been changed.

I want to filter what data is being sent to Splunk because I will waste a lot of GB of data if I resend all of the data from a time span, just to update a few events in Splunk search.

0 Karma


That isn’t how Splunk works. It’s not a database so has no feature like that.

0 Karma
Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...