topic Re: Upsert Data on Ingest in Getting Data In

Upsert Data on Ingest

lampert_marksu — Wed, 19 Feb 2014 15:04:12 GMT

I have an index set up and functioning properly on initial ingest. What I'd like to accomplish is the "upsert" of events on future files.

Example

File 1 (childCount_08-13-2013.txt)

Date, Name, # of Kids

08/13/2013, Nicky Blank, 4

08/13/2013, Mike Dorn, 2

08/13/2013, Les Paul, 1

File 2 (childCount_08-23-2013.txt)

Date, Name, # of Kids

08/23/2013, Nicky Blank, 6

08/23/2013, Phillip Jacks, 3

08/23/2013, Tina Walls, 1

DESIRED OUTPUT (Contains all unique entries and updates any existing entries)

08/23/2013, Nicky Blank, 6

08/13/2013, Mike Dorn, 2

08/13/2013, Les Paul, 1

08/23/2013, Phillip Jacks, 3

08/23/2013, Tina Walls, 1

Any assistance would be greatly appreciated.

Re: Upsert Data on Ingest

Ayn — Wed, 19 Feb 2014 16:01:15 GMT

Data in Splunk's index cannot be modified - once it's indexed the data stays the same. What you could do though is create tables like the one you show using stats for instance.

<basesearch> | stats latest(Date) as Date,latest(num_kids) by Name

Re: Upsert Data on Ingest

lampert_marksu — Wed, 19 Feb 2014 16:10:52 GMT

Ah, that makes sense. Thanks for the feedback; this is helpful.