Getting Data In

Configurations that Filters Existing Orders from AS/400 File

_gkollias
Builder

I'm looking to come up with some configurations that filter out existing orders from files I (currently) manually copy to a local directory where Splunk then picks them up and indexes the order info for that hour.

Each file that comes out from the AS/400 has a total of all orders with various information on that order(CustomerNumber, PONumber, Date, Time, etc.) up to that particular hour.

Basically every time Splunk Picks up one of those files, I want it so that Splunk only indexes the NEW orders, rather than indexing the same order data from the previous hours. Otherwise, there will be a large amount of duplicate data being indexed.

Is there a way I can do this? Let me know if any information is needed to dig in to this further.

Thanks in advance!

Tags (2)
0 Karma

lguinn2
Legend

This might be possible if you

(1) Keep the same file name. In other words, overwrite the old file with the new file each hour.

(2) Make sure that the beginning of the file (up to the point of the new data) has not changed.

But if Splunk figures out that this is a different file, it will index it from the beginning, causing the duplication that you are trying to avoid.

There is no way for Splunk to compare inbound data with existing data before indexing. However, it is possible to "dedup" data being retreived during a search - although you have to do it explicitly with the uniq command.

I would test it out.

_gkollias
Builder

Awesome - thanks for your suggestions. I will some testing around it a bit more and come back with some feedback

0 Karma
Get Updates on the Splunk Community!

2024 Splunk Career Impact Survey | Earn a $20 gift card for participating!

Hear ye, hear ye! The time has come again for Splunk's annual Career Impact Survey!  We need your help by ...

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...