Getting Data In

How do I only see the latest data from a dynamically changing CSV file?

ragedsparrow
SplunkTrust
SplunkTrust

I have a customer's CSV file that is dynamically updated. By that' what i mean is that the last row is sometimes deleted and readded with updated information:

DATE,HOUR,WORKFILE_COUNT,RATE_PER_WORKFILE,EST_NBR_WORKFILES_PER_HOUR,WORK_HOLD_FILE_COUNT,WORK_DONE_FILE_COUNT,UNPROCESSED_FILE_COUNT,BAD_FILE_COUNT
 09/13/15,15,1172,247.238,14560.9,0,0,0,13
 09/13/15,16,1190,247.011,14574.3,0,0,0,13
 09/13/15,17,1217,227.313,15837.2,0,0,0,13
 09/13/15,18,782,231.839,15528,0,0,0,13
 09/13/15,19,648,240.61,14962,0,0,0,13
 09/13/15,20,629,238.669,15083.6,0,0,0,13
 09/13/15,21,394,242.297,14857.8,0,0,0,13
 09/13/15,22,493,295.181,12195.9,0,0,0,13
 09/13/15,23,257,280.949,12813.7,0,0,0,13

May change to:

DATE,HOUR,WORKFILE_COUNT,RATE_PER_WORKFILE,EST_NBR_WORKFILES_PER_HOUR,WORK_HOLD_FILE_COUNT,WORK_DONE_FILE_COUNT,UNPROCESSED_FILE_COUNT,BAD_FILE_COUNT
 09/13/15,15,1172,247.238,14560.9,0,0,0,13
 09/13/15,16,1190,247.011,14574.3,0,0,0,13
 09/13/15,17,1217,227.313,15837.2,0,0,0,13
 09/13/15,18,782,231.839,15528,0,0,0,13
 09/13/15,19,648,240.61,14962,0,0,0,13
 09/13/15,20,629,238.669,15083.6,0,0,0,13
 09/13/15,21,394,242.297,14857.8,0,0,0,13
 09/13/15,22,493,295.181,12195.9,0,0,0,13
 09/13/15,23,257,291.949,12886.7,0,0,0,13

Splunk, is forwarding both of the lines

09/13/15,23,257,280.949,12813.7,0,0,0,13
09/13/15,23,257,291.949,12886.7,0,0,0,13

My timestamps key off the date and hour in the 1st and 2nd columns, and I only need the latest data for the timestamp, but I'm getting multiple events with the same timestamp due to the updating rows. Any suggestions?

Tags (1)
0 Karma
1 Solution

woodcock
Esteemed Legend

You can use:

... | sort - _indextime | dedup DATE HOUR WORKFILE_COUNT

View solution in original post

woodcock
Esteemed Legend

You can use:

... | sort - _indextime | dedup DATE HOUR WORKFILE_COUNT

ragedsparrow
SplunkTrust
SplunkTrust

Ha! I should have known that. Thank you for your answer. It's perfect. I appreciate it.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...