I have a customer's CSV file that is dynamically updated. By that' what i mean is that the last row is sometimes deleted and readded with updated information:
DATE,HOUR,WORKFILE_COUNT,RATE_PER_WORKFILE,EST_NBR_WORKFILES_PER_HOUR,WORK_HOLD_FILE_COUNT,WORK_DONE_FILE_COUNT,UNPROCESSED_FILE_COUNT,BAD_FILE_COUNT
09/13/15,15,1172,247.238,14560.9,0,0,0,13
09/13/15,16,1190,247.011,14574.3,0,0,0,13
09/13/15,17,1217,227.313,15837.2,0,0,0,13
09/13/15,18,782,231.839,15528,0,0,0,13
09/13/15,19,648,240.61,14962,0,0,0,13
09/13/15,20,629,238.669,15083.6,0,0,0,13
09/13/15,21,394,242.297,14857.8,0,0,0,13
09/13/15,22,493,295.181,12195.9,0,0,0,13
09/13/15,23,257,280.949,12813.7,0,0,0,13
May change to:
DATE,HOUR,WORKFILE_COUNT,RATE_PER_WORKFILE,EST_NBR_WORKFILES_PER_HOUR,WORK_HOLD_FILE_COUNT,WORK_DONE_FILE_COUNT,UNPROCESSED_FILE_COUNT,BAD_FILE_COUNT
09/13/15,15,1172,247.238,14560.9,0,0,0,13
09/13/15,16,1190,247.011,14574.3,0,0,0,13
09/13/15,17,1217,227.313,15837.2,0,0,0,13
09/13/15,18,782,231.839,15528,0,0,0,13
09/13/15,19,648,240.61,14962,0,0,0,13
09/13/15,20,629,238.669,15083.6,0,0,0,13
09/13/15,21,394,242.297,14857.8,0,0,0,13
09/13/15,22,493,295.181,12195.9,0,0,0,13
09/13/15,23,257,291.949,12886.7,0,0,0,13
Splunk, is forwarding both of the lines
09/13/15,23,257,280.949,12813.7,0,0,0,13
09/13/15,23,257,291.949,12886.7,0,0,0,13
My timestamps key off the date and hour in the 1st and 2nd columns, and I only need the latest data for the timestamp, but I'm getting multiple events with the same timestamp due to the updating rows. Any suggestions?
You can use:
... | sort - _indextime | dedup DATE HOUR WORKFILE_COUNT
You can use:
... | sort - _indextime | dedup DATE HOUR WORKFILE_COUNT
Ha! I should have known that. Thank you for your answer. It's perfect. I appreciate it.