Getting Data In
Highlighted

filtering on yesterdays date?

Champion

Hi,

I have a csv file, in a nice format (see below). The data is for rolling 7/10/21 day reports, that customers control, and we want to import into Splunk. Since it's rolling data, a large chunk of this data is duplicate from previous days. All I want to process is "yesterday's data". Is there a way to look at the data, and filter out unnecessary data?

Sample data:

Timestamp=17-May-12 15:45:00,host=APF-US211i-RH-Cpu-0,metric=CPUUtilization,value=31.00000000,DURATION=66
Timestamp=17-May-12 15:46:06,host=APF-US211i-RH-Cpu-0,metric=CPU
Utilization,value=30.00000000,DURATION=3
Timestamp=17-May-12 15:46:09,host=APF-US211i-RH-Cpu-0,metric=CPUUtilization,value=32.00000000,DURATION=32
Timestamp=17-May-12 15:46:41,host=APF-US211i-RH-Cpu-0,metric=CPU
Utilization,value=31.00000000,DURATION=30
Timestamp=17-May-12 15:47:11,host=APF-US211i-RH-Cpu-0,metric=CPUUtilization,value=31.00000000,DURATION=27
Timestamp=17-May-12 15:47:38,host=APF-US211i-RH-Cpu-0,metric=CPU
Utilization,value=31.00000000,DURATION=65
Timestamp=17-May-12 15:48:43,host=APF-US211i-RH-Cpu-0,metric=CPUUtilization,value=29.00000000,DURATION=31
Timestamp=17-May-12 15:49:14,host=APF-US211i-RH-Cpu-0,metric=CPU
Utilization,value=32.00000000,DURATION=29
Timestamp=17-May-12 15:49:43,host=APF-US211i-RH-Cpu-0,metric=CPUUtilization,value=33.00000000,DURATION=31
Timestamp=17-May-12 15:50:14,host=APF-US211i-RH-Cpu-0,metric=CPU
Utilization,value=32.00000000,DURATION=30
Timestamp=17-May-12 15:50:44,host=APF-US211i-RH-Cpu-0,metric=CPUUtilization,value=28.00000000,DURATION=65
Timestamp=17-May-12 15:51:49,host=APF-US211i-RH-Cpu-0,metric=CPU
Utilization,value=27.00000000,DURATION=2
Timestamp=17-May-12 15:51:51,host=APF-US211i-RH-Cpu-0,metric=CPUUtilization,value=29.00000000,DURATION=32
Timestamp=17-May-12 15:52:23,host=APF-US211i-RH-Cpu-0,metric=CPU
Utilization,value=30.00000000,DURATION=30
Timestamp=17-May-12 15:52:53,host=APF-US211i-RH-Cpu-0,metric=CPUUtilization,value=30.00000000,DURATION=27
Timestamp=17-May-12 15:53:20,host=APF-US211i-RH-Cpu-0,metric=CPU
Utilization,value=30.00000000,DURATION=30
Timestamp=17-May-12 15:53:50,host=APF-US211i-RH-Cpu-0,metric=CPUUtilization,value=34.00000000,DURATION=61
Timestamp=17-May-12 15:54:51,host=APF-US211i-RH-Cpu-0,metric=CPU
Utilization,value=33.00000000,DURATION=30
Timestamp=17-May-12 15:55:21,host=APF-US211i-RH-Cpu-0,metric=CPUUtilization,value=31.00000000,DURATION=29
Timestamp=17-May-12 15:55:50,host=APF-US211i-RH-Cpu-0,metric=CPU
Utilization,value=31.00000000,DURATION=31

Tags (2)
Highlighted

Re: filtering on yesterdays date?

Splunk Employee
Splunk Employee

I think you can just do this with the time manipulation.

... earliest = -1d@d latest = -0d@d | ...

http://docs.splunk.com/Documentation/Splunk/latest/User/ChangeTheTimeRangeOfYourSearch

Highlighted

Re: filtering on yesterdays date?

Champion

Not a search, I want to stop duplicate data from getting into the system.

0 Karma
Highlighted

Re: filtering on yesterdays date?

Legend

No, the filters that can be used for filtering events before they go into the index are regex based and work event by event only, so there's no mechanism for looking for any duplicates in the index and filter based on that (which is wise, I imagine that would have a severe impact on performance...)

View solution in original post

Highlighted

Re: filtering on yesterdays date?

Champion

Makes sense. Thanks. I'll try to pre-parse the data.

0 Karma