topic Re: Different timestamp values for the same file indexed. in Splunk Enterprise

Different timestamp values for the same file indexed.

peterkn — Wed, 17 Jun 2020 02:11:49 GMT

Hi,

I have a data input (Directory Monitor) for /opt/splunk/data/test

Everyday a new csv file is copy pasted in this directory, and Splunk would start indexing them.

However all rows in this csv file are indexed with different timestamp (_time) values. Eg the file has 3382 events indexed, but doing a

index=caseload host=XXXX source="/opt/splunk/data/test/testfile.csv" | stats count by _time

would yield something like

_time	count
2015-04-17 04:56:49	22
2016-01-08 19:51:49	33
2016-01-18 12:20:09	11
2016-02-07 21:15:09	18

shouldn't it be all current time which is "2020-06-17 10:10:10" for instance, instead of various different timestamps, I'm thinking it is trying to find some value per row that represents a timestamp and parse it, but I don't even see any "2015-04-17" in those 22 rows.

How do I make all the Directory Monitors to index each event using current timestamp?

Re: Different timestamp values for the same file indexed.

rnowitzki — Wed, 17 Jun 2020 07:56:21 GMT

Hi @peterkn ,

Can you share a few lines of the csv?

Splunk tries to get a time for each event / row. It seems that is uses a field that not really is a timestamp or fails to get the correct values from it. Maybe some number that could be interpreted as a unix/epoch timestamp.

You have to tell Splunk where the timestamp is and how to interprete it. Either in the UI or in props.conf.
=> Explained here.

The "current time" / time when indexing is the last option being used.

If there is no timestamp, you could add one to each row with (e.g.) sed. If a script copies the file, it would be an easy enhancement.
Or you configure props.conf as decribed here to really use the current time/index time.

Re: Different timestamp values for the same file indexed.

peterkn — Thu, 18 Jun 2020 06:48:59 GMT

You're an absolute champion.

Due to the nature of the data in the file I'm not legally allowed to share it's content unfortunately.

For those who are interested, instead of modifying/creating props.conf, I changed the timestamp setting from when adding a new file to an index (Settings>Add Data>Upload), select the sourcetype (csv in my case), under Source Type there is a dropdown for Timestamp, my defaulted to "Automatic" so I changed it to "Current", upon clicking "Next" it will ask me to save, select yes, overwrite. This setting will apply to all input monitors, please make sure you restart Splunk.

Thanks again @rnowitzki

Re: Different timestamp values for the same file indexed.

rnowitzki — Thu, 18 Jun 2020 07:22:00 GMT

You're welcome.

And you actually changed props.conf with that, but using the UI instead of CLI/vi 🙂

Happy splunking.