Splunk Enterprise

Different timestamp values for the same file indexed.

peterkn
Explorer

Hi, 

I have a data input (Directory Monitor) for /opt/splunk/data/test

Everyday a new csv file is copy pasted in this directory, and Splunk would start indexing them. 

However all rows in this csv file are indexed with different timestamp (_time) values. Eg the file has 3382 events indexed, but doing a 

 

 

index=caseload host=XXXX source="/opt/splunk/data/test/testfile.csv" | stats count by _time

 

 

would yield something like

_timecount
2015-04-17 04:56:4922
2016-01-08 19:51:4933
2016-01-18 12:20:0911
2016-02-07 21:15:0918

 

shouldn't it be all current time which is "2020-06-17 10:10:10" for instance, instead of various different timestamps, I'm thinking it is trying to find some value per row that represents a timestamp and parse it, but I don't even see any "2015-04-17" in those 22 rows. 

How do I make all the Directory Monitors to index each event using current timestamp? 

0 Karma
1 Solution

rnowitzki
Builder

Hi @peterkn ,

Can you share a few lines of the csv? 

Splunk tries to get a time for each event / row. It seems that is uses a field that not really is a timestamp or fails to get the correct values from it. Maybe some number that could be interpreted as a unix/epoch timestamp.

You have to tell Splunk where the timestamp is  and how to interprete it. Either in the UI or in props.conf.
=> Explained here. 

The "current time" / time when indexing is the last option being used.

If there is no timestamp, you could add one to each row with (e.g.) sed. If a script copies the file, it would be an easy enhancement.
Or you configure props.conf as decribed here to really use the current time/index time.

--
Karma and/or Solution tagging appreciated.

View solution in original post

rnowitzki
Builder

Hi @peterkn ,

Can you share a few lines of the csv? 

Splunk tries to get a time for each event / row. It seems that is uses a field that not really is a timestamp or fails to get the correct values from it. Maybe some number that could be interpreted as a unix/epoch timestamp.

You have to tell Splunk where the timestamp is  and how to interprete it. Either in the UI or in props.conf.
=> Explained here. 

The "current time" / time when indexing is the last option being used.

If there is no timestamp, you could add one to each row with (e.g.) sed. If a script copies the file, it would be an easy enhancement.
Or you configure props.conf as decribed here to really use the current time/index time.

--
Karma and/or Solution tagging appreciated.

peterkn
Explorer

You're an absolute champion. 

Due to the nature of the data in the file I'm not legally allowed to share it's content unfortunately. 

For those who are interested, instead of modifying/creating props.conf, I changed the timestamp setting from when adding a new file to an index (Settings>Add Data>Upload), select the sourcetype (csv in my case), under Source Type there is a dropdown for Timestamp, my defaulted to "Automatic" so I changed it to "Current", upon clicking "Next" it will ask me to save, select yes, overwrite. This setting will apply to all input monitors, please make sure you restart Splunk. 

Thanks again @rnowitzki 

0 Karma

rnowitzki
Builder

You're welcome.

And you actually changed props.conf with that, but using the UI instead of CLI/vi 🙂

Happy splunking.

--
Karma and/or Solution tagging appreciated.
0 Karma
Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...