How to edit props.conf to use the "last modified" time of the file as the timestamp for each event?

[aholzer]

I have configured monitoring for a set of files. I have configured the props.conf to use the 'last modified' time of the file as the timestamp for each event. However, the events are being indexed for 04/10/2014 for some reason.

Sample files on the server with their last modified times. These are the files I'm monitoring; what I'm indexing is the events in these files:

-rw-r--r--. 1 datasvcs datasvcs  14239 Jan 27 11:00 filename20170127_10.gz
-rw-r--r--. 1 datasvcs datasvcs  14497 Jan 27 12:00 filename20170127_11.gz
-rw-r--r--. 1 datasvcs datasvcs  14143 Jan 27 13:00 filename20170127_12.gz

Indexed data for said files. Notice the _time from below ("2014-04-10") vs the last modified time from above ("Jan 27" no year, which implies current year):

source                                              _time       count
/data/datasvcs/directory_path/filename20170127_10.gz    2014-04-10  465
/data/datasvcs/directory_path/filename20170127_11.gz    2014-04-10  473
/data/datasvcs/directory_path/filename20170127_12.gz    2014-04-10  453

Props.conf configuration. I thought that using "DATETIME_CONFIG = NONE" forced the indexers to use the "last modified" time of the source file rather than trying to parse out a timestamp from the events:

/opt/splunk/etc/slave-apps/<app_name>/local/props.conf [sourcetype_name]
/opt/splunk/etc/slave-apps/<app_name>/local/props.conf DATETIME_CONFIG = NONE
/opt/splunk/etc/slave-apps/<app_name>/local/props.conf FIELD_NAMES = <list of fields>
/opt/splunk/etc/slave-apps/<app_name>/local/props.conf INDEXED_EXTRACTIONS = csv
/opt/splunk/etc/slave-apps/<app_name>/local/props.conf KV_MODE = none
/opt/splunk/etc/slave-apps/<app_name>/local/props.conf NO_BINARY_CHECK = true
/opt/splunk/etc/slave-apps/<app_name>/local/props.conf SHOULD_LINEMERGE = false
/opt/splunk/etc/slave-apps/<app_name>/local/props.conf TIMESTAMP_FIELDS = nanos
/opt/splunk/etc/slave-apps/<app_name>/local/props.conf TIME_FORMAT = %s%9N
/opt/splunk/etc/slave-apps/<app_name>/local/props.conf category = Custom
/opt/splunk/etc/slave-apps/<app_name>/local/props.conf description = CSV for Oceans Order Accepted (SeqOrderAcceptedMessage)
/opt/splunk/etc/slave-apps/<app_name>/local/props.conf disabled = false
/opt/splunk/etc/slave-apps/<app_name>/local/props.conf pulldown_type = true
/opt/splunk/etc/system/default/props.conf              ANNOTATE_PUNCT = True
/opt/splunk/etc/system/default/props.conf              AUTO_KV_JSON = true
/opt/splunk/etc/system/default/props.conf              BREAK_ONLY_BEFORE =
/opt/splunk/etc/system/default/props.conf              BREAK_ONLY_BEFORE_DATE = True
/opt/splunk/etc/system/default/props.conf              CHARSET = UTF-8
/opt/splunk/etc/system/default/props.conf              HEADER_MODE =
/opt/splunk/etc/system/default/props.conf              LEARN_MODEL = true
/opt/splunk/etc/system/default/props.conf              LEARN_SOURCETYPE = true
/opt/splunk/etc/system/default/props.conf              LINE_BREAKER_LOOKBEHIND = 100
/opt/splunk/etc/system/default/props.conf              MAX_DAYS_AGO = 2000
/opt/splunk/etc/system/default/props.conf              MAX_DAYS_HENCE = 2
/opt/splunk/etc/system/default/props.conf              MAX_DIFF_SECS_AGO = 3600
/opt/splunk/etc/system/default/props.conf              MAX_DIFF_SECS_HENCE = 604800
/opt/splunk/etc/system/default/props.conf              MAX_EVENTS = 256
/opt/splunk/etc/system/default/props.conf              MAX_TIMESTAMP_LOOKAHEAD = 128
/opt/splunk/etc/system/default/props.conf              MUST_BREAK_AFTER =
/opt/splunk/etc/system/default/props.conf              MUST_NOT_BREAK_AFTER =
/opt/splunk/etc/system/default/props.conf              MUST_NOT_BREAK_BEFORE =
/opt/splunk/etc/system/default/props.conf              SEGMENTATION = indexing
/opt/splunk/etc/system/default/props.conf              SEGMENTATION-all = full
/opt/splunk/etc/system/default/props.conf              SEGMENTATION-inner = inner
/opt/splunk/etc/system/default/props.conf              SEGMENTATION-outer = outer
/opt/splunk/etc/system/default/props.conf              SEGMENTATION-raw = none
/opt/splunk/etc/system/default/props.conf              SEGMENTATION-standard = standard
/opt/splunk/etc/system/default/props.conf              TRANSFORMS =
/opt/splunk/etc/system/default/props.conf              TRUNCATE = 10000
/opt/splunk/etc/system/default/props.conf              detect_trailing_nulls = false
/opt/splunk/etc/system/default/props.conf              maxDist = 100
/opt/splunk/etc/system/default/props.conf              priority =
/opt/splunk/etc/system/default/props.conf              sourcetype =

Any assistance would be appreciated.

[aholzer · Accepted solution]

Turns out this has been answered before. The solution is to put the props.conf for the sourcetypes in question on the universal forwarder. See the answer below:

https://answers.splunk.com/answers/455406/why-am-i-getting-dateparserverbose-warnings-althou.html

[adayton20]

Thanks for sharing! Learned something new.

[adayton20]

I think I see what you’re asking now. Apologies if I misunderstand again. Just to clarify, you want to use the date_modified in the event as the timestamp of the event (_time)?

I didn’t realize this at first, but there is no year in the timestamp of your events, so Splunk is attempting to extract something that isn’t there. You'd have to specify how far into the event you want Splunk to look in order for Splunk to use the current year appended to the month and day.

I took your events and ingested them in my own Splunk instance to test this. I had to keep changing the times in the logs so I didn't have to clear the fish bucket, so the dates and times are different, but the format is the same. It took me a few tries but I ended up getting it working.

I used the following stanza in my props.conf

[testtime]
TIME_PREFIX = datasvcs\s+\d+\s+ 
TIME_FORMAT = %b %d %H:%M
MAX_TIMESTAMP_LOOKAHEAD = 65

Raw Logs with timestamps:

After tabling the _time with the raw events:

Hope that helps!

[aholzer]

@adayton20 - thanks again for the response, but I believe you've missed the mark again.

The first sample that I provided in my question is a list of files that I'm monitoring, not the events that I will be indexing. Each of those files listed in the sample will have their own events which is what I want indexed. The problem is that I want all the events inside the files to be indexed with the "last modified" time of the file I'm monitoring, rather than Splunk try to guess from the event, because the events don't have timestamps.

I'll update my question to try to make this clearer.

[adayton20]

Yikes! 0 for 2.

Alright, lets try this.

Is there any way you can provide me with a sample of the data? Or is this secret squirrel stuff? What filetype(s) reside inside? Is the data inside the files always formatted the same way or does it vary? Are you creating these gzipped files manually or are they being exported in an automated fashion from another source? Do these logs reside on a Linux box?

Surprisingly, I've had to do something like this (kind of ) for zip files exported from an antivirus. I wrote a PowerShell script to unzip the files, convert them to txt, append a timestamp, and feed it to Splunk. In your case, it may be a little more work, but I think it can be done. Need more infos though.

[adayton20]

Have you tried using TIME_PREFIX?

http://docs.splunk.com/Documentation/Splunk/6.5.1/Data/Configuretimestamprecognition

Try something like TIME_PREFIX = datasvcs\s+\d+\s

[aholzer]

Thanks for the response, but TIME_PREFIX works with a regular expression to look for the timestamp inside the event itself. My problem has to do with applying the "last modified" time of the file itself as the _time for each of the events within the file.