Ok, so I have a folder that contains other folders, that in turn contain a folder, which, bare with me here, in turn contains txt files.
All clear for now?
The problem: I want to import the data from those .txt files into Splunk, so I can search various inputs from those .txt files. For some reason, Splunk indexes data from those .txt files but with a wrong time stamp. Others are indexed with modified file time stamp. Others get indexed with I don't know what time stamp.
Quick side note: I already managed to import data, so I can look for what I need.
Question/Issue: how can I tell Splunk to look for the time modified from the .txt file?
It seems that because in some .txt files i have added date, and in some i didn't, those without date are getting "calculated" date.
I have to test my theory before i can conclude this.
Thank you all.
Here's how Splunk assigns timestamps to events (https://docs.splunk.com/Documentation/Splunk/7.2.4/Data/HowSplunkextractstimestamps)
Splunk software uses the following precedence rules to assign timestamps to events:
It looks for a time or date in the event itself using an explicit TIME_FORMAT, if provided. You configure the TIME_FORMAT attribute in props.conf.
If no TIME_FORMAT was configured for the data, Splunk software attempts to automatically identify a time or date in the event itself. It uses the source type of the event (which includes TIME_FORMAT information) to try to find the timestamp.
If an event has a time and date, but not a year, Splunk software determines the year, as described in How Splunk software determines timestamps with no year, and builds the timestamp from that.
If no events in a source have a date, Splunk software tries to find a date in the source name or file name. Time of day is not identified in filenames. (This requires that the events have a time, even though they don't have a date.)
For file sources, if no date can be identified in the file name, Splunk software uses the file modification time.
As a last resort, Splunk software sets the timestamp to the current system time when indexing each event.
It sounds like you need to configure a props.conf for the sourcetypes to extract or assign the timestamp:
If you are referring about props.conf from C:\Program Files\Splunk\etc\system\local, i don't have any file there.
Other than that, how can i force Splunk to look for moddified field from properties?
The only way to force splunk to do something, is to configure it that way.
You will need to create a props.conf that defines how to extract the timestamps from the events.
If you have a multi-server implementation, I recommend creating an app which is just a folder structure:
In that app, create your props.conf:
In that props.conf you will then need to define how you want Splunk to extract the timestamp:
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%dT%H:%M:%S%z
MAX_TIMESTAMP_LOOKAHEAD = 21
SHOULD_LINEMERGE = false
I recommend pushing that app to the forwarder monitoring the file, and your indexers.
What are the props.conf settings for that sourcetype? Are the files consistent in how timestamps are placed and formatted?
I think i didn't stressed enough the idea:
Splunk get some time stamps from .txt file's Modified field from properties, other time stamps are get from i don't know where, because that time stamp is out of range when that file was created and written in.
Hope this clarifies a bit my dilema.
The problem is that i don't have props.conf if you are saying about: C:\Program Files\Splunk\etc\system\local.
is there any location i can look for props.conf?