Getting Data In

How do you Import data from .txt files from folders within a folder?

Communicator

Hi all,

Ok, so I have a folder that contains other folders, that in turn contain a folder, which, bare with me here, in turn contains txt files.

All clear for now?

The problem: I want to import the data from those .txt files into Splunk, so I can search various inputs from those .txt files. For some reason, Splunk indexes data from those .txt files but with a wrong time stamp. Others are indexed with modified file time stamp. Others get indexed with I don't know what time stamp.

Quick side note: I already managed to import data, so I can look for what I need.

Question/Issue: how can I tell Splunk to look for the time modified from the .txt file?

Thank you,

Bogdan

0 Karma

Communicator

Hi all,

It seems that because in some .txt files i have added date, and in some i didn't, those without date are getting "calculated" date.

I have to test my theory before i can conclude this.

Thank you all.

Bogdan.

0 Karma

Builder

Here's how Splunk assigns timestamps to events (https://docs.splunk.com/Documentation/Splunk/7.2.4/Data/HowSplunkextractstimestamps)

Splunk software uses the following precedence rules to assign timestamps to events:

  1. It looks for a time or date in the event itself using an explicit TIME_FORMAT, if provided. You configure the TIME_FORMAT attribute in props.conf.

  2. If no TIME_FORMAT was configured for the data, Splunk software attempts to automatically identify a time or date in the event itself. It uses the source type of the event (which includes TIME_FORMAT information) to try to find the timestamp.

  3. If an event has a time and date, but not a year, Splunk software determines the year, as described in How Splunk software determines timestamps with no year, and builds the timestamp from that.

  4. If no events in a source have a date, Splunk software tries to find a date in the source name or file name. Time of day is not identified in filenames. (This requires that the events have a time, even though they don't have a date.)

  5. For file sources, if no date can be identified in the file name, Splunk software uses the file modification time.

  6. As a last resort, Splunk software sets the timestamp to the current system time when indexing each event.

It sounds like you need to configure a props.conf for the sourcetypes to extract or assign the timestamp:

https://docs.splunk.com/Documentation/Splunk/7.2.5/admin/Propsconf#Timestamp_extraction_configuratio...

Communicator

Hi solarboyz1,

If you are referring about props.conf from C:\Program Files\Splunk\etc\system\local, i don't have any file there.

Other than that, how can i force Splunk to look for moddified field from properties?

Thank you,

Bogdan.

0 Karma

Builder

The only way to force splunk to do something, is to configure it that way.

You will need to create a props.conf that defines how to extract the timestamps from the events.

If you have a multi-server implementation, I recommend creating an app which is just a folder structure:

C:\Program Files\Splunk\etc\apps\MyApp\

In that app, create your props.conf:
C:\Program Files\Splunk\etc\apps\MyApp\local\props.conf

In that props.conf you will then need to define how you want Splunk to extract the timestamp:

[your_sourcetype]
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%dT%H:%M:%S%z
MAX_TIMESTAMP_LOOKAHEAD = 21
SHOULD_LINEMERGE = false

I recommend pushing that app to the forwarder monitoring the file, and your indexers.

0 Karma

SplunkTrust
SplunkTrust

What are the props.conf settings for that sourcetype? Are the files consistent in how timestamps are placed and formatted?

---
If this reply helps you, an upvote would be appreciated.
0 Karma

Communicator

Hi richgalloway,

I think i didn't stressed enough the idea:

Splunk get some time stamps from .txt file's Modified field from properties, other time stamps are get from i don't know where, because that time stamp is out of range when that file was created and written in.

Hope this clarifies a bit my dilema.

Thank you,

Bogdan.

0 Karma

Communicator

Hi richgalloway,

The problem is that i don't have props.conf if you are saying about: C:\Program Files\Splunk\etc\system\local.

is there any location i can look for props.conf?

Thank you.

Bogdan

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!