topic Re: How do you Import data from .txt files from folders within a folder? in Getting Data In

How do you Import data from .txt files from folders within a folder?

bogdan_nicolesc — Wed, 27 Mar 2019 19:48:21 GMT

Hi all,

Ok, so I have a folder that contains other folders, that in turn contain a folder, which, bare with me here, in turn contains txt files.

All clear for now?

The problem: I want to import the data from those .txt files into Splunk, so I can search various inputs from those .txt files. For some reason, Splunk indexes data from those .txt files but with a wrong time stamp. Others are indexed with modified file time stamp. Others get indexed with I don't know what time stamp.

Quick side note: I already managed to import data, so I can look for what I need.

Question/Issue: how can I tell Splunk to look for the time modified from the .txt file?

Thank you,

Bogdan

Re: How do you Import data from .txt files from folders within a folder?

richgalloway — Wed, 27 Mar 2019 20:02:05 GMT

What are the props.conf settings for that sourcetype? Are the files consistent in how timestamps are placed and formatted?

Re: How do you Import data from .txt files from folders within a folder?

solarboyz1 — Tue, 29 Sep 2020 23:53:33 GMT

Here's how Splunk assigns timestamps to events (https://docs.splunk.com/Documentation/Splunk/7.2.4/Data/HowSplunkextractstimestamps)

Splunk software uses the following precedence rules to assign timestamps to events:

It looks for a time or date in the event itself using an explicit TIME_FORMAT, if provided. You configure the TIME_FORMAT attribute in props.conf.
If no TIME_FORMAT was configured for the data, Splunk software attempts to automatically identify a time or date in the event itself. It uses the source type of the event (which includes TIME_FORMAT information) to try to find the timestamp.
If an event has a time and date, but not a year, Splunk software determines the year, as described in How Splunk software determines timestamps with no year, and builds the timestamp from that.
If no events in a source have a date, Splunk software tries to find a date in the source name or file name. Time of day is not identified in filenames. (This requires that the events have a time, even though they don't have a date.)
For file sources, if no date can be identified in the file name, Splunk software uses the file modification time.
As a last resort, Splunk software sets the timestamp to the current system time when indexing each event.

It sounds like you need to configure a props.conf for the sourcetypes to extract or assign the timestamp:

https://docs.splunk.com/Documentation/Splunk/7.2.5/admin/Propsconf#Timestamp_extraction_configuration

Re: How do you Import data from .txt files from folders within a folder?

bogdan_nicolesc — Thu, 28 Mar 2019 09:07:57 GMT

Hi richgalloway,

I think i didn't stressed enough the idea:

Splunk get some time stamps from .txt file's Modified field from properties, other time stamps are get from i don't know where, because that time stamp is out of range when that file was created and written in.

Hope this clarifies a bit my dilema.

Thank you,

Bogdan.

Re: How do you Import data from .txt files from folders within a folder?

bogdan_nicolesc — Thu, 28 Mar 2019 11:40:50 GMT

Hi all,

It seems that because in some .txt files i have added date, and in some i didn't, those without date are getting "calculated" date.

I have to test my theory before i can conclude this.

Thank you all.

Bogdan.

Re: How do you Import data from .txt files from folders within a folder?

bogdan_nicolesc — Fri, 29 Mar 2019 12:29:52 GMT

Hi richgalloway,

The problem is that i don't have props.conf if you are saying about: C:\Program Files\Splunk\etc\system\local.

is there any location i can look for props.conf?

Thank you.

Bogdan

Re: How do you Import data from .txt files from folders within a folder?

bogdan_nicolesc — Fri, 29 Mar 2019 12:32:23 GMT

Hi solarboyz1,

If you are referring about props.conf from C:\Program Files\Splunk\etc\system\local, i don't have any file there.

Other than that, how can i force Splunk to look for moddified field from properties?

Thank you,

Bogdan.

Re: How do you Import data from .txt files from folders within a folder?

solarboyz1 — Tue, 29 Sep 2020 23:50:20 GMT

The only way to force splunk to do something, is to configure it that way.

You will need to create a props.conf that defines how to extract the timestamps from the events.

If you have a multi-server implementation, I recommend creating an app which is just a folder structure:

C:\Program Files\Splunk\etc\apps\MyApp\

In that app, create your props.conf:
C:\Program Files\Splunk\etc\apps\MyApp\local\props.conf

In that props.conf you will then need to define how you want Splunk to extract the timestamp:

[your_sourcetype]
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%dT%H:%M:%S%z
MAX_TIMESTAMP_LOOKAHEAD = 21
SHOULD_LINEMERGE = false

I recommend pushing that app to the forwarder monitoring the file, and your indexers.