Getting Data In

Extracting Timestamp from a txt file

TimothyPeh
Engager

Hi all,

I know that there are several post on this question before, but I can't seem to figure out the correct answer to my question.

My data comes in a form of text file, where each line is a record. There are no delimiters, hence the only way to tell is by positions.

D HE12*201303250000**0100GTB27000170000100014000000000000045056000450560003300012 0003300012 000000000100000 00000 000000000000000 00000 00045056000450560003300012 0003300012

The portion in bold is the date-time stand which I'm trying to extract.
So it translate to:
Year:2013
Month: 03
Day: 25
Time: 0000 (Midnight)

This is my current props.conf

[CPOF]
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = FALSE
PULLDOWN_TYPE = 1
TIME_PREFIX = ???
TIME_FORMAT = %Y%m%d%h%m

Not sure what to fill in for the time_prefix and also if my time_format is correct?

Thanks!

Tags (2)
0 Karma
1 Solution

BobM
Builder

Hi Timothy

You have a mistake in your time format %m is the month %M is the minute. You have entered month twice. Also with no delimiters in your data splunk could take the date from the wrong place so it would be a good practice to force it's location with TIME_PREFIX and MAX_TIMESTAMP_LOOKAHEAD. The time prefix should be a PCRE regular expression to match what comes before the timestamp.

I would try the following

[CPOF]
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = FALSE
PULLDOWN_TYPE = 1
TIME_PREFIX = \w \w\w\d\d
TIME_FORMAT = %Y%m%d%h%M
MAX_TIMESTAMP_LOOKAHEAD = 12

You may have to play with the time prefix if your data changes, you might try "^.{6}" to say six digits in from the beginning of the line.

Bob

View solution in original post

BobM
Builder

Hi Timothy

You have a mistake in your time format %m is the month %M is the minute. You have entered month twice. Also with no delimiters in your data splunk could take the date from the wrong place so it would be a good practice to force it's location with TIME_PREFIX and MAX_TIMESTAMP_LOOKAHEAD. The time prefix should be a PCRE regular expression to match what comes before the timestamp.

I would try the following

[CPOF]
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = FALSE
PULLDOWN_TYPE = 1
TIME_PREFIX = \w \w\w\d\d
TIME_FORMAT = %Y%m%d%h%M
MAX_TIMESTAMP_LOOKAHEAD = 12

You may have to play with the time prefix if your data changes, you might try "^.{6}" to say six digits in from the beginning of the line.

Bob

BobM
Builder

No! I'm afraid this is not retroactive. All new data will use this configuration but old data will not be updated. The only way round it is to force splunk to delete and re-index the data. 😞

TimothyPeh
Engager

Hi bob,

Thanks! However, I have another problem. This solution works when I use it with data preview.

When I tried to edit the props.conf file for my existing data that were indexed already, nothing seems to have changed at all.

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In September, the Splunk Threat Research Team had two releases of new security content via the Enterprise ...

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...