topic Re: Timestamp Extraction Issue in Getting Data In

Timestamp Extraction Issue

himynamesdave — Mon, 28 Sep 2020 15:20:39 GMT

My data looks like this:

{ EC_reference="C0000001", Entity_name="Charter 88", Entity_type="Third Party", Regulated_donee_type="", Recd_by="", Reported_under_62:12="", Is_sponsorship="", Donor_name="Joseph Rowntree Reform Trust, The Garden House", Donor_type="Company", Company_reg_num=":357963", Postcode="YO30 6WQ", Type_of_donation="Cash", Nature_Provision="", Purpose="", How_dealt_with="", Value_GBP="50000", Received_date="23-03-2001 00:00:00", Accepted_date="23-03-2001 00:00:00", Reported_date="07-09-2001 00:00:00", Compliance_breach="None" }

In my props.conf I have:

# your settings

NO_BINARY_CHECK=1

SHOULD_LINEMERGE=false

TIME_FORMAT=%d-%m-%Y %H:%M:%S

TIME_PREFIX=Accepted_date=\"

Though Splunk cannot ID a timestamp. I'm not sure what I've done wrong here...

Re: Timestamp Extraction Issue

lukejadamec — Thu, 21 Nov 2013 22:05:34 GMT

The default MAX_TIMESTAMP_LOOKAHEAD is 150 characters. That might be why it's not finding it.

The setting for the log you posted would be 394 characters, but it looks like the size of the log can vary considerably.

Try setting MAX_TIMESTAMP_LOOKAHEAD = 1000

The documentation says this will effect indexing performance because it takes Splunk longer to find the timestamp:

http://docs.splunk.com/Documentation/Splunk/latest/Data/Tunetimestampextractionforbetterindexingperformance

See this post for a similar example:

http://answers.splunk.com/answers/4338/time-stamping-problem-using-time_prefix

Re: Timestamp Extraction Issue

agodoy — Thu, 21 Nov 2013 22:16:00 GMT

Not sure which field has the timestamp that you want to use. I have read of something like this working before when the time format alone would not.

TIME_FORMAT=Received_date="%d-%m-%Y %H:%M:%S"

Re: Timestamp Extraction Issue

himynamesdave — Mon, 28 Sep 2020 15:20:42 GMT

Thanks for the suggestion, but declaring MAX_TIMESTAMP_LOOKAHEAD=x hasn't worker either 😞

# your settings
MAX_TIMESTAMP_LOOKAHEAD=1000
NO_BINARY_CHECK=1
SHOULD_LINEMERGE=false
TIME_FORMAT=%d-%m-%Y %H:%M:%S
TIME_PREFIX=Accepted_date\="

Re: Timestamp Extraction Issue

himynamesdave — Thu, 21 Nov 2013 22:52:12 GMT

I was not aware you could use a text string in the TIME_FORMAT field. Looking at the docs it calls for a .

HOWEVER, when I change the props.conf file to the following, Splunk can identify the %H:%M:%S part of the timestamp :S

# your settings
MAX_TIMESTAMP_LOOKAHEAD=1000
NO_BINARY_CHECK=1
SHOULD_LINEMERGE=false
TIME_FORMAT=Accepted_date="%d-%m-%Y %H:%M:%S"

Re: Timestamp Extraction Issue

lukejadamec — Thu, 21 Nov 2013 23:52:11 GMT

You don't need to escape the = in the TIME_PREFIX

Re: Timestamp Extraction Issue

himynamesdave — Fri, 22 Nov 2013 13:41:13 GMT

Ah, OK. Although Splunk still fails to read the timestamp after removing the \

Re: Timestamp Extraction Issue

davebrooking — Mon, 28 Sep 2020 15:20:53 GMT

I think the issue could have something to do with the age of the timestamp, there is another setting MAX_DAYS_AGO that defaults to 2000 days (approx 5.5 years) your date is 2001 which is over 12 years ago. Try setting MAX_DAYS_AGO to its maximum value 10951 (approx 30 years)

Re: Timestamp Extraction Issue

himynamesdave — Fri, 22 Nov 2013 17:34:34 GMT

BINGO! Thanks for your help.

Re: Timestamp Extraction Issue

lukejadamec — Fri, 22 Nov 2013 23:16:23 GMT

Nice catch