Getting Data In

Timestamp Extraction Issue

himynamesdave
Contributor

My data looks like this:

{ EC_reference="C0000001", Entity_name="Charter 88", Entity_type="Third Party", Regulated_donee_type="", Recd_by="", Reported_under_62:12="", Is_sponsorship="", Donor_name="Joseph Rowntree Reform Trust, The Garden House", Donor_type="Company", Company_reg_num=":357963", Postcode="YO30 6WQ", Type_of_donation="Cash", Nature_Provision="", Purpose="", How_dealt_with="", Value_GBP="50000", Received_date="23-03-2001 00:00:00", Accepted_date="23-03-2001 00:00:00", Reported_date="07-09-2001 00:00:00", Compliance_breach="None" }

In my props.conf I have:

# your settings
NO_BINARY_CHECK=1
SHOULD_LINEMERGE=false
TIME_FORMAT=%d-%m-%Y %H:%M:%S
TIME_PREFIX=Accepted_date=\"

Though Splunk cannot ID a timestamp. I'm not sure what I've done wrong here...

Tags (1)
0 Karma
1 Solution

davebrooking
Contributor

I think the issue could have something to do with the age of the timestamp, there is another setting MAX_DAYS_AGO that defaults to 2000 days (approx 5.5 years) your date is 2001 which is over 12 years ago. Try setting MAX_DAYS_AGO to its maximum value 10951 (approx 30 years)

View solution in original post

davebrooking
Contributor

I think the issue could have something to do with the age of the timestamp, there is another setting MAX_DAYS_AGO that defaults to 2000 days (approx 5.5 years) your date is 2001 which is over 12 years ago. Try setting MAX_DAYS_AGO to its maximum value 10951 (approx 30 years)

lukejadamec
Super Champion

Nice catch

0 Karma

himynamesdave
Contributor

BINGO! Thanks for your help.

0 Karma

agodoy
Communicator

Not sure which field has the timestamp that you want to use. I have read of something like this working before when the time format alone would not.

TIME_FORMAT=Received_date="%d-%m-%Y %H:%M:%S"
0 Karma

himynamesdave
Contributor

I was not aware you could use a text string in the TIME_FORMAT field. Looking at the docs it calls for a .

HOWEVER, when I change the props.conf file to the following, Splunk can identify the %H:%M:%S part of the timestamp :S

# your settings
MAX_TIMESTAMP_LOOKAHEAD=1000
NO_BINARY_CHECK=1
SHOULD_LINEMERGE=false
TIME_FORMAT=Accepted_date="%d-%m-%Y %H:%M:%S"
0 Karma

lukejadamec
Super Champion

The default MAX_TIMESTAMP_LOOKAHEAD is 150 characters. That might be why it's not finding it.

The setting for the log you posted would be 394 characters, but it looks like the size of the log can vary considerably.

Try setting MAX_TIMESTAMP_LOOKAHEAD = 1000

The documentation says this will effect indexing performance because it takes Splunk longer to find the timestamp:

http://docs.splunk.com/Documentation/Splunk/latest/Data/Tunetimestampextractionforbetterindexingperf...

See this post for a similar example:

http://answers.splunk.com/answers/4338/time-stamping-problem-using-time_prefix

himynamesdave
Contributor

Ah, OK. Although Splunk still fails to read the timestamp after removing the \

0 Karma

lukejadamec
Super Champion

You don't need to escape the = in the TIME_PREFIX

0 Karma

himynamesdave
Contributor

Thanks for the suggestion, but declaring MAX_TIMESTAMP_LOOKAHEAD=x hasn't worker either 😞

# your settings
MAX_TIMESTAMP_LOOKAHEAD=1000
NO_BINARY_CHECK=1
SHOULD_LINEMERGE=false
TIME_FORMAT=%d-%m-%Y %H:%M:%S
TIME_PREFIX=Accepted_date\="
0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...