topic Re: How do I edit my props.conf for proper timestamp extraction? in Getting Data In

How do I edit my props.conf for proper timestamp extraction?

hvaithia — Fri, 24 Apr 2015 00:15:14 GMT

My log sample event looks like this
"id": "2015-03-02_20-10-12", "keepLog": false
"id": "2015-03-19_10-26-38", "keepLog": false

etc. I have to extract this timestamp for the various events. My props.conf looks like below,

[json_no_timestamp]
TIME_PREFIX = :\s
TIME_FORMAT = %Y-%m-%d_%H-%M-%S

I also tried

[json_no_timestamp]
TIME_PREFIX = "id":\s
TIME_FORMAT = %Y-%m-%d_%H-%M-%S

but still doesn't work. Its still showing the system time for the events. How should my TIME prefix and format look like?

Re: How do I edit my props.conf for proper timestamp extraction?

stephanefotso — Fri, 24 Apr 2015 04:30:52 GMT

Hello!
Try this:

  [json_no_timestamp]
 TIME_PREFIX = "id":\s\"
 TIME_FORMAT = %Y-%m-%d_%H-%M-%S

Re: How do I edit my props.conf for proper timestamp extraction?

esix_splunk — Fri, 24 Apr 2015 04:49:57 GMT

To build on this...

   [json_no_timestamp]
  TIME_PREFIX = ^\"id\":\s\"
  TIME_FORMAT = %Y-%m-%d_%H-%M-%S

Look from the beginning of the line and also, just to be safe, escape the quotes..

Re: How do I edit my props.conf for proper timestamp extraction?

hvaithia — Mon, 28 Sep 2020 19:42:16 GMT

I think its my bad to have described my logs in part. But my actual logs look like below

{"building": false, "changeSet": {"items": [], "kind": null}, "builtOn": "rhel6", "description": null, "artifacts": [], "timestamp": 1430241584496, "number": 13, "actions": [{"causes": [{"upstreamBuild": 14, "shortDescription": "Started by upstream project \"answers\" build number 14", "upstreamProject": "answers", "upstreamUrl": "job/answers/"}]}, {}, {}, {}, {}, {"highlightsData": "[{\"Previous Job\":\"answers#14\"},{\"Previous Job\":\"answers_se\"},{\"Build host\":\"rhel6\"}]", "highlightsTable": "

Global Patterns

Previous Job:Started by answersPrevious Job:Started by answers #8Build host:Built on rhel6"}], "id": "2015-04-28_17-19-44", "keepLog": false, "url": "http://thefactory.xyz.com:9999/jenkins/job/answers/13/", "culprits": [], "result": "SUCCESS", "executor": null, "duration": 377658, "fullDisplayName": "answers", "estimatedDuration": 298415}

This is just one line. The entire splunk-logged file is a json dump. now I need to extract the time from either the 'ID' field or the 'timestamp' field. Will the same props work for these? (I tried it didn't). My ultimate aim is to have the logs logged in that timestamp (2015-04-28 in this case) instead of when I create this logs. Any help will be largely useful....

Re: How do I edit my props.conf for proper timestamp extraction?

hvaithia — Mon, 28 Sep 2020 19:42:19 GMT

Thanks for the quick response. I think its my bad to have described my logs in part. But my actual logs look like below

Global Patterns

This is just one line. The entire splunk-logged file is a json dump. now I need to extract the time from either the 'ID' field or the 'timestamp' field. Will the same props work for these? (I tried it didn't). My ultimate aim is to have the logs logged in that timestamp (2015-04-28 in this case) instead of when I create this logs. Any help will be largely useful....

Re: How do I edit my props.conf for proper timestamp extraction?

hvaithia — Mon, 28 Sep 2020 19:42:22 GMT

Thanks for the quick response. I think its my bad to have described my logs in part. But my actual logs look like below

Global Patterns

Re: How do I edit my props.conf for proper timestamp extraction?

esix_splunk — Tue, 28 Apr 2015 23:48:35 GMT

Try this

MAX_TIMESTAMP_LOOKAHEAD = 512
TIME_PREFIX  = \"id\"\: \"

This should capture the "id": " group, assuming this is the only id string in the event. You also should look at adjusting the lookahead depending on how far into the event the timestamp is..

Re: How do I edit my props.conf for proper timestamp extraction?

hvaithia — Mon, 28 Sep 2020 19:47:23 GMT

[json_no_timestamp]
TIME_PREFIX=^\"id\":\s\"
TIME_FORMAT=%Y-%m-%d_%H-%M-%S
MAX_TIMESTAMP_LOOKAHEAD = 1048

Here is my updated props. I tried 512 and 1048 for timestamp look ahead, but its still logging the time when I create the log with my script. It's still not taking the time from the ID parameter.

Setting the lookahead will make splunk to look that many number of characters for the timestamp in every line. Did I get it right? I was wondering if I could match the 'timestamp' and deduce time with the epoch time if possible instead of getting it from 'id' field. Can you tell me how I can convert my epoch timestamp into something recognizable by splunk.

Re: How do I edit my props.conf for proper timestamp extraction?

hvaithia — Mon, 28 Sep 2020 19:47:26 GMT

Also tried
[json_no_timestamp]
TIME_PREFIX="timestamp":
TIME_FORMAT=%s%3N
MAX_TIMESTAMP_LOOKAHEAD = 512

still nothing!!

Re: How do I edit my props.conf for proper timestamp extraction?

hvaithia — Mon, 28 Sep 2020 19:48:53 GMT

Thanks to Stephen and esix for their help on this. Really appreciate it. This is what worked for me finally
[json_timestamp]
INDEXED_EXTRACTIONS = json
TIME_FORMAT = %s%3N

TIMESTAMP_FIELDS = timestamp

TIME_PREFIX = timestamp
SHOULD_LINEMERGE = false
description = JavaScript Object Notation
category = Structured
disabled = false
pulldown_type = true

TIMESTAMP_FIELDS didn't unfortunately work with the forwarder (although it worked in the data preview)

Re: How do I edit my props.conf for proper timestamp extraction?

piebob — Fri, 01 May 2015 20:00:23 GMT

when people provide useful information, please upvote them! i accepted your answer, since you provided what worked. 🙂