Getting Data In

How do I edit my props.conf for proper timestamp extraction?

Path Finder

My log sample event looks like this
"id": "2015-03-02_20-10-12", "keepLog": false
"id": "2015-03-19_10-26-38", "keepLog": false

etc. I have to extract this timestamp for the various events. My props.conf looks like below,

[json_no_timestamp]
TIME_PREFIX = :\s
TIME_FORMAT = %Y-%m-%d_%H-%M-%S

I also tried

[json_no_timestamp]
TIME_PREFIX = "id":\s
TIME_FORMAT = %Y-%m-%d_%H-%M-%S

but still doesn't work. Its still showing the system time for the events. How should my TIME prefix and format look like?

0 Karma
1 Solution

Path Finder

Thanks to Stephen and esix for their help on this. Really appreciate it. This is what worked for me finally
[json_timestamp]
INDEXED_EXTRACTIONS = json
TIME_FORMAT = %s%3N

TIMESTAMP_FIELDS = timestamp

TIME_PREFIX = timestamp
SHOULD_LINEMERGE = false
description = JavaScript Object Notation
category = Structured
disabled = false
pulldown_type = true

TIMESTAMP_FIELDS didn't unfortunately work with the forwarder (although it worked in the data preview)

View solution in original post

0 Karma

Path Finder

Thanks to Stephen and esix for their help on this. Really appreciate it. This is what worked for me finally
[json_timestamp]
INDEXED_EXTRACTIONS = json
TIME_FORMAT = %s%3N

TIMESTAMP_FIELDS = timestamp

TIME_PREFIX = timestamp
SHOULD_LINEMERGE = false
description = JavaScript Object Notation
category = Structured
disabled = false
pulldown_type = true

TIMESTAMP_FIELDS didn't unfortunately work with the forwarder (although it worked in the data preview)

View solution in original post

0 Karma

Motivator

when people provide useful information, please upvote them! i accepted your answer, since you provided what worked. 🙂

0 Karma

Path Finder

I think its my bad to have described my logs in part. But my actual logs look like below

{"building": false, "changeSet": {"items": [], "kind": null}, "builtOn": "rhel6", "description": null, "artifacts": [], "timestamp": 1430241584496, "number": 13, "actions": [{"causes": [{"upstreamBuild": 14, "shortDescription": "Started by upstream project \"answers\" build number 14", "upstreamProject": "answers", "upstreamUrl": "job/answers/"}]}, {}, {}, {}, {}, {"highlightsData": "[{\"Previous Job\":\"answers#14\"},{\"Previous Job\":\"answers_se\"},{\"Build host\":\"rhel6\"}]", "highlightsTable": "

Global Patterns

Previous Job:Started by answersPrevious Job:Started by answers #8Build host:Built on rhel6"}], "id": "2015-04-28_17-19-44", "keepLog": false, "url": "http://thefactory.xyz.com:9999/jenkins/job/answers/13/", "culprits": [], "result": "SUCCESS", "executor": null, "duration": 377658, "fullDisplayName": "answers", "estimatedDuration": 298415}

This is just one line. The entire splunk-logged file is a json dump. now I need to extract the time from either the 'ID' field or the 'timestamp' field. Will the same props work for these? (I tried it didn't). My ultimate aim is to have the logs logged in that timestamp (2015-04-28 in this case) instead of when I create this logs. Any help will be largely useful....

0 Karma

Splunk Employee
Splunk Employee

To build on this...

   [json_no_timestamp]
  TIME_PREFIX = ^\"id\":\s\"
  TIME_FORMAT = %Y-%m-%d_%H-%M-%S

Look from the beginning of the line and also, just to be safe, escape the quotes..

Path Finder

[json_no_timestamp]
TIME_PREFIX=^\"id\":\s\"
TIME_FORMAT=%Y-%m-%d_%H-%M-%S
MAX_TIMESTAMP_LOOKAHEAD = 1048

Here is my updated props. I tried 512 and 1048 for timestamp look ahead, but its still logging the time when I create the log with my script. It's still not taking the time from the ID parameter.

Setting the lookahead will make splunk to look that many number of characters for the timestamp in every line. Did I get it right? I was wondering if I could match the 'timestamp' and deduce time with the epoch time if possible instead of getting it from 'id' field. Can you tell me how I can convert my epoch timestamp into something recognizable by splunk.

0 Karma

Path Finder

Also tried
[json_no_timestamp]
TIME_PREFIX="timestamp":
TIME_FORMAT=%s%3N
MAX_TIMESTAMP_LOOKAHEAD = 512

still nothing!!

0 Karma

Path Finder

Thanks for the quick response. I think its my bad to have described my logs in part. But my actual logs look like below

{"building": false, "changeSet": {"items": [], "kind": null}, "builtOn": "rhel6", "description": null, "artifacts": [], "timestamp": 1430241584496, "number": 13, "actions": [{"causes": [{"upstreamBuild": 14, "shortDescription": "Started by upstream project \"answers\" build number 14", "upstreamProject": "answers", "upstreamUrl": "job/answers/"}]}, {}, {}, {}, {}, {"highlightsData": "[{\"Previous Job\":\"answers#14\"},{\"Previous Job\":\"answers_se\"},{\"Build host\":\"rhel6\"}]", "highlightsTable": "

Global Patterns

Previous Job:Started by answersPrevious Job:Started by answers #8Build host:Built on rhel6"}], "id": "2015-04-28_17-19-44", "keepLog": false, "url": "http://thefactory.xyz.com:9999/jenkins/job/answers/13/", "culprits": [], "result": "SUCCESS", "executor": null, "duration": 377658, "fullDisplayName": "answers", "estimatedDuration": 298415}

This is just one line. The entire splunk-logged file is a json dump. now I need to extract the time from either the 'ID' field or the 'timestamp' field. Will the same props work for these? (I tried it didn't). My ultimate aim is to have the logs logged in that timestamp (2015-04-28 in this case) instead of when I create this logs. Any help will be largely useful....

0 Karma

Splunk Employee
Splunk Employee

Try this

MAX_TIMESTAMP_LOOKAHEAD = 512
TIME_PREFIX  = \"id\"\: \"

This should capture the "id": " group, assuming this is the only id string in the event. You also should look at adjusting the lookahead depending on how far into the event the timestamp is..

0 Karma

Motivator

Hello!
Try this:

  [json_no_timestamp]
 TIME_PREFIX = "id":\s\"
 TIME_FORMAT = %Y-%m-%d_%H-%M-%S

Path Finder

Thanks for the quick response. I think its my bad to have described my logs in part. But my actual logs look like below

{"building": false, "changeSet": {"items": [], "kind": null}, "builtOn": "rhel6", "description": null, "artifacts": [], "timestamp": 1430241584496, "number": 13, "actions": [{"causes": [{"upstreamBuild": 14, "shortDescription": "Started by upstream project \"answers\" build number 14", "upstreamProject": "answers", "upstreamUrl": "job/answers/"}]}, {}, {}, {}, {}, {"highlightsData": "[{\"Previous Job\":\"answers#14\"},{\"Previous Job\":\"answers_se\"},{\"Build host\":\"rhel6\"}]", "highlightsTable": "

Global Patterns

Previous Job:Started by answersPrevious Job:Started by answers #8Build host:Built on rhel6"}], "id": "2015-04-28_17-19-44", "keepLog": false, "url": "http://thefactory.xyz.com:9999/jenkins/job/answers/13/", "culprits": [], "result": "SUCCESS", "executor": null, "duration": 377658, "fullDisplayName": "answers", "estimatedDuration": 298415}

This is just one line. The entire splunk-logged file is a json dump. now I need to extract the time from either the 'ID' field or the 'timestamp' field. Will the same props work for these? (I tried it didn't). My ultimate aim is to have the logs logged in that timestamp (2015-04-28 in this case) instead of when I create this logs. Any help will be largely useful....

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!