Need help extracting timestamp from unstructured data (JSON)

[nmohammed]

Need help to extract timestamp and structure data -

{"time":"2017-12-12 16:25:27.418 +05:30", "severity":"INFORMATION", "_tag":"", "correlation_id":"null","Message":"[RequestController] RequestController - POST - api/request - [STARTS]", "client_id":"null", "instance_id":"null", "class_name_hierrarchy":"","method":"", "response_time":""} {"time":"2017-12-12 16:25:27.418 +05:30", "severity":"INFORMATION", "_tag":"", "correlation_id":"null","Message":"[EPCAdapter] - CreateRequest - Before Get OAuthToken - ", "client_id":"null", "instance_id":"null", "class_name_hierrarchy":"","method":"", "response_time":""} {"time":"2017-12-12 16:25:27.418 +05:30", "severity":"INFORMATION", "_tag":"", "correlation_id":"null","Message":"[HttpCommunicationHelper] In Execute Request Method", "client_id":"null", "instance_id":"null", "class_name_hierrarchy":"","method":"", "response_time":""} {"time":"2017-12-12 16:25:27.418 +05:30", "severity":"INFORMATION", "_tag":"", "correlation_id":"null","Message":"[HttpCommunicationHelper] Request Type is POST", "client_id":"null", "instance_id":"null", "class_name_hierrarchy":"","method":"", "response_time":""} {"time":"2017-12-12 16:25:27.418 +05:30", "severity":"INFORMATION", "_tag":"", "correlation_id":"null","Message":"[HttpCommunicationHelper] Request Base URL is https://int.api.ellielabs.com/", "client_id":"null", "instance_id":"null", "class_name_hierrarchy":"","method":"", "response_time":""}

[lguinn2]

You should input this file on a test machine somewhere using the Add Data wizard. The wizard will show you a preview of how the data will be parsed, and allow you to experiment with various props.conf settings, such as linebreaking and timestamping.

For JSON inputs, you can use INDEXED_EXTRACTIONS if you want, or simply parse the file as plain text.
INDEXED_EXTRACTIONS will consume more disk space and potentially lower the search performance, but it may be your best choice when the data format is variable.

Option 1: no indexed extractions (recommended if it works) props.conf

[yoursourcetypehere]
TRUNCATE = 0
CHARSET = UTF-8
KV_MODE=JSON
SHOULD_LINEMERGE=true
BREAK_ONLY_BEFORE={"time":"
TIME_PREFIX={"time":"

Option 2: indexed extractions props.conf

[yoursourcetypehere]
TRUNCATE = 0
CHARSET = UTF-8
KV_MODE = none
INDEXED_EXTRACTIONS=JSON
TIMESTAMP_FIELDS = time

Finally, you may need to make sure that each new event starts on a new line in the log file.

[micahkemp · Accepted solution]

This should work:

[<sourcetype>]
SHOULD_LINEMERGE = FALSE
LINE_BREAKER = ([\s\n\r]+){"time":"
TIME_PREFIX = {"time":"
TIME_FORMAT = %Y-%m-%d %H:%M:%S.%3Q %:z

The above assumes that each event starts with {"time", and each instance of {"time" starts an entirely new event.

[nmohammed]

Sorry for the later reply, while I was out for sometime. But this worked as expected.

Thank you all for help.

[somesoni2]

On Indexer/HeavyForwarder, props.conf (event breaking and timestamp parsing configuration)

[YourSourceType]
SHOULD_LINEMERGE = false
LINE_BREAKER =([\r\n]+)(?=\{\"time\")
TIME_PREFIX = ^\{\"time\"\:
TIME_FORMAT = %Y-%m-%d %H:%M:%S.%3N %:z
MAX_TIMESTAMP_LOOKAHEAD = 30

On Search Head, props.conf (field extraction configuration)

[YourSourceType]
KV_MODE = json