Getting Data In

Handling events with the same timestamp

Staging_2
New Member

I am extracting logs from a file which contain entries with two timestamp log entries:
1. eventTimestamp
2. timestamp

The later is included by my logging framework. I occasionally write events where the timestamp is the same. In these cases the events gets grouped together as shown below:

The events below would all appear under the timestamp: 2016-12-28T17:07:55.946Z.
```
{"eventCode":13400,"eventDetails":{"title":"xxxx","source":"xxx","code":"InvalidArgument","subscriptionType":"active","system":"xxx"},"userDetails":{"userAgent":"xxxx","userApp":"sc"},"eventTimestamp":"2016-12-28T17:07:55.946Z","uid":"xxx","accountDetails":{"account":"xx","email":"xxxxx,"environment":"xxx"},"level":"info","message":"","timestamp":"2016-12-28T17:07:39.593Z"}
{"eventCode":13400,"eventDetails":{"title":"xxxx","source":"xxx","code":"InvalidArgument","subscriptionType":"active","system":"xxx"},"userDetails":{"userAgent":"","userApp":"sc"},"eventTimestamp":"2016-12-28T17:07:56.766Z","uid":"xxx","accountDetails":{"account":"xxxx","email":"xxxx","environment":"xxx"},"level":"info","message":"","timestamp":"2016-12-28T17:07:39.593Z"}

```

I have tried creating a props.conf file with the following configuration:


[api_reporting]
SHOULD_LINEMERGE=false
TIME_PREFIX=eventTimestamp
MAX_TIMESTAMP_LOOKAHEAD=10

However, I am continuing to experience the issue. I have followed [1] to determine if my props.conf file is read and it seems to be the case. The configuration given above were taken from [2].

[1] https://docs.splunk.com/Documentation/Splunk/6.5.1/Troubleshooting/Usebtooltotroubleshootconfigurati...
[2] https://answers.splunk.com/answers/80488/splunk-treating-multiple-lines-as-one-event-since-they-have...

Tags (2)
0 Karma

somesoni2
Revered Legend

Try this for your props.conf (on Indexer or Heavy Forwarder)

[api_reporting]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?=\{\"eventCode\")
TIME_PREFIX = eventTimestamp\":\"
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N
MAX_TIMESTAMP_LOOKAHEAD = 23
0 Karma

Staging_2
New Member

Thank You.

I did not get a chance to try it out yet. I will update the thread once I get a chance to test it.

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Manage Edge Processor pipelines end to end with APIs

This guide uses the Data Management APIs to create an Edge Processor, onboard one instance, deploy a pipeline ...

Splunk Brings Proof of Control to Digital Asset Transfers

Digital assets now sit inside the core of financial services. HSBC launched its Tokenized Deposit Service in ...

Manage Ingest Processor pipelines end to end with APIs

This guide uses the Data Management APIs to create an Ingest Processor pipeline, deploy it to the ...