I need some help getting me config right in pros.conf.
When the data comes I can see the _time is not set to the value passed for TimeStamp. It is set to the time the event was ingested. For legacy reasons we will have a queue between the app and logging hosts so there will be latency on the messages so setting it to the passed time is critical... The one thing I have noticed is that the decimal places on TimeStamp vary form 5 to 7 digits and we are using %9N. I tried setting this to %3N hoping it would just ignore the characters after but no joy. Do we need to make sure we fix the number of decimals in the logging code?
I do not see any data issues when checking DataQuality.
Any help is very much appreciated!
I am using the following query to evaluate the drift. I know it is not being set as I have cause a delay by sitting on a breakpoint in the logging code.
index=telemetry_*_event_*
| fields _time, TimeStamp, index
| fields - _raw
| eval epoch_time_span=strptime('TimeStamp',"%Y-%m-%dT%H:%M:%S.%9N")
| eval diff=epoch_time_span - _time
| table _time, TimeStamp, epoch_time_span, diff, index
| sort diff
Sample results:
_time TimeStamp epoch_time_span diff
2020-05-08T16:04:10.324-0600 2020-05-08T16:04:04.5663643Z 1588953845 -5.758
2020-05-08T16:01:19.641-0600 2020-05-08T16:01:19.5349868Z 1588953680 -0.106
2020-05-08T15:54:05.559-0600 2020-05-08T15:54:05.4668267Z 1588953245 -0.092
2020-05-08T15:54:17.723-0600 2020-05-08T15:54:17.715911Z 1588953258 -0.007
2020-05-08T16:01:31.924-0600 2020-05-08T16:01:31.9176148Z 1588953692 -0.006
2020-05-08T16:01:34.754-0600 2020-05-08T16:01:34.7519748Z 1588953695 -0.002
Here is the _raw data from splunk:
{"TimeStamp":"2020-05-08T16:04:20.6492094Z","Level":"Debug","Properties":{"Action":"XXXX","Channel":"XXXX","CorrelationID":"7c003283-a81e-4b11-97ff-c926e53f4fa6","Host":"XXXX","ServiceID":{"Application":{"Name":"XXXX"},"Environment":"development","Tenant":"XXXX"},"ProcessID":"22908","ProcessName":"XXXX","ThreadID":"18872","ThreadName":"XXXX","User":{"Domain":"XXXX","ID":"4","Name":"XXXX","Location":"XXXX","Custom":{"OrganizationCode":"XXXX"}},"Performance":{"DataCallCount":1,"ElapsedTime":122.0,"ElapsedTimeSpan":"0:00:00.122","HasError":false,"Outbound":false,"Ticks":1223834}},"Version":"0.0.1.19100"}
Here is the config:
[telemetry_source_type]
#Internal
pulldown_type = true
#Meta data
category = Structured
description = JSON based source Type for Telemetry events
disabled = false
#We can change these
ANNOTATE_PUNCT = false
BREAK_ONLY_BEFORE_DATE = false
DATETIME_CONFIG = NONE
KV_MODE = json
LINE_BREAKER = ([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD = 32
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%9N
TIME_PREFIX = "TimeStamp":"
TRUNCATE = 4194304
TZ = GMT
... View more