topic Re: Event timestamp is getting truncated. in Getting Data In

Event timestamp is getting truncated.

SagarSplunk — Mon, 19 Feb 2018 06:51:50 GMT

Hi All,

Need your help inputs on below issue.

We have applied Line breaking configuration its working fine on PREPROD but on PROD time stamp in the events are getting truncated intermittently.

Log File Entry (truncated text in bold):-
2018/02/19 04:47:55.09,GPP.TODP.ACK.PAYMNT.INSTRCTN.OUT,414d51205052444750503031202020205a87e19520f85d05,414d51204c41554b4e53494c202020205a7eab8f24c04bf6,pain.002.001.06,,,I02JF4754HOD0D1N,569d7e66fba74b5faebf2bd3eda595edK0,,W8281188204,

Event In Splunk after indexing:-

47:55.09,GPP.TODP.ACK.PAYMNT.INSTRCTN.OUT,414d51205052444750503031202020205a87e19520f85d05,414d51204c41554b4e53494c202020205a7eab8f24c04bf6,pain.002.001.06,,,I02JF4754HOD0D1N,569d7e66fba74b5faebf2bd3eda595edK0,,W8281188204,

Re: Event timestamp is getting truncated.

493669 — Mon, 19 Feb 2018 06:54:57 GMT

can you provide your props.conf

Re: Event timestamp is getting truncated.

niketn — Mon, 19 Feb 2018 07:08:45 GMT

@SagarSplunk, have you checked the props.conf for both Pre-Prod and Prod are the same or not (provided data is the same in both the environments)?

Re: Event timestamp is getting truncated.

SagarSplunk — Mon, 19 Feb 2018 10:38:02 GMT

@niketnilay Yes both environments are having same props.conf configuration.

Re: Event timestamp is getting truncated.

SagarSplunk — Mon, 19 Feb 2018 10:40:47 GMT

EXTRACT-isoClearSysRef = ^(?:[^,\n],){6}(?P\w+)
EXTRACT-mqCoreId = ^(?:[^,\n],){3}(?P[^,]+)
EXTRACT-timeStamp,queueName,MQMsgId,isoMsgDefId,TX_UID = ^(?P[^,]+),(?P[^,]+),(?P[^,]+)(?:[^,\n],){2}(?P[^,]+)[^,\n],(?P[^,]+)
EXTRACT-timeStamp,queueName,isoMsgDefId,isoMsgId = ^(?P[^,]+),(?P[^,]+)(?:[^,\n],){3}(?P[^,]+),,,(?P[^,]+)
EXTRACT-timeStamp,queueName,mqCoreId,isoMsgDefId,isoMsgId,isoOriginalMsgId,isoOriginalInstructionId = ^(?P[^,]+),(?P[^,]+),[a-f0-9]+,(?P[a-f0-9]+),(?P[^,]+),,,(?P[^,]+),(?P\w+)(?:[^,\n],){2}(?P[^,]+)
EXTRACT-timeStamp,queueName,mqMsgId,isoMsgDefId,isoClearSysRef,isoInstructionId = ^(?P[^,]+),(?P[^,]+),(?P[^,]+)(?:[^,\n],){2}(?P[^,]+),,(?P\w+)(?:[^,\n],){3}(?P[^,]+)
EXTRACT-timeStamp,queueName,mqMsgId,isoMsgDefId,isoInstructionId = ^(?P[^,]+),(?P[^,]+),(?P[^,]+)(?:[^,\n],){2}(?P[^,]+)(?:[^,\n],){6}(?P\w+)
EXTRACT-timeStamp,queueName,mqMsgId,isoMsgDefId,isoMsgId,isoInstructionId = ^(?P[^,]+),(?P[^,]+)[^,\n],(?P[^,]+)(?:[^,\n],){2}(?P[^,]+),,,(?P[^,]+),,(?P[^,]+)
EXTRACT-timeStamp,queueName,mqMsgId,mqCoreId,isoMsgDefId,isoClearSysRef,isoMsgId,isoOriginalInstructionId = ^(?P[^,]+),(?P[^,]+),(?P[^,]+),(?P[^,]+),(?P[^,]+),,,(?P[^,]+),(?P[^,]+)(?:[^,\n],){2}(?P\w+)
EXTRACT-timeStamp,queueName,mqMsgId,mqCoreId,isoMsgDefId,isoMsgId,isoInstructionId = ^(?P[^,]+),(?P[^,]+),(?P[^,]+),(?P[^,]+),(?P\w+.\d+.\d+.\d+)(?:[^,\n],){4}(?P\w+),,(?P[^,]+)EXTRACT-timeStamp,queueName,MsgID = ^(?P\d+/\d+/\d+\s+\d+:\d+:\d+.\d+),(?P[^,]+)[^,\n]*,(?P[^,]+)

Re: Event timestamp is getting truncated.

FrankVl — Mon, 19 Feb 2018 14:06:36 GMT

That only shows your EXTRACTS. You mention you also defined linebreaking config? What does that look like?

Re: Event timestamp is getting truncated.

richgalloway — Tue, 29 Sep 2020 18:05:23 GMT

Are the date and time on separate lines?
You say you have applied line breaking, but I see nothing in your props related to line breaking.
I also see nothing in the props relating to parsing timestamps. At a minimum, you should have TIME_PREFIX, TIME_FORMAT, and MAX_TIMESTAMP_LOOKAHEAD.

Re: Event timestamp is getting truncated.

SagarSplunk — Tue, 20 Feb 2018 13:40:57 GMT

Hi.there is is no prefix event starts with the time stamps only e.g. 2018/02/19 04:47:55.09
The issue is not line breaking, The data itself is not present in events.

If Line breaking is the issue, truncated event data will get added to earlier event but that is not case here , data is not at all indexing.

Re: Event timestamp is getting truncated.

richgalloway — Thu, 22 Feb 2018 13:54:08 GMT

If the timestamp is at the beginning of the line then set TIME_PREFIX = ^. You should also set TIME_FORMAT to something, probably %Y/%m/%d %H:%M:%S.%2N.
Have you verified the inputs.conf file in PROD references the correct sourcetype?

Re: Event timestamp is getting truncated.

skoelpin — Thu, 22 Feb 2018 14:38:24 GMT

Is this your search head props.conf? You should paste the indexer props

If this is your indexer props, then you need to set your base configs which tells Splunk how to identify the timestamp and linebreak