Getting Data In
Highlighted

Event timestamp is getting truncated.

Engager

Hi All,

Need your help inputs on below issue.

We have applied Line breaking configuration its working fine on PREPROD but on PROD time stamp in the events are getting truncated intermittently.

Log File Entry (truncated text in bold):-
2018/02/19 04:47:55.09,GPP.TODP.ACK.PAYMNT.INSTRCTN.OUT,414d51205052444750503031202020205a87e19520f85d05,414d51204c41554b4e53494c202020205a7eab8f24c04bf6,pain.002.001.06,,,I02JF4754HOD0D1N,569d7e66fba74b5faebf2bd3eda595edK0,,W8281188204,

Event In Splunk after indexing:-

47:55.09,GPP.TODP.ACK.PAYMNT.INSTRCTN.OUT,414d51205052444750503031202020205a87e19520f85d05,414d51204c41554b4e53494c202020205a7eab8f24c04bf6,pain.002.001.06,,,I02JF4754HOD0D1N,569d7e66fba74b5faebf2bd3eda595edK0,,W8281188204,

0 Karma
Highlighted

Re: Event timestamp is getting truncated.

Super Champion

can you provide your props.conf

0 Karma
Highlighted

Re: Event timestamp is getting truncated.

Engager

EXTRACT-isoClearSysRef = ^(?:[^,\n],){6}(?P\w+)
EXTRACT-mqCoreId = ^(?:[^,\n]
,){3}(?P[^,]+)
EXTRACT-timeStamp,queueName,MQMsgId,isoMsgDefId,TX_UID = ^(?P[^,]+),(?P[^,]+),(?P[^,]+)(?:[^,\n],){2}(?P[^,]+)[^,\n],(?P[^,]+)
EXTRACT-timeStamp,queueName,isoMsgDefId,isoMsgId = ^(?P[^,]+),(?P[^,]+)(?:[^,\n],){3}(?P[^,]+),,,(?P[^,]+)
EXTRACT-timeStamp,queueName,mqCoreId,isoMsgDefId,isoMsgId,isoOriginalMsgId,isoOriginalInstructionId = ^(?P[^,]+),(?P[^,]+),[a-f0-9]+,(?P[a-f0-9]+),(?P[^,]+),,,(?P[^,]+),(?P\w+)(?:[^,\n]
,){2}(?P[^,]+)
EXTRACT-timeStamp,queueName,mqMsgId,isoMsgDefId,isoClearSysRef,isoInstructionId = ^(?P[^,]+),(?P[^,]+),(?P[^,]+)(?:[^,\n],){2}(?P[^,]+),,(?P\w+)(?:[^,\n],){3}(?P[^,]+)
EXTRACT-timeStamp,queueName,mqMsgId,isoMsgDefId,isoInstructionId = ^(?P[^,]+),(?P[^,]+),(?P[^,]+)(?:[^,\n],){2}(?P[^,]+)(?:[^,\n],){6}(?P\w+)
EXTRACT-timeStamp,queueName,mqMsgId,isoMsgDefId,isoMsgId,isoInstructionId = ^(?P[^,]+),(?P[^,]+)[^,\n],(?P[^,]+)(?:[^,\n],){2}(?P[^,]+),,,(?P[^,]+),,(?P[^,]+)
EXTRACT-timeStamp,queueName,mqMsgId,mqCoreId,isoMsgDefId,isoClearSysRef,isoMsgId,isoOriginalInstructionId = ^(?P[^,]+),(?P[^,]+),(?P[^,]+),(?P[^,]+),(?P[^,]+),,,(?P[^,]+),(?P[^,]+)(?:[^,\n],){2}(?P\w+)
EXTRACT-timeStamp,queueName,mqMsgId,mqCoreId,isoMsgDefId,isoMsgId,isoInstructionId = ^(?P[^,]+),(?P[^,]+),(?P[^,]+),(?P[^,]+),(?P\w+.\d+.\d+.\d+)(?:[^,\n]
,){4}(?P\w+),,(?P[^,]+)EXTRACT-timeStamp,queueName,MsgID = ^(?P\d+/\d+/\d+\s+\d+:\d+:\d+.\d+),(?P[^,]+)[^,\n]*,(?P[^,]+)

0 Karma
Highlighted

Re: Event timestamp is getting truncated.

Ultra Champion

That only shows your EXTRACTS. You mention you also defined linebreaking config? What does that look like?

0 Karma
Highlighted

Re: Event timestamp is getting truncated.

SplunkTrust
SplunkTrust

Is this your search head props.conf? You should paste the indexer props

If this is your indexer props, then you need to set your base configs which tells Splunk how to identify the timestamp and linebreak

0 Karma
Highlighted

Re: Event timestamp is getting truncated.

Legend

@SagarSplunk, have you checked the props.conf for both Pre-Prod and Prod are the same or not (provided data is the same in both the environments)?




| eval message="Happy Splunking!!!"


0 Karma
Highlighted

Re: Event timestamp is getting truncated.

Engager

@niketnilay Yes both environments are having same props.conf configuration.

0 Karma
Highlighted

Re: Event timestamp is getting truncated.

SplunkTrust
SplunkTrust

Are the date and time on separate lines?
You say you have applied line breaking, but I see nothing in your props related to line breaking.
I also see nothing in the props relating to parsing timestamps. At a minimum, you should have TIMEPREFIX, TIMEFORMAT, and MAXTIMESTAMPLOOKAHEAD.

---
If this reply helps you, an upvote would be appreciated.
0 Karma
Highlighted

Re: Event timestamp is getting truncated.

Engager

Hi.there is is no prefix event starts with the time stamps only e.g. 2018/02/19 04:47:55.09
The issue is not line breaking, The data itself is not present in events.

If Line breaking is the issue, truncated event data will get added to earlier event but that is not case here , data is not at all indexing.

0 Karma
Highlighted

Re: Event timestamp is getting truncated.

SplunkTrust
SplunkTrust

If the timestamp is at the beginning of the line then set TIME_PREFIX = ^. You should also set TIME_FORMAT to something, probably %Y/%m/%d %H:%M:%S.%2N.
Have you verified the inputs.conf file in PROD references the correct sourcetype?

---
If this reply helps you, an upvote would be appreciated.
0 Karma