Getting Data In

Event timestamp is getting truncated.

SagarSplunk
Engager

Hi All,

Need your help inputs on below issue.

We have applied Line breaking configuration its working fine on PREPROD but on PROD time stamp in the events are getting truncated intermittently.

Log File Entry (truncated text in bold):-
2018/02/19 04:47:55.09,GPP.TODP.ACK.PAYMNT.INSTRCTN.OUT,414d51205052444750503031202020205a87e19520f85d05,414d51204c41554b4e53494c202020205a7eab8f24c04bf6,pain.002.001.06,,,I02JF4754HOD0D1N,569d7e66fba74b5faebf2bd3eda595edK0,,W8281188204,

Event In Splunk after indexing:-

47:55.09,GPP.TODP.ACK.PAYMNT.INSTRCTN.OUT,414d51205052444750503031202020205a87e19520f85d05,414d51204c41554b4e53494c202020205a7eab8f24c04bf6,pain.002.001.06,,,I02JF4754HOD0D1N,569d7e66fba74b5faebf2bd3eda595edK0,,W8281188204,

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Are the date and time on separate lines?
You say you have applied line breaking, but I see nothing in your props related to line breaking.
I also see nothing in the props relating to parsing timestamps. At a minimum, you should have TIME_PREFIX, TIME_FORMAT, and MAX_TIMESTAMP_LOOKAHEAD.

---
If this reply helps you, Karma would be appreciated.
0 Karma

SagarSplunk
Engager

Hi.there is is no prefix event starts with the time stamps only e.g. 2018/02/19 04:47:55.09
The issue is not line breaking, The data itself is not present in events.

If Line breaking is the issue, truncated event data will get added to earlier event but that is not case here , data is not at all indexing.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

If the timestamp is at the beginning of the line then set TIME_PREFIX = ^. You should also set TIME_FORMAT to something, probably %Y/%m/%d %H:%M:%S.%2N.
Have you verified the inputs.conf file in PROD references the correct sourcetype?

---
If this reply helps you, Karma would be appreciated.
0 Karma

niketn
Legend

@SagarSplunk, have you checked the props.conf for both Pre-Prod and Prod are the same or not (provided data is the same in both the environments)?

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

SagarSplunk
Engager

@niketnilay Yes both environments are having same props.conf configuration.

0 Karma

493669
Super Champion

can you provide your props.conf

0 Karma

SagarSplunk
Engager

EXTRACT-isoClearSysRef = ^(?:[^,\n],){6}(?P\w+)
EXTRACT-mqCoreId = ^(?:[^,\n]
,){3}(?P[^,]+)
EXTRACT-timeStamp,queueName,MQMsgId,isoMsgDefId,TX_UID = ^(?P[^,]+),(?P[^,]+),(?P[^,]+)(?:[^,\n],){2}(?P[^,]+)[^,\n],(?P[^,]+)
EXTRACT-timeStamp,queueName,isoMsgDefId,isoMsgId = ^(?P[^,]+),(?P[^,]+)(?:[^,\n],){3}(?P[^,]+),,,(?P[^,]+)
EXTRACT-timeStamp,queueName,mqCoreId,isoMsgDefId,isoMsgId,isoOriginalMsgId,isoOriginalInstructionId = ^(?P[^,]+),(?P[^,]+),[a-f0-9]+,(?P[a-f0-9]+),(?P[^,]+),,,(?P[^,]+),(?P\w+)(?:[^,\n]
,){2}(?P[^,]+)
EXTRACT-timeStamp,queueName,mqMsgId,isoMsgDefId,isoClearSysRef,isoInstructionId = ^(?P[^,]+),(?P[^,]+),(?P[^,]+)(?:[^,\n],){2}(?P[^,]+),,(?P\w+)(?:[^,\n],){3}(?P[^,]+)
EXTRACT-timeStamp,queueName,mqMsgId,isoMsgDefId,isoInstructionId = ^(?P[^,]+),(?P[^,]+),(?P[^,]+)(?:[^,\n],){2}(?P[^,]+)(?:[^,\n],){6}(?P\w+)
EXTRACT-timeStamp,queueName,mqMsgId,isoMsgDefId,isoMsgId,isoInstructionId = ^(?P[^,]+),(?P[^,]+)[^,\n],(?P[^,]+)(?:[^,\n],){2}(?P[^,]+),,,(?P[^,]+),,(?P[^,]+)
EXTRACT-timeStamp,queueName,mqMsgId,mqCoreId,isoMsgDefId,isoClearSysRef,isoMsgId,isoOriginalInstructionId = ^(?P[^,]+),(?P[^,]+),(?P[^,]+),(?P[^,]+),(?P[^,]+),,,(?P[^,]+),(?P[^,]+)(?:[^,\n],){2}(?P\w+)
EXTRACT-timeStamp,queueName,mqMsgId,mqCoreId,isoMsgDefId,isoMsgId,isoInstructionId = ^(?P[^,]+),(?P[^,]+),(?P[^,]+),(?P[^,]+),(?P\w+.\d+.\d+.\d+)(?:[^,\n]
,){4}(?P\w+),,(?P[^,]+)EXTRACT-timeStamp,queueName,MsgID = ^(?P\d+/\d+/\d+\s+\d+:\d+:\d+.\d+),(?P[^,]+)[^,\n]*,(?P[^,]+)

0 Karma

skoelpin
SplunkTrust
SplunkTrust

Is this your search head props.conf? You should paste the indexer props

If this is your indexer props, then you need to set your base configs which tells Splunk how to identify the timestamp and linebreak

0 Karma

FrankVl
Ultra Champion

That only shows your EXTRACTS. You mention you also defined linebreaking config? What does that look like?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...