topic Re: Line Breaking Issue - text and Json in Getting Data In

Line Breaking Issue - text and Json

arunsoni — Sun, 03 Nov 2024 13:54:29 GMT

Hello,

Below is my log file and I want to break as two log events in splunk using props.conf(regex)

2024-07-31T01:38:09.930Z [INFO] ContentGenerator {"recordType":"CGStats","statType":"global","workFlow":"","front":{"hlsMaster":{"requests":0,"responses":0,"responseMCHit":0,"responseAwaitingDecision":0,"msecSum":"0","msecBins":{"5000":0,"15000":0,"above":0,"50":0,"100":0,"500":0,"1000":0},"errors":0,"codes":{"404":0,"200":0},"codeCategory":{"6":0,"0":0}},"hlsVariant":{"requests":10,"responses":10,"responseMCHit":0,"responseAwaitingDecision":0,"msecSum":"1208","msecBins":{"50":8,"100":0,"500":2,"1000":0,"5000":0,"15000":0,"above":0},"errors":0,"codes":{"404":0,"504":0,"200":10},"codeCategory":{"19":0,"0":10,"5":0}},"dashMPD":{"requests":0,"responses":0,"responseMCHit":0,"responseAwaitingDecision":0,"msecSum":"0","msecBins":{"500":0,"1000":0,"5000":0,"15000":0,"above":0,"50":0,"100":0},"errors":0,"codes":{"200":0},"codeCategory":{}}},"back":{"origin":{"hlsMaster":{"requests":0,"requestCacheCount":0,"responses":0,"response**bleep**":0,"responsesMiss":0,"responsesHeld":0,"msecSum":"0","msecBins":{"500":0,"1000":0,"5000":0,"15000":0,"above":0,"50":0,"100":0},"errors":0,"codes":{"404":0,"200":0}},"hlsVariant":{"requests":12,"requestCacheCount":12,"responses":12,"response**bleep**":0,"responsesMiss":12,"responsesHeld":0,"msecSum":"201","msecBins":{"500":0,"1000":0,"5000":0,"15000":0,"above":0,"50":12,"100":0},"errors":0,"codes":{"504":0,"200":12,"404":0}},"dashMPD":{"requests":0,"requestCacheCount":0,"responses":0,"response**bleep**":0,"responsesMiss":0,"responsesHeld":0,"msecSum":"0","msecBins":{"500":0,"1000":0,"5000":0,"15000":0,"above":0,"50":0,"100":0},"errors":0,"codes":{"200":0}}},"advert":{"hlsMaster":{"requests":0,"requestCacheCount":0,"responses":0,"response**bleep**":0,"responsesMiss":0,"responsesHeld":0,"msecSum":"0","msecBins":{"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0,"above":0},"errors":0,"codes":{"200":0,"404":0}},"hlsVariant":{"requests":0,"requestCacheCount":0,"responses":0,"response**bleep**":0,"responsesMiss":0,"responsesHeld":0,"msecSum":"0","msecBins":{"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0,"above":0},"errors":0,"codes":{"200":0}},"dashMPD":{"requests":0,"requestCacheCount":0,"responses":0,"response**bleep**":0,"responsesMiss":0,"responsesHeld":0,"msecSum":"0","msecBins":{"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0,"above":0},"errors":0,"codes":{"200":0}}},"altcon":{"hlsMaster":{"requests":0,"requestCacheCount":0,"responses":0,"response**bleep**":0,"responsesMiss":0,"responsesHeld":0,"msecSum":"0","msecBins":{"above":0,"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0},"errors":0,"codes":{"200":0}},"hlsVariant":{"requests":0,"requestCacheCount":0,"responses":0,"response**bleep**":0,"responsesMiss":0,"responsesHeld":0,"msecSum":"0","msecBins":{"above":0,"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0},"errors":0,"codes":{"200":0}},"dashMPD":{"requests":0,"requestCacheCount":0,"responses":0,"response**bleep**":0,"responsesMiss":0,"responsesHeld":0,"msecSum":"0","msecBins":{"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0,"above":0},"errors":0,"codes":{"200":0}}}},"decision":{"hls":{"ads":{"markers":0,"opportunities":0,"opportunityDrops":{"8":0,"0":0,"3":0,"4":0,"5":0},"requests":0,"responses":0,"msecSum":"0","msecBins":{"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0,"above":0},"admux":{"responses":0,"timeouts":0,"msecSum":"0","msecBins":{"500":0,"1000":0,"5000":0,"15000":0,"above":0,"50":0,"100":0}},"placedPerOp":{"0":0,"2":0,"3":0},"placements":0,"opAborts":0,"spliceDrops":0,"spliceFails":0,"spliceStarts":0,"spliceEnds":0,"spliceTrims":0,"spliceAborts":0,"preDecision":{"added":0,"request":0,"response":0,"consumed":0,"timeouts":0,"dropped":0}},"acds":{"markers":0,"opportunities":0,"opportunityDrops":{},"requests":0,"responses":0,"msecSum":"0","msecBins":{"above":0,"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0},"altconmux":{"responses":0,"timeouts":0,"msecSum":"0","msecBins":{"above":0,"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0}},"placedPerOp":{},"placements":0,"opAborts":0,"spliceDrops":0,"spliceFails":0,"spliceStarts":0,"spliceEnds":0,"spliceTrims":0,"spliceAborts":0}},"dash":{"ads":{"markers":0,"opportunities":0,"opportunityDrops":{},"requests":0,"responses":0,"msecSum":"0","msecBins":{"above":0,"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0},"admux":{"responses":0,"timeouts":0,"msecSum":"0","msecBins":{"5000":0,"15000":0,"above":0,"50":0,"100":0,"500":0,"1000":0}},"placedPerOp":{},"placements":0,"opAborts":0,"spliceDrops":0,"spliceFails":0,"spliceStarts":0,"spliceEnds":0,"spliceTrims":0,"spliceAborts":0,"preDecision":{"added":0,"request":0,"response":0,"consumed":0,"timeouts":0,"dropped":0}},"acds":{"markers":0,"opportunities":0,"opportunityDrops":{},"requests":0,"responses":0,"msecSum":"0","msecBins":{"5000":0,"15000":0,"above":0,"50":0,"100":0,"500":0,"1000":0},"altconmux":{"responses":0,"timeouts":0,"msecSum":"0","msecBins":{"500":0,"1000":0,"5000":0,"15000":0,"above":0,"50":0,"100":0}},"placedPerOp":{},"placements":0,"opAborts":0,"spliceDrops":0,"spliceFails":0,"spliceStarts":0,"spliceEnds":0,"spliceTrims":0,"spliceAborts":0}}},"session":{"hls":{"requests":0,"responses":0,"restoreRequests":0,"restoreResponses":0,"errors":0,"codes":{"0":0},"restoreErrors":0,"restoreCodes":{"1":0},"msecSum":"0","msecBins":{"5000":0,"15000":0,"above":0,"50":0,"100":0,"500":0,"1000":0},"restoreMsecSum":"0","restoreMsecBins":{"above":0,"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0},"reconfig":0,"reserveLimit":0},"dash":{"requests":0,"responses":0,"restoreRequests":0,"restoreResponses":0,"errors":0,"codes":{},"restoreErrors":0,"restoreCodes":{},"msecSum":"0","msecBins":{"500":0,"1000":0,"5000":0,"15000":0,"above":0,"50":0,"100":0},"restoreMsecSum":"0","restoreMsecBins":{"500":0,"1000":0,"5000":0,"15000":0,"above":0,"50":0,"100":0},"reconfig":0,"reserveLimit":0}},"timestamp":{"statsResetTime":1207442342,"nsTimestamp":2984280751}}

2024-07-31T01:38:09.931Z [INFO] ContentGenerator {"recordType":"CGHealth","ContentGenerator":{"KnownSessions":1,"WaitingForResponse":0,"PendingDeleteSessions":0,"UnderRecovery":0,"jobQueue":0,"JobsEnqueued":5221688,"JobsDequeued":5221688,"AllocatedSessions":1,"CGStatsSessions":1,"HPIReqs":8,"ManifestCacheObjs":83,"SavedState":29159,"HlsCount":1,"DashCount":0,"HpiReq":346395,"HpiCancel":0,"GitRef":"41d2f857114d10689016ff5074144a580b1ba544","Status":200},"DecisionQueue":{"adReqQueue":{"queuedJobs":658,"dequeuedJobs":658,"lowExceeded":0,"highExceeded":0,"maxQueueDepth":1,"deadline":0,"lowCount":0,"highCount":0,"outstanding":0,"lowWater":250,"highWater":500},"boReqQueue":{"queuedJobs":0,"dequeuedJobs":0,"lowExceeded":0,"highExceeded":0,"maxQueueDepth":0,"deadline":0,"lowCount":0,"highCount":0,"outstanding":0,"lowWater":250,"highWater":500}},"MQMessages":{"Messages":{"1511":2,"1508":22,"1514":352,"704":359,"706":6,"1044":658,"709":372,"9":4693470}}}

2024-07-31T01:39:09.058Z [INFO] ContentGenerator {"recordType":"CGStats","statType":"global","workFlow":"","front":{"hlsMaster":{"requests":0,"responses":0,"responseMCHit":0,"responseAwaitingDecision":0,"msecSum":"0","msecBins":{"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0,"above":0},"errors":0,"codes":{"200":0,"404":0},"codeCategory":{"0":0,"6":0}},"hlsVariant":{"requests":10,"responses":10,"responseMCHit":0,"responseAwaitingDecision":0,"msecSum":"1305","msecBins":{"500":0,"1000":2,"5000":0,"15000":0,"above":0,"50":8,"100":0},"errors":0,"codes":{"504":0,"200":10,"404":0},"codeCategory":{"5":0,"19":0,"0":10}},"dashMPD":{"requests":0,"responses":0,"responseMCHit":0,"responseAwaitingDecision":0,"msecSum":"0","msecBins":{"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0,"above":0},"errors":0,"codes":{"200":0},"codeCategory":{}}},"back":{"origin":{"hlsMaster":{"requests":0,"requestCacheCount":0,"responses":0,"response**bleep**":0,"responsesMiss":0,"responsesHeld":0,"msecSum":"0","msecBins":{"above":0,"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0},"errors":0,"codes":{"404":0,"200":0}},"hlsVariant":{"requests":12,"requestCacheCount":12,"responses":12,"response**bleep**":0,"responsesMiss":12,"responsesHeld":0,"msecSum":"287","msecBins":{"50":12,"100":0,"500":0,"1000":0,"5000":0,"15000":0,"above":0},"errors":0,"codes":{"504":0,"200":12,"404":0}},"dashMPD":{"requests":0,"requestCacheCount":0,"responses":0,"response**bleep**":0,"responsesMiss":0,"responsesHeld":0,"msecSum":"0","msecBins":{"above":0,"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0},"errors":0,"codes":{"200":0}}},"advert":{"hlsMaster":{"requests":0,"requestCacheCount":0,"responses":0,"response**bleep**":0,"responsesMiss":0,"responsesHeld":0,"msecSum":"0","msecBins":{"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0,"above":0},"errors":0,"codes":{"404":0,"200":0}},"hlsVariant":{"requests":0,"requestCacheCount":0,"responses":0,"response**bleep**":0,"responsesMiss":0,"responsesHeld":0,"msecSum":"0","msecBins":{"5000":0,"15000":0,"above":0,"50":0,"100":0,"500":0,"1000":0},"errors":0,"codes":{"200":0}},"dashMPD":{"requests":0,"requestCacheCount":0,"responses":0,"response**bleep**":0,"responsesMiss":0,"responsesHeld":0,"msecSum":"0","msecBins":{"above":0,"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0},"errors":0,"codes":{"200":0}}},"altcon":{"hlsMaster":{"requests":0,"requestCacheCount":0,"responses":0,"response**bleep**":0,"responsesMiss":0,"responsesHeld":0,"msecSum":"0","msecBins":{"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0,"above":0},"errors":0,"codes":{"200":0}},"hlsVariant":{"requests":0,"requestCacheCount":0,"responses":0,"response**bleep**":0,"responsesMiss":0,"responsesHeld":0,"msecSum":"0","msecBins":{"above":0,"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0},"errors":0,"codes":{"200":0}},"dashMPD":{"requests":0,"requestCacheCount":0,"responses":0,"response**bleep**":0,"responsesMiss":0,"responsesHeld":0,"msecSum":"0","msecBins":{"500":0,"1000":0,"5000":0,"15000":0,"above":0,"50":0,"100":0},"errors":0,"codes":{"200":0}}}},"decision":{"hls":{"ads":{"markers":0,"opportunities":0,"opportunityDrops":{"8":0,"0":0,"3":0,"4":0,"5":0},"requests":0,"responses":0,"msecSum":"0","msecBins":{"500":0,"1000":0,"5000":0,"15000":0,"above":0,"50":0,"100":0},"admux":{"responses":0,"timeouts":0,"msecSum":"0","msecBins":{"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0,"above":0}},"placedPerOp":{"0":0,"2":0,"3":0},"placements":0,"opAborts":0,"spliceDrops":0,"spliceFails":0,"spliceStarts":0,"spliceEnds":0,"spliceTrims":0,"spliceAborts":0,"preDecision":{"added":0,"request":0,"response":0,"consumed":0,"timeouts":0,"dropped":0}},"acds":{"markers":0,"opportunities":0,"opportunityDrops":{},"requests":0,"responses":0,"msecSum":"0","msecBins":{"500":0,"1000":0,"5000":0,"15000":0,"above":0,"50":0,"100":0},"altconmux":{"responses":0,"timeouts":0,"msecSum":"0","msecBins":{"above":0,"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0}},"placedPerOp":{},"placements":0,"opAborts":0,"spliceDrops":0,"spliceFails":0,"spliceStarts":0,"spliceEnds":0,"spliceTrims":0,"spliceAborts":0}},"dash":{"ads":{"markers":0,"opportunities":0,"opportunityDrops":{},"requests":0,"responses":0,"msecSum":"0","msecBins":{"above":0,"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0},"admux":{"responses":0,"timeouts":0,"msecSum":"0","msecBins":{"15000":0,"above":0,"50":0,"100":0,"500":0,"1000":0,"5000":0}},"placedPerOp":{},"placements":0,"opAborts":0,"spliceDrops":0,"spliceFails":0,"spliceStarts":0,"spliceEnds":0,"spliceTrims":0,"spliceAborts":0,"preDecision":{"added":0,"request":0,"response":0,"consumed":0,"timeouts":0,"dropped":0}},"acds":{"markers":0,"opportunities":0,"opportunityDrops":{},"requests":0,"responses":0,"msecSum":"0","msecBins":{"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0,"above":0},"altconmux":{"responses":0,"timeouts":0,"msecSum":"0","msecBins":{"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0,"above":0}},"placedPerOp":{},"placements":0,"opAborts":0,"spliceDrops":0,"spliceFails":0,"spliceStarts":0,"spliceEnds":0,"spliceTrims":0,"spliceAborts":0}}},"session":{"hls":{"requests":0,"responses":0,"restoreRequests":0,"restoreResponses":0,"errors":0,"codes":{"0":0},"restoreErrors":0,"restoreCodes":{"1":0},"msecSum":"0","msecBins":{"15000":0,"above":0,"50":0,"100":0,"500":0,"1000":0,"5000":0},"restoreMsecSum":"0","restoreMsecBins":{"5000":0,"15000":0,"above":0,"50":0,"100":0,"500":0,"1000":0},"reconfig":0,"reserveLimit":0},"dash":{"requests":0,"responses":0,"restoreRequests":0,"restoreResponses":0,"errors":0,"codes":{},"restoreErrors":0,"restoreCodes":{},"msecSum":"0","msecBins":{"500":0,"1000":0,"5000":0,"15000":0,"above":0,"50":0,"100":0},"restoreMsecSum":"0","restoreMsecBins":{"above":0,"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0},"reconfig":0,"reserveLimit":0}},"timestamp":{"statsResetTime":1207442342,"nsTimestamp":1982904320}}2024-07-31T01:38:09.930Z [INFO] ContentGenerator {"recordType":"CGStats","statType":"global","workFlow":"","front":{"hlsMaster":{"requests":0,"responses":0,"responseMCHit":0,"responseAwaitingDecision":0,"msecSum":"0","msecBins":{"5000":0,"15000":0,"above":0,"50":0,"100":0,"500":0,"1000":0},"errors":0,"codes":{"404":0,"200":0},"codeCategory":{"6":0,"0":0}},"hlsVariant":{"requests":10,"responses":10,"responseMCHit":0,"responseAwaitingDecision":0,"msecSum":"1208","msecBins":{"50":8,"100":0,"500":2,"1000":0,"5000":0,"15000":0,"above":0},"errors":0,"codes":{"404":0,"504":0,"200":10},"codeCategory":{"19":0,"0":10,"5":0}},"dashMPD":{"requests":0,"responses":0,"responseMCHit":0,"responseAwaitingDecision":0,"msecSum":"0","msecBins":{"500":0,"1000":0,"5000":0,"15000":0,"above":0,"50":0,"100":0},"errors":0,"codes":{"200":0},"codeCategory":{}}},"back":{"origin":{"hlsMaster":{"requests":0,"requestCacheCount":0,"responses":0,"response**bleep**":0,"responsesMiss":0,"responsesHeld":0,"msecSum":"0","msecBins":{"500":0,"1000":0,"5000":0,"15000":0,"above":0,"50":0,"100":0},"errors":0,"codes":{"404":0,"200":0}},"hlsVariant":{"requests":12,"requestCacheCount":12,"responses":12,"response**bleep**":0,"responsesMiss":12,"responsesHeld":0,"msecSum":"201","msecBins":{"500":0,"1000":0,"5000":0,"15000":0,"above":0,"50":12,"100":0},"errors":0,"codes":{"504":0,"200":12,"404":0}},"dashMPD":{"requests":0,"requestCacheCount":0,"responses":0,"response**bleep**":0,"responsesMiss":0,"responsesHeld":0,"msecSum":"0","msecBins":{"500":0,"1000":0,"5000":0,"15000":0,"above":0,"50":0,"100":0},"errors":0,"codes":{"200":0}}},"advert":{"hlsMaster":{"requests":0,"requestCacheCount":0,"responses":0,"response**bleep**":0,"responsesMiss":0,"responsesHeld":0,"msecSum":"0","msecBins":{"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0,"above":0},"errors":0,"codes":{"200":0,"404":0}},"hlsVariant":{"requests":0,"requestCacheCount":0,"responses":0,"response**bleep**":0,"responsesMiss":0,"responsesHeld":0,"msecSum":"0","msecBins":{"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0,"above":0},"errors":0,"codes":{"200":0}},"dashMPD":{"requests":0,"requestCacheCount":0,"responses":0,"response**bleep**":0,"responsesMiss":0,"responsesHeld":0,"msecSum":"0","msecBins":{"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0,"above":0},"errors":0,"codes":{"200":0}}},"altcon":{"hlsMaster":{"requests":0,"requestCacheCount":0,"responses":0,"response**bleep**":0,"responsesMiss":0,"responsesHeld":0,"msecSum":"0","msecBins":{"above":0,"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0},"errors":0,"codes":{"200":0}},"hlsVariant":{"requests":0,"requestCacheCount":0,"responses":0,"response**bleep**":0,"responsesMiss":0,"responsesHeld":0,"msecSum":"0","msecBins":{"above":0,"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0},"errors":0,"codes":{"200":0}},"dashMPD":{"requests":0,"requestCacheCount":0,"responses":0,"response**bleep**":0,"responsesMiss":0,"responsesHeld":0,"msecSum":"0","msecBins":{"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0,"above":0},"errors":0,"codes":{"200":0}}}},"decision":{"hls":{"ads":{"markers":0,"opportunities":0,"opportunityDrops":{"8":0,"0":0,"3":0,"4":0,"5":0},"requests":0,"responses":0,"msecSum":"0","msecBins":{"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0,"above":0},"admux":{"responses":0,"timeouts":0,"msecSum":"0","msecBins":{"500":0,"1000":0,"5000":0,"15000":0,"above":0,"50":0,"100":0}},"placedPerOp":{"0":0,"2":0,"3":0},"placements":0,"opAborts":0,"spliceDrops":0,"spliceFails":0,"spliceStarts":0,"spliceEnds":0,"spliceTrims":0,"spliceAborts":0,"preDecision":{"added":0,"request":0,"response":0,"consumed":0,"timeouts":0,"dropped":0}},"acds":{"markers":0,"opportunities":0,"opportunityDrops":{},"requests":0,"responses":0,"msecSum":"0","msecBins":{"above":0,"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0},"altconmux":{"responses":0,"timeouts":0,"msecSum":"0","msecBins":{"above":0,"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0}},"placedPerOp":{},"placements":0,"opAborts":0,"spliceDrops":0,"spliceFails":0,"spliceStarts":0,"spliceEnds":0,"spliceTrims":0,"spliceAborts":0}},"dash":{"ads":{"markers":0,"opportunities":0,"opportunityDrops":{},"requests":0,"responses":0,"msecSum":"0","msecBins":{"above":0,"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0},"admux":{"responses":0,"timeouts":0,"msecSum":"0","msecBins":{"5000":0,"15000":0,"above":0,"50":0,"100":0,"500":0,"1000":0}},"placedPerOp":{},"placements":0,"opAborts":0,"spliceDrops":0,"spliceFails":0,"spliceStarts":0,"spliceEnds":0,"spliceTrims":0,"spliceAborts":0,"preDecision":{"added":0,"request":0,"response":0,"consumed":0,"timeouts":0,"dropped":0}},"acds":{"markers":0,"opportunities":0,"opportunityDrops":{},"requests":0,"responses":0,"msecSum":"0","msecBins":{"5000":0,"15000":0,"above":0,"50":0,"100":0,"500":0,"1000":0},"altconmux":{"responses":0,"timeouts":0,"msecSum":"0","msecBins":{"500":0,"1000":0,"5000":0,"15000":0,"above":0,"50":0,"100":0}},"placedPerOp":{},"placements":0,"opAborts":0,"spliceDrops":0,"spliceFails":0,"spliceStarts":0,"spliceEnds":0,"spliceTrims":0,"spliceAborts":0}}},"session":{"hls":{"requests":0,"responses":0,"restoreRequests":0,"restoreResponses":0,"errors":0,"codes":{"0":0},"restoreErrors":0,"restoreCodes":{"1":0},"msecSum":"0","msecBins":{"5000":0,"15000":0,"above":0,"50":0,"100":0,"500":0,"1000":0},"restoreMsecSum":"0","restoreMsecBins":{"above":0,"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0},"reconfig":0,"reserveLimit":0},"dash":{"requests":0,"responses":0,"restoreRequests":0,"restoreResponses":0,"errors":0,"codes":{},"restoreErrors":0,"restoreCodes":{},"msecSum":"0","msecBins":{"500":0,"1000":0,"5000":0,"15000":0,"above":0,"50":0,"100":0},"restoreMsecSum":"0","restoreMsecBins":{"500":0,"1000":0,"5000":0,"15000":0,"above":0,"50":0,"100":0},"reconfig":0,"reserveLimit":0}},"timestamp":{"statsResetTime":1207442342,"nsTimestamp":2984280751}}
2024-07-31T01:38:09.931Z [INFO] ContentGenerator {"recordType":"CGHealth","ContentGenerator":{"KnownSessions":1,"WaitingForResponse":0,"PendingDeleteSessions":0,"UnderRecovery":0,"jobQueue":0,"JobsEnqueued":5221688,"JobsDequeued":5221688,"AllocatedSessions":1,"CGStatsSessions":1,"HPIReqs":8,"ManifestCacheObjs":83,"SavedState":29159,"HlsCount":1,"DashCount":0,"HpiReq":346395,"HpiCancel":0,"GitRef":"41d2f857114d10689016ff5074144a580b1ba544","Status":200},"DecisionQueue":{"adReqQueue":{"queuedJobs":658,"dequeuedJobs":658,"lowExceeded":0,"highExceeded":0,"maxQueueDepth":1,"deadline":0,"lowCount":0,"highCount":0,"outstanding":0,"lowWater":250,"highWater":500},"boReqQueue":{"queuedJobs":0,"dequeuedJobs":0,"lowExceeded":0,"highExceeded":0,"maxQueueDepth":0,"deadline":0,"lowCount":0,"highCount":0,"outstanding":0,"lowWater":250,"highWater":500}},"MQMessages":{"Messages":{"1511":2,"1508":22,"1514":352,"704":359,"706":6,"1044":658,"709":372,"9":4693470}}}
2024-07-31T01:39:09.058Z [INFO] ContentGenerator {"recordType":"CGStats","statType":"global","workFlow":"","front":{"hlsMaster":{"requests":0,"responses":0,"responseMCHit":0,"responseAwaitingDecision":0,"msecSum":"0","msecBins":{"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0,"above":0},"errors":0,"codes":{"200":0,"404":0},"codeCategory":{"0":0,"6":0}},"hlsVariant":{"requests":10,"responses":10,"responseMCHit":0,"responseAwaitingDecision":0,"msecSum":"1305","msecBins":{"500":0,"1000":2,"5000":0,"15000":0,"above":0,"50":8,"100":0},"errors":0,"codes":{"504":0,"200":10,"404":0},"codeCategory":{"5":0,"19":0,"0":10}},"dashMPD":{"requests":0,"responses":0,"responseMCHit":0,"responseAwaitingDecision":0,"msecSum":"0","msecBins":{"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0,"above":0},"errors":0,"codes":{"200":0},"codeCategory":{}}},"back":{"origin":{"hlsMaster":{"requests":0,"requestCacheCount":0,"responses":0,"response**bleep**":0,"responsesMiss":0,"responsesHeld":0,"msecSum":"0","msecBins":{"above":0,"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0},"errors":0,"codes":{"404":0,"200":0}},"hlsVariant":{"requests":12,"requestCacheCount":12,"responses":12,"response**bleep**":0,"responsesMiss":12,"responsesHeld":0,"msecSum":"287","msecBins":{"50":12,"100":0,"500":0,"1000":0,"5000":0,"15000":0,"above":0},"errors":0,"codes":{"504":0,"200":12,"404":0}},"dashMPD":{"requests":0,"requestCacheCount":0,"responses":0,"response**bleep**":0,"responsesMiss":0,"responsesHeld":0,"msecSum":"0","msecBins":{"above":0,"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0},"errors":0,"codes":{"200":0}}},"advert":{"hlsMaster":{"requests":0,"requestCacheCount":0,"responses":0,"response**bleep**":0,"responsesMiss":0,"responsesHeld":0,"msecSum":"0","msecBins":{"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0,"above":0},"errors":0,"codes":{"404":0,"200":0}},"hlsVariant":{"requests":0,"requestCacheCount":0,"responses":0,"response**bleep**":0,"responsesMiss":0,"responsesHeld":0,"msecSum":"0","msecBins":{"5000":0,"15000":0,"above":0,"50":0,"100":0,"500":0,"1000":0},"errors":0,"codes":{"200":0}},"dashMPD":{"requests":0,"requestCacheCount":0,"responses":0,"response**bleep**":0,"responsesMiss":0,"responsesHeld":0,"msecSum":"0","msecBins":{"above":0,"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0},"errors":0,"codes":{"200":0}}},"altcon":{"hlsMaster":{"requests":0,"requestCacheCount":0,"responses":0,"response**bleep**":0,"responsesMiss":0,"responsesHeld":0,"msecSum":"0","msecBins":{"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0,"above":0},"errors":0,"codes":{"200":0}},"hlsVariant":{"requests":0,"requestCacheCount":0,"responses":0,"response**bleep**":0,"responsesMiss":0,"responsesHeld":0,"msecSum":"0","msecBins":{"above":0,"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0},"errors":0,"codes":{"200":0}},"dashMPD":{"requests":0,"requestCacheCount":0,"responses":0,"response**bleep**":0,"responsesMiss":0,"responsesHeld":0,"msecSum":"0","msecBins":{"500":0,"1000":0,"5000":0,"15000":0,"above":0,"50":0,"100":0},"errors":0,"codes":{"200":0}}}},"decision":{"hls":{"ads":{"markers":0,"opportunities":0,"opportunityDrops":{"8":0,"0":0,"3":0,"4":0,"5":0},"requests":0,"responses":0,"msecSum":"0","msecBins":{"500":0,"1000":0,"5000":0,"15000":0,"above":0,"50":0,"100":0},"admux":{"responses":0,"timeouts":0,"msecSum":"0","msecBins":{"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0,"above":0}},"placedPerOp":{"0":0,"2":0,"3":0},"placements":0,"opAborts":0,"spliceDrops":0,"spliceFails":0,"spliceStarts":0,"spliceEnds":0,"spliceTrims":0,"spliceAborts":0,"preDecision":{"added":0,"request":0,"response":0,"consumed":0,"timeouts":0,"dropped":0}},"acds":{"markers":0,"opportunities":0,"opportunityDrops":{},"requests":0,"responses":0,"msecSum":"0","msecBins":{"500":0,"1000":0,"5000":0,"15000":0,"above":0,"50":0,"100":0},"altconmux":{"responses":0,"timeouts":0,"msecSum":"0","msecBins":{"above":0,"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0}},"placedPerOp":{},"placements":0,"opAborts":0,"spliceDrops":0,"spliceFails":0,"spliceStarts":0,"spliceEnds":0,"spliceTrims":0,"spliceAborts":0}},"dash":{"ads":{"markers":0,"opportunities":0,"opportunityDrops":{},"requests":0,"responses":0,"msecSum":"0","msecBins":{"above":0,"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0},"admux":{"responses":0,"timeouts":0,"msecSum":"0","msecBins":{"15000":0,"above":0,"50":0,"100":0,"500":0,"1000":0,"5000":0}},"placedPerOp":{},"placements":0,"opAborts":0,"spliceDrops":0,"spliceFails":0,"spliceStarts":0,"spliceEnds":0,"spliceTrims":0,"spliceAborts":0,"preDecision":{"added":0,"request":0,"response":0,"consumed":0,"timeouts":0,"dropped":0}},"acds":{"markers":0,"opportunities":0,"opportunityDrops":{},"requests":0,"responses":0,"msecSum":"0","msecBins":{"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0,"above":0},"altconmux":{"responses":0,"timeouts":0,"msecSum":"0","msecBins":{"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0,"above":0}},"placedPerOp":{},"placements":0,"opAborts":0,"spliceDrops":0,"spliceFails":0,"spliceStarts":0,"spliceEnds":0,"spliceTrims":0,"spliceAborts":0}}},"session":{"hls":{"requests":0,"responses":0,"restoreRequests":0,"restoreResponses":0,"errors":0,"codes":{"0":0},"restoreErrors":0,"restoreCodes":{"1":0},"msecSum":"0","msecBins":{"15000":0,"above":0,"50":0,"100":0,"500":0,"1000":0,"5000":0},"restoreMsecSum":"0","restoreMsecBins":{"5000":0,"15000":0,"above":0,"50":0,"100":0,"500":0,"1000":0},"reconfig":0,"reserveLimit":0},"dash":{"requests":0,"responses":0,"restoreRequests":0,"restoreResponses":0,"errors":0,"codes":{},"restoreErrors":0,"restoreCodes":{},"msecSum":"0","msecBins":{"500":0,"1000":0,"5000":0,"15000":0,"above":0,"50":0,"100":0},"restoreMsecSum":"0","restoreMsecBins":{"above":0,"50":0,"100":0,"500":0,"1000":0,"5000":0,"15000":0},"reconfig":0,"reserveLimit":0}},"timestamp":{"statsResetTime":1207442342,"nsTimestamp":1982904320}}

Expectation:

Event1 : 2024-07-31T01:38:09.930Z [INFO] ContentGenerator

Event 2 : complete json

Re: Line Breaking Issue - text and Json

scelikok — Sun, 03 Nov 2024 19:15:00 GMT

Hi @arunsoni ,

You can try below props;

[your_sourcetype] DATETIME_CONFIG=CURRENT SHOULD_LINEMERGE=false LINE_BREAKER=(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z\s\[\w+\]\s\w+\s)\{ TRUNCATE=20000

Re: Line Breaking Issue - text and Json

PickleRick — Sun, 03 Nov 2024 20:52:26 GMT

Your line breaker will consume the matched data. You'd need to do a non-capturing group. But it's tricky since a line breaker here would need to match two different strings preceeded or followed by two different things. It might be doable, but it's gonna be difficult and ugly.

But there is another issue here of whic h@arunsoni should be aware of.

If you even manage to break your events this way - one of your events will contain a timestamp, the other will not. One will be a valid (I assume) json, the other will be not. Your data will be inconsistent.

Re: Line Breaking Issue - text and Json

arunsoni — Sun, 03 Nov 2024 21:34:09 GMT

@PickleRick . Timestamp will be taken from other event that wont be an issue. That what the requirement and need help on writing regex to match the pattern.

Re: Line Breaking Issue - text and Json

PickleRick — Sun, 03 Nov 2024 21:40:37 GMT

What do you mean by "timestamp will be taken"? Timestamp is either parsed out of the event or assumes to be the time of ingestion (or can be explicitly provided for HEC input).

Re: Line Breaking Issue - text and Json

arunsoni — Sun, 03 Nov 2024 22:02:40 GMT

@PickleRick if there is no timestamp within a log entry then other event which has timestamp will be added to it.

Re: Line Breaking Issue - text and Json

PickleRick — Sun, 03 Nov 2024 22:27:51 GMT

Ok, right you are. The docs are not very good around this - indeed if a timestamp cannot be parsed it will be assumed to be from the previous event, but in your case that would mean you'd have to make sure whole blob gets forwarded to a single downstream (idx or HF).

Re: Line Breaking Issue - text and Json

PickleRick — Sun, 03 Nov 2024 22:40:46 GMT

Yoj can try this as your line breaker

([\r\n](?=\d{4}-\d{2}-\d{2}T)|(?<=ContentGenerator) )

See https://regex101.com/r/gw5YHj/1

Re: Line Breaking Issue - text and Json

richgalloway — Mon, 04 Nov 2024 01:00:19 GMT

This is correct, however, Splunk will log a message every time it copies a timestamp from a previous event. These messages will affect the metrics on the Data Quality dashboard in the Monitoring Console/Cloud Monitoring Console.

Re: Line Breaking Issue - text and Json

scelikok — Mon, 04 Nov 2024 05:51:32 GMT

@PickleRick , you are right about the line breaker, I used a capturing group to keep only JSON messages.

Re: Line Breaking Issue - text and Json

PickleRick — Mon, 04 Nov 2024 07:29:35 GMT

That's true. In fact if the "header" part is constant except for the changing timestamp of course I'd simply SEDCMD it away. Then you'd have a pure json payload, a proper timestamp and no unnecessary "header" bloat in your index.

Re: Line Breaking Issue - text and Json

arunsoni — Mon, 04 Nov 2024 19:23:40 GMT

@scelikok regex is correct but below if it is applied then timestamp wont be there in the event. Splunk will take as current time which completely misleads. I want to have 2 events for a single log entry.

first event should have till 2024-11-04T19:05:46.323Z [INFO] ContentGenerator

second event should have full JSON and even the JSON wont have timestamp in it but first event timestamp is written to this JSON.