topic Re: Merge Events in Splunk Search

Merge Events

ckarthikin — Tue, 04 Feb 2025 10:34:26 GMT

Hi,

Some of my events doesn't have an timestamp and its has been written as multiple line items in the log.

I want to merge the multiple line items into previous line item

Below are the examples, i want to merge the line items which doesn't have a timestamp, need to add it with previous line

2024-05-24 14:11:51.7212|INFO|Services.Voice.VoiceManager|Wake word detected. hey_mentor
2024-05-24 14:11:51.7212|INFO|Services.Sound.SoundManager|Playing Sound.VoiceStart_TEMP
2024-05-24 14:11:53.9271|INFO|Services.Voice.VoiceManager|Received command
Spoken text: hey mentor turn off L E D
Intent Name: ChangeImageTransformOnOff
Intent Value: turn { OnOff off } { ImageTransformsOnOff L E D }
Slot 1: OnOff=off
Slot 2: ImageTransformsOnOff=L E D
2024-05-24 14:11:53.9271|INFO|NotificationService|Notify: [Illumination set to Off]
2024-05-24 14:11:59.5010|INFO|Services.Voice.VoiceManager|Wake word detected. hey_mentor
2024-05-24 14:11:59.5010|INFO|Services.Sound.SoundManager|Playing Sound.VoiceStart_TEMP
2024-05-24 14:12:01.8935|INFO|Services.Voice.VoiceManager|Received command
Spoken text: hey mentor turn on L E D
Intent Name: ChangeImageTransformOnOff
Intent Value: turn { OnOff on } { ImageTransformsOnOff L E D }
Slot 1: OnOff=on
Slot 2: ImageTransformsOnOff=L E D
2024-05-24 14:12:01.8935|INFO|NotificationService|Notify: [Illumination set to On]
2024-05-24 14:12:01.8935|INFO|Services.Sound.SoundManager|Playing Sound.VoiceStop_TEMP
2024-05-24 14:12:06.7081|INFO|Controls.Live.LiveModel|IsReady=True, Pause <<

-------

Could any please help me how to write a query to achieve this.

Re: Merge Events

gcusello — Tue, 04 Feb 2025 11:41:49 GMT

Hi @ckarthikin ,

where do your logs come from? which technology?

did you used a standard add-on or not?

it seems to be a parsing error.

youshould try adding to your sourcetype SHOULD_LINEMERGE = True, in this way you configure a multiline sourcetype.

Ciao.

Giuseppe

Re: Merge Events

ckarthikin — Tue, 04 Feb 2025 12:01:45 GMT

Hi Giuseppe,

Thank you for the response. Its coming from one of our device and these logs has been already uploaded to splunk and indexed already . now i want to write SPL query to merge the line items

Re: Merge Events

ckarthikin — Tue, 04 Feb 2025 12:02:32 GMT

Hi Giuseppe,

Thank you for the response. Its coming from one of our device and these logs has been already uploaded to splunk and indexed already . now i want to write SPL query to merge the line items

Re: Merge Events

ITWhisperer — Tue, 04 Feb 2025 13:30:19 GMT

The events will have a value for _time. If you sort by this, are these events in the order you showed?

Your events which do have timestamps in the event seem to have the same timestamp for the events either side of the events without timestamps. Is this always the case?

For the events without timestamps, is each line in a different event of multiple events?

Please provide more detail so we can see what needs to be done and work out a solution for you.

Re: Merge Events

ckarthikin — Tue, 04 Feb 2025 14:20:28 GMT

Hi, Thanks for the response.

we don't have _time but we have Time column (Indexed time - it will be same for all events so we cant use Time column). My expectation is without timestamp events need to be merged with previous events using any logic need not save results and it will be used for some calculation and then it will be saved in saved search)

Yes always this is the case for all logs, so i need to write a query to transform this, Please help on this and share your comments

Re: Merge Events

richgalloway — Tue, 04 Feb 2025 15:31:35 GMT

This can be achieved using props.conf. Try these settings to start with

[mysourcetype] ```The "Great Eight" settings``` SHOULD_LINEMERGE = false ```Break lines only between a line ending and a date (year)``` LINE_BREAKER = ([\r\n]+)\d{4} TIME_PREFIX = ^ TIME_FORMAT = %Y-%m-%d %H:%M:%S.%4N MAX_TIMESTAMP_LOOKAHEAD = 30 TRUNCATE = 10000 ```Two settings for UFs``` EVENT_BREAKER_ENABLE = true EVENT_BREAKER = ([\r\n]+)\d{4}

Re: Merge Events

ITWhisperer — Tue, 04 Feb 2025 16:35:21 GMT

Assuming your events follow the pattern shown, you could try something like this

| rex "[^\|]+\|(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{4})\|" | streamstats count(time) as eventnumber | stats values(time) as time list(_raw) as event by eventnumber | eval _time=strptime(time,"%F %T.%4N")

This will also reset the _time timestamp to the same as found in the event data

Re: Merge Events

gcusello — Tue, 04 Feb 2025 17:18:15 GMT

Hi @ckarthikin ,

sorry but the issue is at ingestion level: you have to assign a correctly defined sourcetype (standard or custom) to your data, then you can search your data correctly parsed and aggregated.

so the questions are the ones before:

which technology?

which add-on used for parsing?

if none, you have to create a correct sourcetype and apply it to your data source.

Ciao.

Giuseppe