How to combine multiple events into a single event based on identical timestamp?

[human96]

Hi, I am exporting my SAS server but it's splitting one big event to multiple small events with identical timestamp. I want to combine these small events to one event in splunk (index_time/search_time) .

Please refer to the below _raw log.

2021-09-16T14:56:13,979 INFO [00000003] :sas - NOTE: Unable to open SASUSER.PROFILE. WORK.PROFILE will be opened instead.
2021-09-16T14:56:13,980 INFO [00000003] :sas - NOTE: All profile changes will be lost at the end of the session.
2021-09-16T14:56:13,980 INFO [00000003] :sas -
2021-09-16T14:56:14,003 INFO [00000006] :sas -
2021-09-16T14:56:14,003 INFO [00000006] :sas - NOTE: Copyright (c) 2016 by SAS Institute Inc., Cary, NC, USA.
2021-09-16T14:56:14,003 INFO [00000006] :sas - NOTE: SAS (r) Proprietary Software 9.4 (TS1M7)
2021-09-16T14:56:14,003 INFO [00000006] :sas - Licensed to MSF -SI TECH DATA (DMA DEV), Site 70251144.
2021-09-16T14:56:14,003 INFO [00000006] :sas - NOTE: This session is executing on the Linux 3.10.0-1160.83.1.el7.x86_64 (LIN X64) platform.
2021-09-16T14:56:14,003 INFO [00000006] :sas -
2021-09-16T14:56:14,003 INFO [00000006] :sas -
2021-09-16T14:56:14,003 INFO [00000006] :sas -
2021-09-16T14:56:14,003 INFO [00000006] :sas - NOTE: Additional host information:
2021-09-16T14:56:14,003 INFO [00000006] :sas -
2021-09-16T14:56:14,003 INFO [00000006] :sas - Linux LIN X64 3.10.0-1160.83.1.el7.x86_64 #1 SMP Mon Dec 19 10:44:06 UTC 2022 x86_64 Red Hat Enterprise Linux Server release 7.9 (Maipo)
2021-09-16T14:56:14,003 INFO [00000006] :sas -
2021-09-16T14:56:14,006 INFO [00000006] :sas - You are running SAS 9. Some SAS 8 files will be automatically converted
2021-09-16T14:56:14,007 INFO [00000006] :sas - by the V9 engine; others are incompatible. Please see
2021-09-16T14:56:14,007 INFO [00000006] :sas - http://support.sas.com/rnd/migration/planning/platform/64bit.html
2021-09-16T14:56:14,007 INFO [00000006] :sas -
2021-09-16T14:56:14,007 INFO [00000006] :sas - PROC MIGRATE will preserve current SAS file attributes and is
2021-09-16T14:56:14,007 INFO [00000006] :sas - recommended for converting all your SAS libraries from any
2021-09-16T14:56:14,007 INFO [00000006] :sas - SAS 8 release to SAS 9. For details and examples, please see
2021-09-16T14:56:14,007 INFO [00000006] :sas - http://support.sas.com/rnd/migration/index.html
2021-09-16T14:56:14,007 INFO [00000006] :sas -
2021-09-16T14:56:14,007 INFO [00000006] :sas -
2021-09-16T14:56:14,007 INFO [00000006] :sas - This message is contained in the SAS news file, and is presented upon
2021-09-16T14:56:14,007 INFO [00000006] :sas - initialization. Edit the file "news" in the "misc/base" directory to
2021-09-16T14:56:14,007 INFO [00000006] :sas - display site-specific news and information in the program log.
2021-09-16T14:56:14,007 INFO [00000006] :sas - The command line option "-nonews" will prevent this display.
2021-09-16T14:56:14,007 INFO [00000006] :sas -
2021-09-16T14:56:14,007 INFO [00000006] :sas -
2021-09-16T14:56:14,007 INFO [00000006] :sas -
2021-09-16T14:56:14,007 INFO [00000006] :sas -
2021-09-16T14:56:14,008 INFO [00000006] :sas - NOTE: SAS initialization used:
2021-09-16T14:56:14,008 INFO [00000006] :sas - real time 0.19 seconds
2021-09-16T14:56:14,008 INFO [00000006] :sas - cpu time 0.08 seconds
2021-09-16T14:56:14,008 INFO [00000006] :sas -
2021-09-16T14:56:14,331 INFO [00000005] :sas - SAH011001I SAS Metadata Server (8561), State, starting
2021-09-16T14:56:14,362 INFO [00000009] :sas - The maximum number of cluster nodes was set to 8 as a result of the OMA.MAXIMUM_CLUSTER_NODES option.
2021-09-16T14:56:14,362 INFO [00000009] :sas - OMACONFIG option 1 found with value OMA.SASSEC_LOCAL_PW_SAVE and processed.
2021-09-16T14:56:15,160 INFO [00000009] :sas - Using AES with 64-bit salt and 10000 iterations for password storage.
2021-09-16T14:56:15,160 INFO [00000009] :sas - Using SASPROPRIETARY for password fetch.
2021-09-16T14:56:15,160 INFO [00000009] :sas - Using SHA-256 with 64-bit salt and 10000 iterations for password hash.
2021-09-16T14:56:15,169 INFO [00000009] :sas - SAS Metadata Authorization Facility Initialization.
2021-09-16T14:56:15,169 INFO [00000009] :sas - SAS is an adminUser.
2021-09-16T14:56:15,169 INFO [00000009] :sas - SASTRUST@SASPWI is a trustedUser.
2021-09-16T14:56:15,170 INFO [00000009] :sas - SASADM@SASPWI is an unrestricted adminUser.

Thanks in advance.

[PickleRick]

Ugh, that's an ugly beast.

If you have this ingested already, this ship has sailed obviously.

But when you're ingesting that... well, the problem is with finding how to tell one event from another.

The best solution would be to reconfigure the source, if possible to send the data in some more... friendly format.

If you can't do that, there's not much you can do in index time because there's no way of identifying event boundaries (splunk can detect timestamps but cannot do some fancy "compare different timestamps" logic). You could try to write your data to a file and write external scripted/modular input processing this file. Other than that, plain vanilla splunk functionality won't help you much here.

[human96]

Thanks for the quick response @PickleRick. could you please explain more about your best solution?

PickleRick:
The best solution would be to reconfigure the source, if possible to send the data in some more... friendly format.

[PickleRick]

As I said - this solution would be to go to the admins of the solution which is sending those logs to make it send in some more compact form - not so spread across separate lines. Or at least make the logs somehow better delimited. But of course this would only affect newly ingested events, not old ones that are already indexed.

[gcusello]

Hi @human96,

you could use the transaction command, if you want to combine all the contents of all events,

<your_search>
| transaction _time

to avoid a too high fragmentation, you could use the bin command to group timestamps, e.g. to group events with the same minute, avoiding fragmentation for milliseconds, you could try with :

<your_search>
| bin span=1m _time
| transaction _time

otherwise, if you want to display only some fields, you could use the stats command, that's much performant than transaction

<your_search>
| bin span=1m _time
| stats values(*) A * BY _time

eventually defining the fields to display instead of all

Ciao.

Giuseppe

[human96]

Thank you so much for your detailed quick response @gcusello. 

The third column of the raw log ([00000006]) is called Thread identifier, I want to take that value in account.
So, my logic would be that if the Thread identifier and identical time stamp is same then it should be one event.

How can we do that in SPL ? & is it possible to accomplish this in index time extraction ?