Using |bucket for events instead of _time?

[SilviaGebel]

Hi, I am currently trying to find a connection between events, that are from different sources and have different timestamps, so they have no relation at all.

My thought is to use the bucket command, to bucket for example, 5 events and see if during that timespan of theses 5 events, an event of a different source is occuring.

Has anyone tried this?

[SilviaGebel · Accepted solution]

I solved the problem by using | filldown . This will give me the last measured value for all time points until the next values was measured. The temperature was only recorded when a change occured, so filldown is perfect!

[Runals]

I'd suggest looking into the transaction command. In that you can combine multiple events into based on a number of different criteria: common fields, max or min timeframe between individual events, define what is the start or stop of an event, etc

[SilviaGebel]

Transaction could help, yes. I would need to have a command like:
combine all events that occur in a certain timespan, but only choose the events closest to fieldX. Every event has one specific field.

[Runals]

Sorry - am taking a closer look at your data now. Assuming you have data in fields named something like Raumtemperatur and Luftfeuchtigkeit I'd start with something like

index=foo | bucket span=5m _time | stats avg(Raumtemperatur) as Raumtemperatur avg(Luftfeuchtigkeit) as Luftfeuchtigkeit by _time

I did 5 minutes and avg because there didn't seem to be data for each minute from your sample and am not sure how granular you want to get. Alternatively you could use max instead - what it comes to is figuring out how you want to handle buckets where there happens to be more than one event. If the primary way to consume the data is visually you can show a timechart where the data points are connected. If the primary way is in a table and you want very granular timespans you will probably need to check out streamstats to fill in the "blanks."

Update: if those fields aren't created throw this in there (assuming I'm looking at this right)

... | rex "Raumtemperatur;.+?,(?<Raumtemperatur>\d+)" | ... rex "Luftfeuchtigkeit;.+?,(?<Luftfeuchtigkeit>\d+)" | bucket ...

[SilviaGebel]

Thank you for your time and help! The fields Raumtemperatur (values: ~23°C in the raw text right after the timestamp) and Luftfeuchtigkeit (~16%) are already extracted.
I want to see what humiditiy (Luftfeuchtigkeit) I have at a certain roomtemperature (Raumtemperatur). So the goal is to have a table which I can then visualize. I want to create a chart, that shows me, what humidity I have at a certain roomtemperatue. (y-axis: humidity; x-axis: roomtemperatur) Or even analyze weather I can predict the humidity by the roomtemperature or vice verca. Kind of like a regression analysis. For that, I have to find some kind of connection between these two attributes.

The problem really is that I cannot connect the attributes by the timestamp because the attributes are measured at random times, independent of each other.

[neelamssantosh]

With bucketing i doubt you can achieve the relation instead of that
Good method is to search by Looking at specific time period for events that happened around the same time can help correlate results and find the root cause(in Date and Time Range with with precise time) while debugging
There must be some id/pid/xx.. for every process figure it out by studying those logs hope that can help you to figure out the relation.

[SilviaGebel]

380462;._PLC.CVI.Z010 [\xE5\xA1C]: aktuelle Raumtemperatur;06.03.2015 15:35;24;\xE5\xA1C;268298
380462;._PLC.CVI.Z010 [\xE5\xA1C]: aktuelle Raumtemperatur;06.03.2015 15:20;23,9;\xE5\xA1C;268298
380462;._PLC.CVI.Z010 [\xE5\xA1C]: aktuelle Raumtemperatur;06.03.2015 15:08;23,8;\xE5\xA1C;268298
380462;._PLC.CVI.Z010 [\xE5\xA1C]: aktuelle Raumtemperatur;06.03.2015 15:01;23,7;\xE5\xA1C;268298
380462;._PLC.CVI.Z010 [\xE5\xA1C]: aktuelle Raumtemperatur;06.03.2015 14:50;23,8;\xE5\xA1C;268298
380462;._PLC.CVI.Z010 [\xE5\xA1C]: aktuelle Raumtemperatur;06.03.2015 14:45;23,7;\xE5\xA1C;268298
380462;._PLC.CVI.Z010 [\xE5\xA1C]: aktuelle Raumtemperatur;06.03.2015 14:43;23,8;\xE5\xA1C;268298
380462;._PLC.CVI.Z010 [\xE5\xA1C]: aktuelle Raumtemperatur;06.03.2015 14:40;23,7;\xE5\xA1C;268298

380461;._PLC.CVI.Z009 [%]: aktuelle Luftfeuchtigkeit;06.03.2015 15:28;16,7;%;4796
380461;._PLC.CVI.Z009 [%]: aktuelle Luftfeuchtigkeit;06.03.2015 15:27;16,6;%;4796
380461;._PLC.CVI.Z009 [%]: aktuelle Luftfeuchtigkeit;06.03.2015 15:26;16,4;%;4796
380461;._PLC.CVI.Z009 [%]: aktuelle Luftfeuchtigkeit;06.03.2015 15:25;16,9;%;4796
380461;._PLC.CVI.Z009 [%]: aktuelle Luftfeuchtigkeit;06.03.2015 15:23;16,7;%;4796
380461;._PLC.CVI.Z009 [%]: aktuelle Luftfeuchtigkeit;06.03.2015 15:22;17,1;%;4796
380461;._PLC.CVI.Z009 [%]: aktuelle Luftfeuchtigkeit;06.03.2015 15:21;16,6;%;4796
380461;._PLC.CVI.Z009 [%]: aktuelle Luftfeuchtigkeit;06.03.2015 15:20;16,8;%;4796

[neelamssantosh]

hope,
Raumtemperatur and Luftfeuchtigkeit keywords can help you to seperate the events.
if you are not sure how many such strings are there use regex to fetch the values..

[SilviaGebel]

Yes, I can use them to seperate the events, but I need to find a way to combine them. I was thinking about commands like |eval coalesce(Raumtemperatur,Luftfeuchtigkeit) in order for the values to be in combined in one event

[SilviaGebel]

unfortunately, I am not looking at logs, but my data are measured data like temperature and humidity, so there are absolutely no common fields in the saved rawdata

[neelamssantosh]

Kindly share couple of your sample log..