Keeping track of a state for each event by looking for previous state changes

[Raistlan]

In broad terms, I am searching for a certain event type and figuring out which state things were in for each event, where the state change is signified by other events.

For example, say I have a heartbeat event, and I have "became happy" and "became sad" events. I am trying to determine, at each heartbeat, whether it was happy or sad. I am having a really hard time figuring out how to pull this off.

The main avenue that I have pursued was to try and do a subsearch for the state change events with "latest=" the time of each heartbeat event, but "latest" can only be assigned a literal string. I.e., I've tried something like these two attempts, but they do not work:

event=heartbeat
| eval heartbeatTime=_time
| eval happy=1
| join type=left id [ search event=became.happy OR event=became.sad
                    | where _time<heartbeatTime
                    | eval happy=if(event=="became.happy", 1, 0)
                    | dedup id
                    ]

or

event=heartbeat
| eval heartbeatTime=_time
| eval happy=1
| join type=left id [ search latest=heartbeatTime event=became.happy OR event=became.sad
                    | eval happy=if(event=="became.happy", 1, 0)
                    | dedup id
                    ]

[martin_mueller · Accepted solution]

Does this produce a sample data set matching your question?

| stats count | eval count = 1 | eval event = "became.happy heartbeat heartbeat became.sad heartbeat became.happy became.sad heartbeat became.happy heartbeat" | makemv event | mvexpand event | accum count | eval _time = now()-count | fields - count

If so, append this to calculate a state field:

... | eval state=case(match(event, "^became"), replace(event, "became.", "")) | filldown state

The producing search before that would be this:

event=heartbeat OR event=became.happy OR event=became.sad

[somesoni2]

You can use "|sort count=0 fieldname" to eliminate 10000 limit.

[Raistlan]

"sort" restricts the number of events down to 10,000, so I did a "reverse"; it's easier and doesn't trim the events.

[martin_mueller]

Looking at it again, you may need to sort by time before the filldown to get the events after a state change affected by that very change rather than the events before.

[Raistlan]

I will try this out today; the filldown seems to be the missing piece; I can then filter out those state change events after the filldown.