Is it possible for a search string to pick up instances where the last [say] 3 logs are identical?

[Mick_OBrien]

Hi All

We have a couple of jobs that occasionally loop around same code returning same message/log - is it possible for a search string to pick up instances where the last [say] 3 logs are identical?

Kinds regards

Mick

[Mick_OBrien]

Issue was 'sort' limited to 10,000 rows so replacing with 'sort 0' and I see what I need to see [no missing jobs]

[Mick_OBrien]

Hi All

Issue finally resolved.  This is what I was told - 'sort' command limit is 10,000 and issue was the rather large number of log being returned and the resolution was to replace 'sort' with 'sort 0' which returns all logs [and now I see everything I need to]

[ITWhisperer · Accepted solution]

Assuming the field you are looking at is called message, you could try something like this

| streamstats count reset_on_change=t by message
| where count>2

[Mick_OBrien]

My index has logs for multiple Robot jobs so I added a search before the suggested string....

index=ee_rpa_uipath_platform_* AND OrganizationUnitID IN ($folder$) | sort OrganizationUnitID, RobotName, _time, Message | streamstats count reset_on_change=true by Message | where count > 2 | table OrganizationUnitID, User, RobotName, ProcessName, MachineName, _time, Message | sort -_time

...but now what I am finding is that ONLY one Robot has its logs being displayed once search complete i.e. whilst search is ongoing other logs for other Robots are displayed in panel but then disappear once search finishes.  Any ideas on why these logs for other Robots are removed from search?

I put the suggested search string in my searh

[yuanliu]

Transaction may be your friend.

index=ee_rpa_uipath_platform_* AND OrganizationUnitID IN ($folder$)
```| sort OrganizationUnitID, RobotName, _time, Message```
| eval robotmessage = OrganizationUnitID . ":" . RobotName . ":" . Message
| transaction robotmessage maxevents=3
| where closed_txn=true AND eventcount > 2

About the commented-out sort: because your end goal will always be some kind of tables grouped by OrganizationUnitID and RobotName, there is no point to sort against these two early; if your events come in "naturally", most likely you do not need to sort by _time.

[bowesmana]

So, if your filter criteria is also by org and robot name, then you can add those into the "BY" clause in the streamstats.

You may not need to use sort if you are also splitting by org+robot as the reset_on_change will reset only when org+robot+message changes.

[ITWhisperer]

There are probably being removed by the where command i.e. consecutive messages are not the same, and you are left with occurrences which appear 3 or more times (as requested).