All Apps and Add-ons

Microsoft Azure Event Hub Pulls - Wrong Offset Error

Communicator

I am getting the following error from Azure Event Hub.

2019-12-06 14:57:58,201 ERROR pid=85173 tid=MainThread file=base_modinput.py:log_error:307 | Splunk Error getting event hub data for hub: [EDITED], resource: 0. Detail: The supplied offset '4312319640' is invalid. The last offset in the system is '-1' TrackingId:7c590add-ea50-46c3-833e-89fc1a5c0518_B11, SystemTracker:[EDITED]:eventhub:[EDITED]~8191, Timestamp:2019-12-06T19:57:57
Reference:6bb997ea-840f-42d4-adc5-fda9e70b881b
TrackingId:2453ebda-ee52-4d11-ba55-404531d515f0_B11
SystemTracker:[EDITED]:[EDITED]~8191|$default
Timestamp:2019-12-06T19:57:57 TrackingId:4a775f58b30e4c20a309c4c49b0939b0_G24, SystemTracker:gateway5, > Timestamp:2019-12-06T19:57:57

How can I fix the offset? Why was the last one -1?

I've done some digging at there's a recommendation to blow up the blob so it will get recreated, but this would produce a lot of work if it happens often.

0 Karma

Splunk Employee
Splunk Employee

Negative one (-1) is the starting point for an event hub.

It sounds like one of two things happened:

  1. An event hub input was created, pulled some events (which set a checkpoint offset of 4312319640), and then deleted. Then, a new input with the same name was created. Check points are stored in the KV store and are not deleted when you delete an input. Therefore, if you create a new input with the same name, the old checkpoint will be retrieved.
  2. The retention on your event hub may be really low and the input has not run in a while. I typically think of an event hub as a conveyor belt and the retention factor is how long the belt is. Each event has an offset. If the offset aged out (fell off the belt) before the input was able to retrieve it, you could experience this.

If one of the above sounds familiar, you can delete the input and create a new one with a different name.

0 Karma