Microsoft Azure Event Hub Pulls - Wrong Offset Error

[guarisma]

I am getting the following error from Azure Event Hub.

2019-12-06 14:57:58,201 ERROR pid=85173 tid=MainThread file=base_modinput.py:log_error:307 | Splunk Error getting event hub data for hub: [EDITED], resource: 0. Detail: The supplied offset '4312319640' is invalid. The last offset in the system is '-1' TrackingId:7c590add-ea50-46c3-833e-89fc1a5c0518_B11, SystemTracker:[EDITED]:eventhub:[EDITED]~8191, Timestamp:2019-12-06T19:57:57
Reference:6bb997ea-840f-42d4-adc5-fda9e70b881b
TrackingId:2453ebda-ee52-4d11-ba55-404531d515f0_B11
SystemTracker:[EDITED]:[EDITED]~8191|$default
Timestamp:2019-12-06T19:57:57 TrackingId:4a775f58b30e4c20a309c4c49b0939b0_G24, SystemTracker:gateway5, > Timestamp:2019-12-06T19:57:57

How can I fix the offset? Why was the last one -1?

I've done some digging at there's a recommendation to blow up the blob so it will get recreated, but this would produce a lot of work if it happens often.

[FabioPeruzzo]

I am facing a similar issue. On my case the Event Hub was recreated in the source (to add more partitions), but even with a new name it is not working. There is any way to "reset" the values in Splunk?

[admsteck]

I'm running into this issue also.  Creating a new Splunk input with the same event hub does not resolve the issue.  Is the Splunk check point unique to the input name, the event hub name, or something else?

Has anyone found a workaround or way to reset the check point that Splunk keeps in it's KV store?

[jconger]

Negative one (-1) is the starting point for an event hub.

It sounds like one of two things happened:

1. An event hub input was created, pulled some events (which set a checkpoint offset of 4312319640), and then deleted. Then, a new input with the same name was created. Check points are stored in the KV store and are not deleted when you delete an input. Therefore, if you create a new input with the same name, the old checkpoint will be retrieved.
2. The retention on your event hub may be really low and the input has not run in a while. I typically think of an event hub as a conveyor belt and the retention factor is how long the belt is. Each event has an offset. If the offset aged out (fell off the belt) before the input was able to retrieve it, you could experience this.

If one of the above sounds familiar, you can delete the input and create a new one with a different name.

[bhsakarchourasi]

Hi @guarisma - Have you resolved this issue, I ran into it and out of 4 partition only getting logs from 3 partitions and loosing 25 percent of logs.

I think @jconger you are correct hence I have removed the input configuration and setup with new name but that didn't resolved my issue. Further, we did the same and configured new event hub with new name in azure than also issue didn't resolved.

Thanks,

Bhaskar

[guarisma]

Was never able to fix this, just did as @jconger and recreated the input.