Microsoft Azure Event Hub Pulls - Wrong Offset Error

[guarisma]

I am getting the following error from Azure Event Hub.

2019-12-06 14:57:58,201 ERROR pid=85173 tid=MainThread file=base_modinput.py:log_error:307 | Splunk Error getting event hub data for hub: [EDITED], resource: 0. Detail: The supplied offset '4312319640' is invalid. The last offset in the system is '-1' TrackingId:7c590add-ea50-46c3-833e-89fc1a5c0518_B11, SystemTracker:[EDITED]:eventhub:[EDITED]~8191, Timestamp:2019-12-06T19:57:57
Reference:6bb997ea-840f-42d4-adc5-fda9e70b881b
TrackingId:2453ebda-ee52-4d11-ba55-404531d515f0_B11
SystemTracker:[EDITED]:[EDITED]~8191|$default
Timestamp:2019-12-06T19:57:57 TrackingId:4a775f58b30e4c20a309c4c49b0939b0_G24, SystemTracker:gateway5, > Timestamp:2019-12-06T19:57:57

How can I fix the offset? Why was the last one -1?

I've done some digging at there's a recommendation to blow up the blob so it will get recreated, but this would produce a lot of work if it happens often.

[FabioPeruzzo]

I am facing a similar issue. On my case the Event Hub was recreated in the source (to add more partitions), but even with a new name it is not working. There is any way to "reset" the values in Splunk?

[admsteck]

I'm running into this issue also.  Creating a new Splunk input with the same event hub does not resolve the issue.  Is the Splunk check point unique to the input name, the event hub name, or something else?

Has anyone found a workaround or way to reset the check point that Splunk keeps in it's KV store?

[jconger]

Negative one (-1) is the starting point for an event hub.

It sounds like one of two things happened:

1. An event hub input was created, pulled some events (which set a checkpoint offset of 4312319640), and then deleted. Then, a new input with the same name was created. Check points are stored in the KV store and are not deleted when you delete an input. Therefore, if you create a new input with the same name, the old checkpoint will be retrieved.
2. The retention on your event hub may be really low and the input has not run in a while. I typically think of an event hub as a conveyor belt and the retention factor is how long the belt is. Each event has an offset. If the offset aged out (fell off the belt) before the input was able to retrieve it, you could experience this.

If one of the above sounds familiar, you can delete the input and create a new one with a different name.

[cornemrc]

We have the same issue here after deleting and recreating an eventhub, no events are streamed due to the offset error.

Workaround with re-creating the input with a differet name does not help. I think the reason here is because the original question was about the Azure-TA but we are using the MSCS App now, as eventhub is no longer supported within the Azure-TA.

I have found the kvstore for the Azure-TA but none for the MSCS App.

So where is the MSCS app storing the offset value to edit them?

[cornemrc]

I have just found a solution for the eventhub offset issue within the MSCS app. 

Just deactivate the modular input, then go to 

Splunk\var\lib\splunk\modinputs\mscs_azure_event_hub

and delete the according file

[eventhubnamespace]-[eventhub]-$Default.v1.ckpt

 and reactivate the input.

Splunk will recreate the file with a corrected timestamp and will reload the missing events from the eventhub.

[bhsakarchourasi]

Hi @guarisma - Have you resolved this issue, I ran into it and out of 4 partition only getting logs from 3 partitions and loosing 25 percent of logs.

I think @jconger you are correct hence I have removed the input configuration and setup with new name but that didn't resolved my issue. Further, we did the same and configured new event hub with new name in azure than also issue didn't resolved.

Thanks,

Bhaskar

[guarisma]

Was never able to fix this, just did as @jconger and recreated the input.