Splunk addon for Microsoft Cloud Services event hub input partial log collection issue

[bhsakarchourasi]

Hi All,

we are facing a issue in Splunk Add-on for Microsoft Cloud Services event hub input, there are multiple inputs we have created and almost all the inputs are collecting partial logs. We are checking the count of event at Azure Log Analytics workspace and at the same time checking events on Splunk there is random difference in event collection.

There are no errors in internal logs, although we can see some warning messages, we tried increasing the ingestion pipeline to 4. tried disabling all the inputs but kept only one to check if that's making any issue. 

Splunk deployment is single instance test environment where  32vCPU and 64 GB memory is assigned, storage is more than 800 IOPS. Not much of the application are installed. 

Splunk support case is also opened but till now they haven't able to find any root cause.

Need suggestions and inputs if someone else has faced such issue.

Little back ground on architecture, we have multiple data sources (Azure Activity & AD) sending logs to one event hub and we are segregating the sourcetypes in splunk by transforming data based on category and resourceId. 

Please help to resolve this issue. 

Thanks

Bhaskar