All Apps and Add-ons

How to change Splunk Add-On for Microsoft Cloud Services 5000 blob limit

dennywebb
Path Finder

Hello, since 2018 our application has been logging to Azure Storage, in a single container, with "folders" broken down as:

/Year2018/Month04/Day12/Hour15/Minute20/Subscription123/User/456/logtype.log

 

My goal is to pull these logs (json) into Splunk, and so I've set up the Add-On and begun ingesting data... but it kept stopping at 2018... never getting to 20(19/20/21/22).  Investigating why, after quite a bit of tinkering around, I found some internal logs that indicated 

The number of blobs in container [containername] is 5000

 Which... upon further research... is the maximum number of records returned without hitting a movenext marker because of forced paging with the API.

So... I mean I can go edit the python script myself... but is there another way/better way to do this or is a fix for this already in the works?  And, if not and I make the change... is there a github or something I can submit the change to?

Labels (2)
0 Karma

dennywebb
Path Finder

Well, reading through the code... it DOES have a marker set so I'm not sure why this would be failing to return the next "batch" of items until it gets to once that are valid...

0 Karma

dennywebb
Path Finder
0 Karma
Get Updates on the Splunk Community!

Earn a $35 Gift Card for Answering our Splunk Admins & App Developer Survey

Survey for Splunk Admins and App Developers is open now! | Earn a $35 gift card!      Hello there,  Splunk ...

Continuing Innovation & New Integrations Unlock Full Stack Observability For Your ...

You’ve probably heard the latest about AppDynamics joining the Splunk Observability portfolio, deepening our ...

Monitoring Amazon Elastic Kubernetes Service (EKS)

As we’ve seen, integrating Kubernetes environments with Splunk Observability Cloud is a quick and easy way to ...