All Apps and Add-ons

How to change Splunk Add-On for Microsoft Cloud Services 5000 blob limit

dennywebb
Path Finder

Hello, since 2018 our application has been logging to Azure Storage, in a single container, with "folders" broken down as:

/Year2018/Month04/Day12/Hour15/Minute20/Subscription123/User/456/logtype.log

 

My goal is to pull these logs (json) into Splunk, and so I've set up the Add-On and begun ingesting data... but it kept stopping at 2018... never getting to 20(19/20/21/22).  Investigating why, after quite a bit of tinkering around, I found some internal logs that indicated 

The number of blobs in container [containername] is 5000

 Which... upon further research... is the maximum number of records returned without hitting a movenext marker because of forced paging with the API.

So... I mean I can go edit the python script myself... but is there another way/better way to do this or is a fix for this already in the works?  And, if not and I make the change... is there a github or something I can submit the change to?

Labels (2)
0 Karma

dennywebb
Path Finder

Well, reading through the code... it DOES have a marker set so I'm not sure why this would be failing to return the next "batch" of items until it gets to once that are valid...

0 Karma

dennywebb
Path Finder
0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...