All Apps and Add-ons

How to change Splunk Add-On for Microsoft Cloud Services 5000 blob limit

dennywebb
Path Finder

Hello, since 2018 our application has been logging to Azure Storage, in a single container, with "folders" broken down as:

/Year2018/Month04/Day12/Hour15/Minute20/Subscription123/User/456/logtype.log

 

My goal is to pull these logs (json) into Splunk, and so I've set up the Add-On and begun ingesting data... but it kept stopping at 2018... never getting to 20(19/20/21/22).  Investigating why, after quite a bit of tinkering around, I found some internal logs that indicated 

The number of blobs in container [containername] is 5000

 Which... upon further research... is the maximum number of records returned without hitting a movenext marker because of forced paging with the API.

So... I mean I can go edit the python script myself... but is there another way/better way to do this or is a fix for this already in the works?  And, if not and I make the change... is there a github or something I can submit the change to?

Labels (2)
0 Karma

dennywebb
Path Finder

Well, reading through the code... it DOES have a marker set so I'm not sure why this would be failing to return the next "batch" of items until it gets to once that are valid...

0 Karma

dennywebb
Path Finder
0 Karma
Get Updates on the Splunk Community!

Developer Spotlight with Brett Adams

In our third Spotlight feature, we're excited to shine a light on Brett—a Splunk consultant, innovative ...

Index This | What can you do to make 55,555 equal 500?

April 2025 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Say goodbye to manually analyzing phishing and malware threats with Splunk Attack ...

In today’s evolving threat landscape, we understand you’re constantly bombarded with phishing and malware ...