About Ethil

Ethil · ‎12-03-2024

@PickleRick , Sorry for the late answer, you are rigth, i think we might have misunderstood how some attributes work in the indexes.conf and thus it was not strong enough to force the rolling of the warm buckets. We will surely rework the conf and see what happens but i think that was the main issue. Thanks a lot for your time and answers !

Ethil · ‎11-27-2024

@PickleRick Ok i see, and yes currentDBSizeMB does correspond to the actual size of the index on the disk.

Ethil · ‎11-27-2024

@PickleRick Thanks again. After reading again your past message and checking with my team, the index saturating in question is a default one so i updated my post with all informations i could give -> _metrics. 1) Checked, no others than _metrics are pointing on the $SPLUNK_DB/_metrics/ directory 2) Warm and Hot buckets only. What are DAS data ? I don't know about what you said last, but that might have cause the issue.

Ethil · ‎11-27-2024

@PickleRick , Okay thanks for your answer, i did check both "| rest /data/indexes/myindex" and btool as you mentioned and both have maxTotalDataSizeMB to 5000 (5GB). I can't check through the GUI "Settings->Indexes" but i guess it's not that important.

Ethil · ‎11-26-2024

Hi @isoutamo , Thanks for your input, but that's not the issue there, i already did clean my saturated index and restarted the indexer and it works fine now. And as I said to @richgalloway , in my post i stated that only one of my indexes was taking way more space than it should and i know which one. The issue is why it did exceed the maxTotalDataSizeMB set in the indexes.conf ? Just adding more space might not be the right solution for us, but i keep in mind the whole thing around using volumes for a better planning of the data storage, thanks.

Ethil · ‎11-26-2024

@richgalloway , Maybe my post was not clear enough sorry, i did state that one of my index on the partition (and i already know which one, the one i gave in the indexes.conf) is saturated with warm buckets (db_*) and taking all the space available, even though it's configurate as shown in the indexes.conf. Of course multiple indexes are using the disk, but only one went highly above the maxTotalDataSizeMB and saturated it.

Ethil · ‎11-26-2024

I did not, as said above in my post, i'm very new to the subject and i asked how to check if the conf was taken into account. Thanks for telling me how, i did check and splunk does seem to take the default conf written.

Ethil · ‎11-26-2024

Thanks for your input ! Your explanations were clear but it does not explain how/why my index did not roll the buckets after reaching the maxTotalDataSizeMB of 5GB and went up to 35GB.

Ethil · ‎11-26-2024

Ok, but the indexes are all set with a maxTotalDataSIze of 5GB (default set up written in my indexes.conf), which from what i understood should have stop each indexes, individually, exceeding this size and force the older warm buckets to cold to avoid saturation. The doc : https://docs.splunk.com/Documentation/Splunk/9.3.2/Admin/Indexesconf maxTotalDataSizeMB = <nonnegative integer> * The maximum size of an index, in megabytes. * If an index grows larger than the maximum size, splunkd freezes the oldest data in the index. * This setting applies only to hot, warm, and cold buckets. It does not apply to thawed buckets. ... However the saturation dit happen with one of them, that is the issue i don't understand. My disk is 40GB, and the saturation of this specific index reached 35GB and thus reached the minimum disk space and thus failed my indexer. The rolling criteria was met, why didn't it rolled the buckets ?

Ethil · ‎11-26-2024

Ok thanks, i get this part, i'll try to rework the indexes.conf. But what i still don't get, and i really would like to know (it's quite important for me to know what was wrong before changing anything) is why it didn't work in the first place ? From what i read in the doc it should have work with a simple conf like this no ? Furthermore, using a Volume and maxVolumeDataSizeMB will help me monitor the global size of all indexes on my volume right ? But i need each indexes to possibly have a specific maxTotalDataSIzeMB and abide by it. If it's not possible or limited (because of whatever reason) feel free to tell me. Thanks again !

Ethil · ‎11-25-2024

Hi @richgalloway , thanks for your input, yes i only gave the configuration for one index because i mainly rely on the default conf written above for all my indexes on the disk, plus this specific index was the only one saturated, thus probably the issue here ? (please correct me if i'm wrong in this statement) For the volumes, i have one in my conf, but i'm not sure how it works and how it's used (i didn't write this conf file myself), i'll try to look into this subject. [volume:MyVolume] path = $SPLUNK_DB Thanks !

Ethil · ‎11-25-2024

[UPDATE] Hello everyone, and thanks in advance for your help. I'm very new to this subject so if anything is unclear, i'll try to explain my problem more in details. I'm using spunk 9.2.1, and i recently observed that my indexer was not indexing logs received. The indexer is in a failure state because my partition $SPLUNK_DB reached the minFreeSpace allowed in server.conf. After further analysis it seems that one of the index _metrics on the partition is saturated with warm buckets (db_*) and taking all the space available. I however have configured all my indexes with the indexes.conf ($SPLUNK_HOME/etc/system/default/indexes.conf) # index specific defaults maxTotalDataSizeMB = 5000 maxDataSize = 1000 maxMemMB = 5 maxGlobalRawDataSizeMB = 0 maxGlobalDataSizeMB = 0 rotatePeriodInSecs = 30 maxHotIdleSecs = 432000 maxHotSpanSecs = 7776000 maxHotBuckets = auto maxWarmDBCount = 300 frozenTimePeriodInSecs = 188697600 ... # there's more but i might not be able to disclose them or it might not be revelant [_metrics] coldPath = $SPLUNK_DB/_metrics/colddb homePath = $SPLUNK_DB/_metrics/db thawedPath = $SPLUNK_DB/_metrics/thaweddb frozenTimePeriodInSecs = 1209600 From what i understand with this conf applied the index should not exceed 5GB, and when reached the warm/hot buckets should be removed, but it seems that's it's not taken into account in my case. The indexer work fine after purging the buckets and restarting it, but i don't get why the conf was not applied ? Is there something i didn't get here ? Is there a way to check the "characteristics" of my index once started ? -> Checked, the conf is correctly applied. If you know anything on this subject please help me 🙂 thank you

Ethil · ‎11-15-2024

Thanks a lot for your help !

Ethil · ‎10-21-2024

Hi @tscroggins , Thanks for your reply, then do you perhaps know if they're any time-range args that work with input-dashboard ? Otherwise, should i use another method ?

Ethil · ‎10-17-2024

Hello everyone, and thanks in advance for your help. I'm very new to this subject so if anything is unclear, i'll try to explain my problem more in details. I'm using spunk 9.2.1, and i'm trying to generate a PDF from one of my dashboard on the last 24 hours, using a splunk API call. I'm using a POST request to the ".../services/pdfgen/render" endpoint. First I couldn't find any documentation on this matter. Furthermore, even when looking at $SPLUNK_HOME/lib/python3.7/sites-packages/splunk/pdf/pdfgen_*.py (endpoint,views,search,utils) i could'nt really understand what arguments to use to ask for the last 24 hours data. I know it should be possible because it is doable on the splunk GUI, where you can choose a time range and render according to it. I saw something looking like time range args : et and lt, which should be earliest time and latest time, but i don't know what type of time data it is expecting an trying random things didn't get me anywhere. If you know anything on this subject please help me 🙂 thank you

Ethil · ‎10-17-2024

Yeah i know the problem was quite specific, sorry for the late answer and thanks for your help. I was able to determine what failed, the GET was actually supposed to be a POST request. I don't really know why but one splunk error message said that the GET is outdated for pdfgen. Anyway thanks again.

Ethil · ‎08-07-2024

Hello everyone, and thanks in advance for your help. I'm very new to this subject so if anything is unclear, i'll try to explain my problem more in details. I'm using spunk 9.2.1, and i'm trying to generate a PDF from one of my dashboard, using a splunk API call. From what i saw online, i should use : GET /services/pdfgen/render?input-dashboard=<DashboardName>&namespace=<AppName>&paper-size=a4-landscape user = MyUser; app = MyApp; dashboard = Security_events_dashboard (i'm using a module that calls the API for me, all i do is precise the endpoint and the parameters and it gives me the response as a string) The problem is that i get this error : Unable to render PDF.<br/><ul><il>Bailing out of Integrated PDF Generation. Exception raised while preparing to render "Untitled" to PDF. [HTTP 404] https://localhost:8089/servicesNS/MyUser/MyApp/data/ui/views/Security_events_dashboard; [{'type': 'ERROR', 'code': None, 'text': 'Could not find object id=Security_events_dashboard'}]</li></ul> On th GUI, signed as MyUser, i can see the dashboard under MyApp, and the permission is set to read for MyUser, Owner = nobody, Sharing = App. To confirm this, on my Search-Head VM i can see the dashboard under $SPLUNK_HOME/etc/apps/MyApp/default/data/ui/views/security_events_dashboard.xml. Plus in $SPLUNK_HOME/etc/apps/MyApp/metadata/default.meta : [views/security_events_dashboard.xml] access = read : [MyUser], write : [admin] owner = nobody version = 9.1.0.1 export = system I've tried using the dashboard name as security_events_dashboard (instead of Security_events_dashboard) but i get the same error. I don't see what i'm missing here, si if anyone could give me hint or two please, thank you

Posts	17
Solutions	1
Karma Given	16
Karma Received	5
Member Since	‎08-07-2024

Online Status	Offline
Date Last Visited	2 weeks ago

Indexer saturation

Render Dashboard to PDF with time range - REST API

Unable to render Dashboard to PDF with REST API

Re: Indexer saturation

Re: Indexer saturation

Re: Indexer saturation

Re: Indexer saturation

Re: Indexer saturation

Re: Indexer saturation

Re: Indexer saturation

Re: Indexer saturation

Re: Indexer saturation

Re: Indexer saturation

Re: Indexer saturation

Indexer saturation

Re: Render Dashboard to PDF with time range - REST...

Re: Render Dashboard to PDF with time range - REST...

Render Dashboard to PDF with time range - REST API

Re: Unable to render Dashboard to PDF with REST AP...

Unable to render Dashboard to PDF with REST API

Join the Conversation