About verbal_666

verbal_666

Don't know what you want to say, but it's the same. Keeping only the "splunk.secret" and remove all the others and reconstruct the new auth path as new.

verbal_666

/opt/splunk/bin/splunk stop cd /opt/splunk/etc/auth rm -fr $(ls -1|egrep -v "^splunk.secret") cd /opt tar -xf splunk-9.4.5-8fb2a6c586a5-linux-amd64.tgz /opt/splunk/bin/splunk restart --accept-license --answer-yes /opt/splunk/bin/splunk show kvstore-status --verbose ... there was a "cd" missing on line #2, obviously 🙂

verbal_666 · ‎06-01-2026

- Install Splunk 9.0 on the new server - Install Splunk 9.1 on the new server - Install Splunk 9.2.x on the new server [then]

verbal_666 · ‎05-05-2026

For my experience you have to dump all Splunk paths with a simple tar (use J or j or z compression as you want). tar -czf splunk_$HOSTNAME.tgz (<splunk_path> es./opt/splunk) +IDXs: move/compress Indexers "external" db paths to new Indexers to keep dbs if out of the <splunk_path=var/lib/splunk> ... i have many dbs in outer fs, like lv /splunkdb/ ... move/tar all "/splunkdb" to new machines... for every node... move tgz to new machines... change & fix conf in .../etc/system/local/ for new hostnames, ips, etc... change any outputs.conf you use to use new Indexers. Check also Deployment if contains some outputs.conf or fixed call to indexers or other piece of old infra, and update it for new infra. Also UFs need to be checked, if the point to fixed DS, any point to DS must be changed with new DS hostname/ip. When you have fixed all confs (also peers in SearchHeads) to new Infrastructure, launch a new run for every node, and you should have you env ready as before 👍 Best way is to change OS, but keep all hostnames/ip, so you don't need to change point call in confs, but use the old ones.

verbal_666 · ‎04-10-2026

So, i think we need a new issue to SPLUNK DEVS to check the bug again 🤕 Meanwhile, i leave workaround on my nodes in 9.3.11 🤧

verbal_666 · ‎04-09-2026

Did you clean all caches around (load balancers, proxies, browsers, etc...)? I updated 9.3.9 to 9.3.11, but i left workaround there. Later i'll try removing it from a test environment.

verbal_666 · ‎02-26-2026

Mmm..... tomorrow, after 24h the issue raised again... i have SH UI delayed!!! Also 60 seconds to open a /report page... yesterday after applying the fix it was not so... I think we need a final patch and fix with new version for 9.3 / 9.4 / 10. 😎

verbal_666 · ‎02-25-2026

Many many thanks again for resolving the issue. I did the same analysis, and found all those CDN calls with errors, but couldn't figure out how to solve it! https://community.splunk.com/t5/Splunk-Enterprise/9-3-6-to-9-3-9-frequently-WebUI-timeout/td-p/758767/jump-to/first-unread-message Great 👍👍👍 PS. quite annoying SPLUNK introduces this really strange bug with 503/delays/timeouts for users 😫

verbal_666 · ‎02-25-2026

Tried the workaround... seems to fix the problem, systems are more responsive 👍👍👍 So it's a already documented bug 😎 👎 Many thanks 👍👍👍

verbal_666 · ‎02-25-2026

Yep! SPL hosts can't reach internet. I'll try the workaround 👍👍👍 Anyway, before upgrading, system was 100% stable... maybe 2 hours about of services down have created a massive queuing on Indexers? I see they are receiving many many events from about 2000 UFs... i have an indicator that in average system pressure is about "12500", after updating it count "20000" and more... i thing it's queues delaying, that delays all infrastructure.... maybe it's a temporary delay for service interruption for upgrading to 9.3.9 🤔 I'll monitor 🤕 Thanks 👍👍👍

verbal_666 · ‎02-24-2026

Hello. I recently updgraded from 9.3.6 to 9.3.9 all SPLUNK machines. With 9.3.6 i did not notice nothing wrong. Now in 9.3.9 i found many timeouts in WebIF, and many 502/503 errors from UI, taking to the terrific horse page... i need to raise in web.conf all timeout configs, the "horse" went away, but UI is extremely slow, sometimes i need to refresh blank pages (F5) to arrive to target destination 😣 Log example, ERROR [699e902bc17fd574134fa0] decorators:421 - Splunkd daemon is not responding: ('Error connecting to /services/apps/local: The read operation timed out',) Traceback (most recent call last): File "/opt/splunk/lib/python3.9/site-packages/splunk/rest/__init__.py", line 609, in simpleRequest serverResponse, serverContent = h.request(uri, method, headers=headers, body=payload) File "/opt/splunk/lib/python3.9/site-packages/httplib2/__init__.py", line 1709, in request Mostra tutte le 47 righe host = myhost source = /opt/splunk/var/log/splunk/web_service.log sourcetype = splunk_web_service ERROR [699e902bc17fd574134fa0] __init__:620 - Socket error communicating with splunkd (error=The read operation timed out), path = /services/apps/local host = myhost source = /opt/splunk/var/log/splunk/web_service.log sourcetype = splunk_web_service Before, 1 click and in 1 second i went to my target UI page, now i need to wait also 60 seconds, and need F5 frequently to update a blank page... What's wrong? 😤 Running on Linux Red Hat 8.10 (Ootpa) witj kernel 4.18. Thanks.

verbal_666 · ‎02-18-2026

@bowesmana wrote: Splunks logic for interpreting earliest and latest is that the range of data returned is >= earliest AND < latest so the behaviour is correct in that it will find all data UP TO, BUT NOT INCLUDING the latest time, so your latest should always be greater than the last event you want to find. This is the final answer 👍👍👍 Maybe i was not really clear, i know there's not SPL SYNTAX different than eraliest/latest "=" (as also editor says, without highlighting the syntax "<=" or ">=" as they are treated as extracted _raw fields), but in my mind the logic was, >= earliest AND <= latest with SPL converting itsself text "DAY:23:59:59" to last millisecond of the day. THAT'S NOT!!! 👍👍👍 The real logic was what you write, >= earliest AND < latest Anb 23:59:59 it's not the END.OF.DAY 👍👍👍since latest is <23:59:59.000, so 23:59:59.000 to .999 was skipped 👍👍👍 Maybe could be highlighted written in the officicial tutorial page how SPLUNK THINKS 😉😉😉 PS: as i said, we can read in many answers, tutorials, discussions, that describes ":23:59:59" as E.O.D. THAT'S NOT 🤷‍♀️ Thanks 👍👍👍

verbal_666 · ‎02-17-2026

Mmm, not really, Ultimately you are setting latest < 23:59:59 - and something like 23:59:59.124 is greater than it, therefore it wouldnt be incluced. I set also search ... earliest>="12/01/2025:00:00:00" latest<="12/01/2025:23:59:59" but the result was the same... also if, teorically, <="DAY:23:59:59" should takes 12/01/2025 23:59:59.558 This is a (558) ms event!!!! And not 12/02/2025 00:00:00.000 This is NOT a (000) ms event!!!! IMHO it's a dangerous behaviour!!! Since in different tutorials or anwers i read that the SPL search ... earliest="12/01/2025:00:00:00" latest="12/01/2025:23:59:59" should export and entire day, but it's not really true. Now, i have many scripts that export data using API, and couldn't understand why i didn't have since today all data for a single day, and the anwer was that today i found that latest="SAMEDAY:23:59:59" was technically wrong and i needed to correct the script to launch eraliest="MYDAY:00:00:00" and earliest="NEXTDAY:00:00:00" 🤧 Anyway, it's not really mentioned in official documentation, https://help.splunk.com/en/splunk-enterprise/search/spl-search-reference/9.3/time-format-variables-and-modifiers/time-modifiers ... so... i think this is the official and correct operation. Only, just wait a moment if someone says latest to 23:59:59 was an entire day, it's not true 🤕 Thanks 👍

verbal_666 · ‎02-17-2026

Hello. I found out that running a search to find events from 00:00:00 to 23:59:59, when i want H24 all events, using latest at 23:59:59 skips events with _time at 23:59:59.xxx, the end is exactly "23:59:58.999". Example, search index=myindex host=myhost source=mysource earliest="12/01/2025:00:00:00" latest="12/01/2025:23:59:59" All events with timestamp 23:59:59 are skipped 😐 i get only to 23:59:58.999 😫 Remediation is to extract 'till next day at 00:00:00, search index=myindex host=myhost source=mysource earliest="12/01/2025:00:00:00" latest="12/02/2025:00:00:00" Is that a bug, normal, or what? 🤔 Thanks.

verbal_666 · ‎02-03-2026

https://advisory.splunk.com/advisories/SVD-2026-0101

verbal_666 · ‎01-20-2026

Can someone, please, open a case to SPLUNK for official communication? I get error after sending a NEW CASE MONGODB Vulnerability CVE-2025-14847 https://www.mongodb.com/company/blog/news/mongodb-server-security-update-december-2025 SPLUNK ENTERPRISE 9.3.X 9.4.X 10.X.X /opt/splunk/bin/splunk cmd mongod -version SECURITY ISSUE

verbal_666 · ‎01-19-2026

👍👍👍

verbal_666 · ‎01-19-2026

I know it, only SPLUNK instance can access mongodb from localhost only and our SPLUNK it's not exposed outside, only in Intranet. But my Company wants a solution or a valid explanation to exclude vulnerabilities on the instances 😥 Maybe a formal official SPLUNK communication could help! 😎 Thanks 👍👍👍

verbal_666 · ‎01-19-2026

Hello. Recently a critical vulnerability was found in ZLIB of MongoDB. https://www.cyber.gov.au/about-us/view-all-content/alerts-and-advisories/vulnerability-in-mongodb-product-mongodb-server-leak Is there a way resolve this internal issue? Thanks.

verbal_666 · ‎12-05-2025

Gosh! I think i did a mistake for Question#1. The credentials/keys are stored in every searchpeer path dir "$SPLUNK_HOME/etc/auth/distServerKeys/search_peer#x/", inside where you can find all the kets... i did confusion with var 🤧 And the files the user deleted, since i need a full reinstall, were at fact inside $SPLUNK_HOME/etc/ (the user removed for an error ☹️ 🙁 all the etc in $SPLUNK_HOME). So, there's no mistery at all. He deleted all previous recorded keys the peer stored when called from SearchHead. So it really needed to recreate them, they were gone 😖 Sorry, problem solved!!! My fault... 😑

verbal_666 · ‎12-05-2025

Oh, really thanks. I was worried about this behavious. Luckily we have daily backups. Next we create a custom app to deploy them to 👍👍👍i really miss this security feature 👍👍👍

verbal_666 · ‎12-05-2025

Hi. QUESTION #1: search peer login credentials In previous versions, i'm talking about v7 and/or v8, in my memory, login credential to search peers were stored into filesystem, inside /var/... in some files named search_peer#1.something, search_peer#2.something etc.. Now i upgraded to 9.4 version, and can't find them anymore. Where are credentials stored? In KVSTORE? I ask this question since today i need to untar a 9.4 version over a same 9.4 version to take some files a user deleted for error🤦‍♂️and after doing it, it was a searchpeer/indexer, on the SearchHead and Deployer (Management Console) the peer was DOWN, i needed to reinsert the login credential again as first time to make it UP & Healthy again. Is it for security or is it a bug? I remember i need to do it also months ago when we upgraded 9.3 to 9.4!!! After upgrading, all peer credentials was gone. I thought it was a temporary bug, but now it seems to be the normal behaviour: reinstalling an instance, over the same, removes all previous registered credentials for searchpeers? QUESTION #2: Splunk upgrading removes all $SPLUNK_HOME/bin/scripts/ And, also, after upgrading, all out custom scripts in $SPLUNK_HOME/bin/scripts/ was totally deleted, SPLUNK UPGRADER remove all of them, i found only the "readme.txt", we had something like 50 custom scripts, all gone 🙄 obviously we have a backup!!! But is it normal??? Thanks.

verbal_666 · ‎12-04-2025

It's the main application that in some cases gets crazy and start writing so many error headers ☹️ i still talk to developers and tell this is the why, and in case the manual how-to to unlock logs 🤧 There is a limit to everything 😂

verbal_666 · ‎12-03-2025

The logs posted in main thread were totally a test, to simplify the real problem. The issue came out with a user who can't read his logs one day. And the problem was that for the same 2 days the Application at 00:00 starts writing on 2 nodes the same identical log errors for about 4k, without any timestamp at all, not even a dot was different from day 1 to day 2 from the first default 256 initCrcLen 😎 first time it happened, months ago, i raised initCrcLen to 1024 and injection went ok, but last day i discovered that the "header" errors were about 4k, so, the 3rd day the logs were stuck, since UF thought it was the same identical file already injected! But it weren't since from 10k about Apps start to write json logs with correct timestamps. In this single case i raise again initCrcLen over the 4k and all was ok. Next days, i can have a 10k identical error header over the same logs. So... in this special case, if issue still come out, i will raise again initCrcLen to maximum and tell user to fix the problem with a simple wortaround: sed -i "1i$(date) $RANDOM" /path/lo/log And... that's all. SPLUNK can do much, but not all 😆 Anyway, it could be useful in future UF versioning developments, a new inputs.conf parameter in stanza to use OS file timestamp CRC as crcSalt, something like, crcSalt = <FILE_TIMESTAMP> or a salt with present date/time, crcSalt = <NOW> 👍👍👍 What do you think about it?

verbal_666 · ‎12-03-2025

Anyway the only solution was to raise the initCrcLength to a high value over the header 👍

Posts	252
Solutions	23
Karma Given	75
Karma Received	28
Member Since	‎03-15-2017

Online Status	Offline
Date Last Visited	2 weeks ago

9.3.6 to 9.3.9 ... frequently WebUI timeout!

Earliest... Latest at 23:59:59 skipped!

MONGODB vulnerability in ZLIB

Where are login credentials to peers stored? And w...

Let's think about an unique way to always and alwa...

Easily find the conf of an extracted field

Deployment Server issue after upgrading 9.1 to 9.4...

9.4.5: SearchHead Cluster csv notification!!!

KVSTORE problem updating from 9.1.x to 9.4.4

Problem with DEPLOYMENT after downgrading from 9.4...

Re: KVSTORE problem updating from 9.1.x to 9.4.4

Re: KVSTORE problem updating from 9.1.x to 9.4.4

Re: Upgrade Splunk Enterprise 8.1.x to 9.2

Re: Upgrade Splunk Enterprise 8.1.x to 9.2

Re: Spike in 503 errors in the Splunk WebUI after ...

Re: Spike in 503 errors in the Splunk WebUI after ...

Re: 9.3.6 to 9.3.9 ... freuently WebUI timeout!

Re: Spike in 503 errors in the Splunk WebUI after ...

Re: 9.3.6 to 9.3.9 ... freuently WebUI timeout!

Re: 9.3.6 to 9.3.9 ... freuently WebUI timeout!

9.3.6 to 9.3.9 ... frequently WebUI timeout!

Re: Earliest... Latest at 23:59:59 skipped!

Re: Earliest... Latest at 23:59:59 skipped!

Earliest... Latest at 23:59:59 skipped!

Re: MONGODB vulnerability in ZLIB

Re: MONGODB vulnerability in ZLIB

Re: MONGODB vulnerability in ZLIB

Re: MONGODB vulnerability in ZLIB

MONGODB vulnerability in ZLIB

Re: Where are login credentials to peers stored? A...

Re: Where are login credentials to peers stored? A...

Where are login credentials to peers stored? And w...

Re: Let's think about an unique way to always and ...

Re: Let's think about an unique way to always and ...

Re: Let's think about an unique way to always and ...

Join the Conversation