About thormanrd

richgalloway · ‎09-26-2023

That's normal and is just like on-prem.

thormanrd · ‎11-18-2022

I getting the same thing from my indexers ever since I upgrade to v9.0.0. We've had SmartStore running for a long time without these errors and now v9.0.0 has introduced this. What is the timeout the the S2BacketCache is hitting? 11-18-2022 17:51:32.786 ERROR S2BucketCache [62311 BatchSearch] - waitFor, timed out trying to localize files from remote storage, cache_id="bid|att_ent_flows~42619~99400A76-ADCE-4D97-A8E9-9DF39952E340|" download_status=queued files={"file_types":["bloomfilter"]} local_files={"file_types":["sourcetypes_data","deletes"]} probe_count=148 11-18-2022 17:51:32.787 ERROR DatabaseDirectoryManager [62311 BatchSearch] - Bucket still not local after waiting, bid=att_ent_flows~42619~99400A76-ADCE-4D97-A8E9-9DF39952E340

thormanrd · ‎10-28-2022

P.S. the <user redacted> is not a built in user either. It happens to be a substring of a valid SSO/SAML account. For example, if there was a valid SSO/SAML account called john@some-domain.com, then <user redacted> would = "john". All of our SSO/SAML accounts are email addresses as Azure SSO expects for authN/authZ.

thormanrd · ‎04-01-2022

We're seeing this issue too and our indexers that are logging this are coming to a screeching halt on indexing their data. What is the issue here?

thormanrd · ‎03-18-2022

There is fundamentally a problem ignoring HTTP responses. Many issues are illuminated with these, such as: HTTP 503 Slow down - typically a keyspace design issue HTTP 502 Read timeout - long reads and head operations with too little timeout setting HTTP 502 Connection timeout - long or busy network connections with too little timeout setting HTTP 500 Internal server error - poor exception processing that needs bug report HTTP 502 Name or Service not known - failure in DNS services HTTP 502 Partial upload - failure in multi-part upload operation that needs bug report

thormanrd · ‎03-18-2022

I'm seeing the same behavior in v8.2.4. I resolved many of the 5xx response by increasing the timeouts for the connections and uploads, but I'm still seeing a handful of 50s Internal server: Error responses. No clue what to do next.

thormanrd · ‎02-24-2021

The requirement for using authentication extension via SAML to Azure SSO include: App Registration (SSO via SAML) App/client secret token Application API permissions For testing the SAML_script_azure.py use the following format: /opt/splunk/bin/splunk cmd python3 /opt/splunk/etc/auth/scripts/SAML_script_azure.py getUserInfo < input.txt Where the contents of the input.txt file is: --username=<user name> --password=************** --scriptSecureArguments=clientId:<app id>;tenantId:<tenant id>;clientSecret<secret> --userInfo=true I can at least get a response from azure: "error": { "code": "Authorization_RequestDenied", "message": "Insufficient privileges to complete the operation.", "innerError": { "date": "2021-02-24T19:22:51", "request-id": "********", "client-request-id": "********" } } }

thormanrd · ‎02-24-2021

Do you have the same matrix for authentication extensions where Azure SSO is the IdP and SAML is the used?

thormanrd · ‎02-12-2021

Cold buckets are still searchable. The cold phase allows the admins to move data that is less likely to be searched to cheaper (i.e. slower) storage devices. This allow for management of storage cost vs accessibility.

thormanrd · ‎02-12-2021

Best thing to do IMHO is deploy your searches as your own app and expose them through Splunk Secure Gateway to the Splunk Mobile app or Splunk TV. It works great and has kept me out of trouble several times.

thormanrd · ‎02-12-2021

Having been a Splunk admin at a large telco for a few years now, I have simplified our monitoring to a few searches on _introspection index. Very fast and lightweight searches that I bundle in my own app and deploy through Splunk Secure Gateway to the Splunk Mobile app and Splunk TV. Keeps our admin team aware of critical measures like cpu, disc, iostats. Also use the cluster master REST API to watch the status of my indexers (e.g. Up, Down, maintenance mode, etc). Works well and has kept me out of trouble several times.

thormanrd · ‎10-30-2020

I have installed the Splunk Cloud Gateway app and registered my device and apps for mobile devices. But when I use a major of the dashboards on my mobile device I see the following exception on the cloud gateway app dashboard. 2020-10-30 13:15:16,214 ERROR [dashboard_request_processor] [dashboard_request_processor] [fetch_search_job_results_visualization_data] [35913] Failed to get search job results search_id=4d646b7793ce9e18cb8a6b82b9c05f02a03ca7ec06d9529006333abb947e32d8, response.code=400, response.text={"messages":[{"type":"FATAL","text":"Error in 'rest' command: Invalid argument: '$datatype$'"}]} I'm running Splunk Enterprise 7.3.3 on Linux and Splunk Cloud Gateway v1.13. The connectivity test with curl to the websockets and gateway server work just fine. I have green status on the gateway dashboard and message statistics. Any suggestions are appreciated.

frechetta93 · ‎07-23-2020

Did you ever get this solved? I have the same issue.

DavidHourani · ‎02-05-2020

You're welcome @thormanrd ! Let me know if you need anything else 🙂

thormanrd · ‎01-14-2020

Looks like the Settings, Indexes page has the statistics for the local file system(s) where indexed data is stored on that node. Since my index master is not an indexer, all the statistics are zero meaning there is no local bucket storage on that node. The Settings, Index Clustering, Indexes is the cluster wide metrics for all indexers cumulatively. I discovered this by browsing to all the indexers Settings, Indexes and found the metrics were different per node. All together they added up to Settings, Index Clustering, Indexes. So, the Settings, Indexes is local. Settings, Index Clustering, Indexes is the sum of all index cluster members. G2G

thormanrd · ‎09-13-2019

When I enable distributed mode on my cluster master all the indexer nodes are added to the peers list. However, When i try to edit server roles, or Apply Changes I get this error in the splunkd_ui_access.log: 10.5.90.160 - ***** [13/Sep/2019:15:36:02.394 +0000] "POST /en-US/splunkd/__raw/services/search/distributed/groups HTTP/1.1" 404 224 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0" - ***************** 0ms Eight of them in a row. And then I get a dialog box that says, "Error Something went wrong. Try again later."

Sukisen1981 · ‎05-06-2019

could be an issue with the AWS side- have you disabled rollback on failure option as mentioned here - https://aws.amazon.com/premiumsupport/knowledge-center/cloudformation-failed-signal/

Posts	28
Solutions	1
Karma Given	3
Karma Received	2
Member Since	‎05-06-2019

Online Status	Offline
Date Last Visited	‎11-18-2022 04:04 PM

SavedSearchFetcher Authentication Extension Failur...

Where is SAML_script_azure.py?

Are Smartstore buckets uploaded to S3 immutable?

Splunk Cloud Gateway invalid rest argument

SAML RealName Alias

What's the best strategy for volume tags when inde...

Index Master statistics different between Settings...

How to resolve 404 in Distributed Monitoring Conso...

Splunk Enterprise deployment on AWS fails.

Re: Smartstore Buckets Immutability

Re: Why am I getting partial results on some index...

Re: SavedSearchFetcher Authentication Extension Fa...

Re: ERROR CMRemotePrimaryManager - Failed to evict...

Re: [SmartStore] HTTP 502 network errors with S3?

Re: Smartstore seems to work but shows multipart-u...

Re: Where is SAML_script_azure.py?

Re: Azure API permissions for Splunk Add-on for Mi...

Re: Hot/Warm and Cold Storage

Re: Run MC Health Checks on scheduled basis and em...

Re: Infrastructure monitoring in Splunk

Splunk Cloud Gateway invalid rest argument

Re: SAML RealName Alias

Re: What's the best strategy for volume tags when ...

Re: Index Master statistics different between Sett...

How to resolve 404 in Distributed Monitoring Conso...

Re: Splunk Enterprise deployment on AWS fails.