About joshiro

joshiro · ‎01-13-2023

I guess we got something wrong with the SH cluster, it is not scheduling the searches evenly across the nodes. And the one with the highest count is not even the captain. Thanks again for the reply, we ll open a support case and try to troubleshoot this issue.

joshiro · ‎01-12-2023

Thanks for the reply. Very helpful, but i still cant clearly understand what the number 46 means. Are the 46 concurrent historical searches that shows this panel scheduled and running at this moment? Or are they currently scheduled to run at another time (not running)?

joshiro · ‎01-12-2023

We are trying to troubleshoot some memory consumption issues with one of the SH cluster nodes. We found that this instance shows high concurrency of scheduled reports 46/15 historical while the other nodes are way below this number. Also in the running historical scheduled reports panel we got a column Mode that shows "historical" as value. What does a "historical" report mean in this context? The Splunk documentation for DMC doesnt explain it. https://docs.splunk.com/Documentation/Splunk/9.0.3/DMC/Scheduleractivity Regards.

joshiro · ‎01-04-2023

Thanks for the reply. The thing is that the HeavyForwarder that contains DB Connect is always online during the rolling restart of the indexers, so unless its using the KV Store of the indexers it shouldnt be unavailable. It might use the KV Store of the indexers for something else but we dont know. We are going to try to stop the HF before the rolling restart of the indexer and then start it after it finishes.

joshiro · ‎12-22-2022

Splunk Enterprise 9.0.1 on premise, clustered search heads and indexers. DB Connect 3.7.0. We found out that every time the indexer cluster is restarted, some events are being duplicated in the indexes around the time of the restart. There are some older threads discussing similar issues but they are using much older versions of the software. Any ideas on how to troubleshoot/debug/workaround this issue?

joshiro · ‎12-19-2022

The search concurrency count in each SH node appears constant even though we are not running rt searches. It seems that the scheduler in the SH cluster has some stuck processes running constantly even after a restart. Any ideas on how to clean stuck processes on the scheduler?

joshiro · ‎11-30-2022

We are running a SHC with Splunk Enterprise OnPrem 9.0.1 and noticed that the concurrent searches in one of the nodes is way higher than the rest (3 times aprox.) even though the scheduler delegation shows its delegating evenly across the nodes. Most of the scheduled searches are from an app that runs dbx queries to keep updated some lookups, these are scheduled to run a few times a week but appear to be running constantly in the scheduler. These concurrent searches run constantly even after a restart of the node. It doesnt happen in a single instance with the same apps, so we think it is a clustering issue. How can we troubleshoot/debug this behaviour?

joshiro · ‎11-30-2022

It is not the first time that they release a package with some issues. Recently, there was an issue with some federated.conf parameters not having the respective spec file definition, and it kept showing a warning on CLI on every restart. They might be having some issues with QA. We are also trying to use SSG and Splunk Mobile and we are encountering several issues that didnt get caught by QA.

joshiro · ‎11-17-2022

Yes, that is exactly the problem. We ended up copying a working version of that savedsearch to the local directory of the splunk_instrumentation app. This makes the report run normally, but still shows errors on CLI because the file in default directory is still broken. What workaround did you apply?

joshiro · ‎11-11-2022

Can you run a diff between the default savedsearches.conf of the splunk_instrumentation in both tar files? It might return that the files are identical, and you ll need to pipe "cat -A" to see hidden special chars. Just run: diff <old-savedsearches> <new-savedsearches> | cat -A A workaround in the 9.0.2 version was to copy the [instrumentation.usage.tlsBestPractices] stanza from the old working file and paste it in the local folder in the app. So splunk_instrumentation uses the old version of that particular stanza. This ll still show the error on CLI because the default file its still broken, but if you run the report it works just fine. Hope this helps.

joshiro · ‎11-10-2022

Thanks for the reply. We already opened a support case, they are aware of this issue and working on a workaround.

joshiro · ‎11-10-2022

After upgrading Splunk Enterprise to 9.0.2 we are encountering the following error on every restart on CLI: Checking conf files for problems... Invalid key in stanza [instrumentation.usage.tlsBestPractices] in /opt/splunk/etc/apps/splunk_instrumentation/default/savedsearches.conf, line 451: | append [| rest /services/configs/conf-pythonSslClientConfig | eval sslVerifyServerCert (value: if(isnull(sslVerifyServerCert),"unset",sslVerifyServerCert), splunk_server=sha256(splunk_server) | stats values(eai:acl.app) as python_configuredApp values(sslVerifyServerCert) as python_sslVerifyServerCert by splunk_server | eval python_configuredSystem=if(python_configuredApp="system","true","false") | fields python_sslVerifyServerCert, splunk_server, python_configuredSystem] | append [| rest /services/configs/conf-web/settings | eval mgmtHostPort=if(isnull(mgmtHostPort),"unset",mgmtHostPort), splunk_server=sha256(splunk_server) | stats values(eai:acl.app) as fwdrMgmtHostPort_configuredApp values(mgmtHostPort) as fwdr_mgmtHostPort by splunk_server | eval fwdrMgmtHostPort_configuredSystem=if(fwdrMgmtHostPort_configuredApp="system","true","false") | fields fwdrMgmtHostPort_sslVerifyServerCert, splunk_server, fwdrMgmtHostPort_configuredSystem] | append [| rest /services/configs/conf-server/sslConfig | eval cliVerifyServerName=if(isnull(cliVerifyServerName),"feature",cliVerifyServerName), splunk_server=sha256(splunk_server) | stats values(cliVerifyServerName) as servername_cliVerifyServerName values(eai:acl.app) as servername_configuredApp by splunk_server | eval cli_configuredSystem=if(cli_configuredApp="system","true","false") | fields cli_sslVerifyServerCert, splunk_server, cli_configuredSystem] | stats values(*) as * by splunk_server | eval date=now() | makejson output=data | eval _time=date, date=strftime(date,"%Y-%m-%d") | fields data date _time). Your indexes and inputs configurations are not internally consistent. For more information, run 'splunk btool check --debug' This was not happening on 9.0.1 so we checked the 'savedsearches.conf' of the splunk_instrumentation app in the 9.0.1 tar and we found that the 9.0.2 'savedsearches.conf' is actually older and different than the 9.0.1 version. ~/Downloads$ diff savedsearches.conf.901 savedsearches.conf.902 | cat -A 447c447$ < | append [| rest /services/configs/conf-server/sslConfig | eval sslVerifyServerCert=if(isnull(sslVerifyServerCert),"unset",sslVerifyServerCert), splunk_server=sha256(splunk_server) | stats values(eai:acl.app) as global_configuredApp values(sslVerifyServerCert) as global_sslVerifyServerCert by splunk_server | eval global_configuredSystem=if(global_configuredApp="system","true","false") | fields global_sslVerifyServerCert, splunk_server, global_configuredSystem] \$ ---$ > | append [| rest /services/configs/conf-server/sslConfig | eval sslVerifyServerCert=if(isnull(sslVerifyServerCert),"unset",sslVerifyServerCert), splunk_server=sha256(splunk_server) | stats values(eai:acl.app) as global_configuredApp values(sslVerifyServerCert) as global_sslVerifyServerCert by splunk_server | eval global_configuredSystem=if(global_configuredApp="system","true","false") | fields global_sslVerifyServerCert, splunk_server, global_configuredSystem] \ $ The difference lies in the scaped end of line character at the end. We also tried to run this search from the GUI and it raises an error confirming that the search is indeed broken: We "solved" it by using the 9.0.1 version in the local folder of the app splunk_instrumentation. Has anyone found out if this broken search is affecting Splunk Enterprise usage in anyway?

joshiro · ‎11-07-2022

On Splunk Enterprise on prem 9.0.1, after using Smart Forecasting (MLTK 5.3.1), and publishing the recently created model, the apply command returns error "Error in 'apply' command: list assignment index out of range". But the same model created using SPL with fit command works fine. Here is the traceback: 11-07-2022 15:17:42.289 ERROR ChunkedExternProcessor [50291 ChunkedExternProcessorStderrLogger] - stderr: Traceback (most recent call last): 11-07-2022 15:17:42.289 ERROR ChunkedExternProcessor [50291 ChunkedExternProcessorStderrLogger] - stderr: File "/opt/splunk/etc/apps/Splunk_ML_Toolkit/bin/processors/ApplyProcessor.py", line 158, in apply 11-07-2022 15:17:42.289 ERROR ChunkedExternProcessor [50291 ChunkedExternProcessorStderrLogger] - stderr: prediction_df = algo.apply(df, process_options) 11-07-2022 15:17:42.289 ERROR ChunkedExternProcessor [50291 ChunkedExternProcessorStderrLogger] - stderr: File "/opt/splunk/etc/apps/Splunk_ML_Toolkit/bin/algos/StateSpaceForecast.py", line 712, in apply 11-07-2022 15:17:42.289 ERROR ChunkedExternProcessor [50291 ChunkedExternProcessorStderrLogger] - stderr: df = self.add_output_metadata(df) 11-07-2022 15:17:42.289 ERROR ChunkedExternProcessor [50291 ChunkedExternProcessorStderrLogger] - stderr: File "/opt/splunk/etc/apps/Splunk_ML_Toolkit/bin/algos/StateSpaceForecast.py", line 353, in add_output_metadata 11-07-2022 15:17:42.289 ERROR ChunkedExternProcessor [50291 ChunkedExternProcessorStderrLogger] - stderr: metadata[i] = 'f' 11-07-2022 15:17:42.289 ERROR ChunkedExternProcessor [50291 ChunkedExternProcessorStderrLogger] - stderr: IndexError: list assignment index out of range 11-07-2022 15:17:42.289 ERROR ChunkedExternProcessor [50291 ChunkedExternProcessorStderrLogger] - stderr: WARNING Error while applying model "modelo4": list assignment index out of range 11-07-2022 15:17:42.289 ERROR ChunkedExternProcessor [50291 ChunkedExternProcessorStderrLogger] - stderr: list assignment index out of range 11-07-2022 15:17:42.290 ERROR ChunkedExternProcessor [50291 ChunkedExternProcessorStderrLogger] - stderr: Traceback (most recent call last): 11-07-2022 15:17:42.290 ERROR ChunkedExternProcessor [50291 ChunkedExternProcessorStderrLogger] - stderr: File "/opt/splunk/etc/apps/Splunk_ML_Toolkit/bin/processors/ApplyProcessor.py", line 158, in apply 11-07-2022 15:17:42.290 ERROR ChunkedExternProcessor [50291 ChunkedExternProcessorStderrLogger] - stderr: prediction_df = algo.apply(df, process_options) 11-07-2022 15:17:42.290 ERROR ChunkedExternProcessor [50291 ChunkedExternProcessorStderrLogger] - stderr: File "/opt/splunk/etc/apps/Splunk_ML_Toolkit/bin/algos/StateSpaceForecast.py", line 712, in apply 11-07-2022 15:17:42.290 ERROR ChunkedExternProcessor [50291 ChunkedExternProcessorStderrLogger] - stderr: df = self.add_output_metadata(df) 11-07-2022 15:17:42.290 ERROR ChunkedExternProcessor [50291 ChunkedExternProcessorStderrLogger] - stderr: File "/opt/splunk/etc/apps/Splunk_ML_Toolkit/bin/algos/StateSpaceForecast.py", line 353, in add_output_metadata 11-07-2022 15:17:42.290 ERROR ChunkedExternProcessor [50291 ChunkedExternProcessorStderrLogger] - stderr: metadata[i] = 'f' 11-07-2022 15:17:42.290 ERROR ChunkedExternProcessor [50291 ChunkedExternProcessorStderrLogger] - stderr: IndexError: list assignment index out of range 11-07-2022 15:17:42.290 ERROR ChunkedExternProcessor [50291 ChunkedExternProcessorStderrLogger] - stderr: During handling of the above exception, another exception occurred: 11-07-2022 15:17:42.290 ERROR ChunkedExternProcessor [50291 ChunkedExternProcessorStderrLogger] - stderr: Traceback (most recent call last): 11-07-2022 15:17:42.290 ERROR ChunkedExternProcessor [50291 ChunkedExternProcessorStderrLogger] - stderr: File "/opt/splunk/etc/apps/Splunk_ML_Toolkit/bin/cexc/__init__.py", line 174, in run 11-07-2022 15:17:42.290 ERROR ChunkedExternProcessor [50291 ChunkedExternProcessorStderrLogger] - stderr: while self._handle_chunk(): 11-07-2022 15:17:42.290 ERROR ChunkedExternProcessor [50291 ChunkedExternProcessorStderrLogger] - stderr: File "/opt/splunk/etc/apps/Splunk_ML_Toolkit/bin/cexc/__init__.py", line 236, in _handle_chunk 11-07-2022 15:17:42.290 ERROR ChunkedExternProcessor [50291 ChunkedExternProcessorStderrLogger] - stderr: ret = self.handler(metadata, body) 11-07-2022 15:17:42.290 ERROR ChunkedExternProcessor [50291 ChunkedExternProcessorStderrLogger] - stderr: File "/opt/splunk/etc/apps/Splunk_ML_Toolkit/bin/apply.py", line 136, in handler 11-07-2022 15:17:42.290 ERROR ChunkedExternProcessor [50291 ChunkedExternProcessorStderrLogger] - stderr: self.controller.execute() 11-07-2022 15:17:42.290 ERROR ChunkedExternProcessor [50291 ChunkedExternProcessorStderrLogger] - stderr: File "/opt/splunk/etc/apps/Splunk_ML_Toolkit/bin/chunked_controller.py", line 220, in execute 11-07-2022 15:17:42.290 ERROR ChunkedExternProcessor [50291 ChunkedExternProcessorStderrLogger] - stderr: self.processor.process() 11-07-2022 15:17:42.290 ERROR ChunkedExternProcessor [50291 ChunkedExternProcessorStderrLogger] - stderr: File "/opt/splunk/etc/apps/Splunk_ML_Toolkit/bin/processors/ApplyProcessor.py", line 177, in process 11-07-2022 15:17:42.290 ERROR ChunkedExternProcessor [50291 ChunkedExternProcessorStderrLogger] - stderr: self.df = self.apply(self.df, self.algo, self.process_options) 11-07-2022 15:17:42.290 ERROR ChunkedExternProcessor [50291 ChunkedExternProcessorStderrLogger] - stderr: File "/opt/splunk/etc/apps/Splunk_ML_Toolkit/bin/processors/ApplyProcessor.py", line 166, in apply 11-07-2022 15:17:42.290 ERROR ChunkedExternProcessor [50291 ChunkedExternProcessorStderrLogger] - stderr: raise RuntimeError(e) 11-07-2022 15:17:42.290 ERROR ChunkedExternProcessor [50291 ChunkedExternProcessorStderrLogger] - stderr: RuntimeError: list assignment index out of range Anyone managed to get the publish option in MLTK working?

joshiro · ‎10-18-2022

We are having the same issue on Splunk Enterprise 9 and DB Connect 3.7.0. Is there another way to workaround this issue without changing the logging settings? Any ETA on a fix for the Splunk DB Connect app?

joshiro · ‎10-18-2022

We are having the same issue on Splunk Enterprise 9.0.1 and DB Connect 3.7.0. Have you managed to get it fixed?

joshiro · ‎10-14-2022

I know that the documentation says it is installed only on SHC, but i still dont understand why CIM defines indexes in default if we are suposed to remove it from the SH bundle and never install it on the indexers. I feel that something wont work correctly if we dont deploy that index definition. Unless its deprecated and shouldnt be there.

joshiro · ‎10-14-2022

We will remove the indexes.conf from the bundle. But we still need to define it in the indexers? is that correct?

joshiro · ‎10-14-2022

The CIM documentation says that we should install CIM only on SH. But it contains an indexes.conf in default. Should we leave the indexes.conf in the SHC? in this case the index defined inside indexes.conf wont be usable because is not defined in the indexer cluster. We dont know if it is correct to define the CIM indexes.conf in the SHC instead of the indexers. Anyone managed to install CIM in a clustered environment without issues?

joshiro · ‎10-05-2022

The new version of an app created using Splunk Add On Builder 4.1.1 on Splunk Enterprise 9.0.1 breaks on upgrade because the older password constant in 'aob_py3/splunktaucclib/rest_handler/credentials.py' was six '*' and the new one is eight '*'. Now is expecting that the password setting in inputs.conf uses the new format. This makes the Input page not load correctly raising error 500. Any ideas on how to solve this problem so the final users can seamlessly upgrade the app and keep the older input configurations? We are trying to work with this solution: https://community.splunk.com/t5/All-Apps-and-Add-ons/Splunk-Addon-Builder-4-package-resetting-password-conf-entries/td-p/584187 But we dont like that we need to patch the credentials.py

joshiro · ‎10-05-2022

We are currently trying to make an app upgrade seamlessly but the addon builder 4 version of the app doesnt like the inputs.conf from the older versions. The cause of this problem is this new password constant, but we need to make it work without the requirement to manually change the current password from six * to eight *. We are testing a patch that modifies the PASSWORD constant directly so the entire lib uses the old constant, but we dont know if it breaks other stuff. Anyone encountered this problem and managed to solve?

joshiro · ‎09-20-2022

We are having the same issue where older projects dont show up. Have you managed to make it work?

joshiro · ‎09-14-2022

We are using Splunk Enterprise OnPrem 8.2.3.3 with the add-on version 4.1.0. Our client configured the permissions according to the documentation but the following error keeps raising: 2022-09-14 10:28:53,294 level=ERROR pid=20311 tid=MainThread logger=splunk_ta_o365.modinputs.graph_api pos=utils.py:wrapper:72 | datainput=b'O365ServicesUserCounts' start_time=1663169316 | message="Data input was interrupted by an unhandled exception." Traceback (most recent call last): File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunksdc/utils.py", line 70, in wrapper return func(*args, **kwargs) File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/graph_api/__init__.py", line 109, in run return consumer.run() File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/graph_api/GraphApiConsumer.py", line 62, in run items = [endpoint.get('message_factory')(item) for item in reports.throttled_get(self._session)] File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/common/portal.py", line 573, in throttled_get return self.get(session) File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/common/portal.py", line 599, in get raise O365PortalError(response) splunk_ta_o365.common.portal.O365PortalError: 403:{"error":{"code":"UnknownError","message":"{\"error\":{\"code\":\"S2SUnauthorized\",\"message\":\"Invalid permission.\"}}","innerError":{"date":"2022-09-14T15:28:38","request-id":"72bea9b3-7f84-4063-9be4-05a4d331f39d","client-request-id":"72bea9b3-7f84-4063-9be4-05a4d331f39d"}}} Also: 2022-09-14 10:28:53,294 level=ERROR pid=20311 tid=MainThread logger=splunk_ta_o365.modinputs.graph_api.GraphApiConsumer pos=GraphApiConsumer.py:run:74 | datainput=b'O365ServicesUserCounts' start_time=1663169316 | message="Error retrieving Graph API Messages." exception=403:{"error":{"code":"UnknownError","message":"{\"error\":{\"code\":\"S2SUnauthorized\",\"message\":\"Invalid permission.\"}}","innerError":{"date":"2022-09-14T15:28:38","request-id":"72bea9b3-7f84-4063-9be4-05a4d331f39d","client-request-id":"72bea9b3-7f84-4063-9be4-05a4d331f39d"}}} Anyone knows what is the missing or misconfigured permission?

joshiro · ‎09-12-2022

We are currently using Splunk Enterprise OnPrem 9.0.0 and when we try to install IT Essentials Work (https://splunkbase.splunk.com/app/5403/) it raises the following error: "Invalid app contents: archive contains more than one immediate subdirectory: and DA-ITSI-DATABASE". Anyone encountered this error and managed to install IT Essentials Work?

joshiro · ‎08-30-2022

We are having the same issue with the message "There was an error processing the upload.Invalid app contents: archive contains more than one immediate subdirectory: and DA-ITSI-DATABASE" Have you managed to install Essentials Work?

joshiro · ‎08-22-2022

We thought about sudoers but you still need to invoke the scripts using sudo and we dont want to modify them or run splunkd with sudo. The binaries that require privileges are dmesg and nfsiostat, we also thought that we can add the suid flag to run them as owner but we dont want to change permisions from the OS binaries. We are still looking for alternatives.

Posts	74
Solutions	6
Karma Given	32
Karma Received	6
Member Since	‎03-08-2021

Online Status	Offline
Date Last Visited	‎10-01-2024 02:50 PM

How do you schedule a report that needs to run mul...

How can we implement read only columns in the Splu...

How can we preload a token from a search result in...

How can we hide a panel just for Mobile?

Why nested macros using the same argument wont wor...

How can we migrate inputs from DB Connect 3.7 to 3...

What does it mean searchparsetmp in the StreamedSe...

What does historical mode mean in the Scheduled Ac...

Why DB Connect duplicates events on indexer cluste...

How to troubleshoot a scheduler that has a high am...

Re: What does historical mode mean in the Schedule...

Re: What does historical mode mean in the Schedule...

What does historical mode mean in the Scheduled Ac...

Re: Why DB Connect duplicates events on indexer cl...

Why DB Connect duplicates events on indexer cluste...

Re: How to troubleshoot a scheduler that has a hig...

How to troubleshoot a scheduler that has a high am...

Re: Why is there splunk_instrumentation error afte...

Re: Why is there splunk_instrumentation error afte...

Re: Why is there splunk_instrumentation error afte...

Re: Why is there splunk_instrumentation error afte...

Why is there splunk_instrumentation error after up...

MLTK published models differ from SPL created mode...

Re: Upgraded Splunk DB Connect V3.2.0 to V3.3.0 se...

Re: Splunk DB Connect ERROR Without Error

Re: Installing CIM in a clustered environment

Re: Installing CIM in a clustered environment

Installing CIM in a clustered environment- Has any...

App created with Splunk Addon Builder 4 doesnt lik...

Re: Splunk Addon Builder 4 package resetting passw...

Re: Export and import project in Addon Builder

Permisions error using Splunk AddOn for Microsoft ...

Error installing Splunk IT Essentials Work- Anyone...

Re: What free App replaces "Splunk App for AWS" wh...

Re: How do you run scripts from Splunk TA for Unix...

Are you a member of the Splunk Community?