About Gregski11

Gregski11 · ‎01-10-2023

alright so next I decided to divide and conquer, as you can see a lot of the headache is attributed to the python_upgrade_readiness_app which I would deem non life threatening so I decided to fandangle with it a little bit starting with these 5 "static pages" files C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/appserver/static/pages/jquery_scan.js C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/appserver/static/pages/python_scan.js C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/appserver/static/pages/setting.js C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/appserver/static/pages/setting_scan.js C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/appserver/static/pages/splunk9x_scan.js I simply renamed them all to origina_file_name.js.differsand copied the identical files from a sister server with no intergrity check issues and restarted Splunk

Gregski11 · ‎01-10-2023

so let's go after the low hanging fruit first: C:\Program Files\Splunk\etc/system/local/README missing the above README file is one that I myself indeed deleted, I simply did not want or need a REAME file cluttering up our /system/local subdirectories I find it interesting that Splunk finds that a missing README file undermines the integrity of the system, it makes me question the validity of this alert in general, what are your thoughts?

Gregski11 · ‎01-10-2023

so it appears that clicking that link pretty much does the same thing as running this command splunk validate files

Gregski11 · ‎01-10-2023

Splunk version 9.0.0 on Windows servers Please allow me to preface this by saying yes I GOOGLED this error and yes I did find some hits on this very Community site though 6 years old, and yes I did follow the many links to other links, which in turn lead me to more orphaned threads, ha ha, hence deciding to make this fresh more current post in which I hope we can present a solution File Integrity checks found 40 files that did not match the system-provided manifest. Review the list of problems reported by the InstalledFileHashChecker in splunkd.log File Integrity Check View ; potentially restore files from installation media, change practices to avoid changing files, or work with support to identify the problem. So if you click that link you get something like this: List of installed files presenting integrity check failures The table below shows files that were installed by the Splunk Enterprise package and have been improperly modified or are missing. Learn more. Search is completed File path Check result List of installed files presenting integrity check failures The table below shows files that were installed by the Splunk Enterprise package and have been improperly modified or are missing. Learn more. Search is completed File path Check result C:\Program Files\Splunk\etc/apps/SplunkForwarder/default/app.conf missing C:\Program Files\Splunk\etc/apps/SplunkForwarder/default/default-mode.conf missing C:\Program Files\Splunk\etc/apps/SplunkForwarder/default/health.conf missing C:\Program Files\Splunk\etc/apps/SplunkForwarder/default/outputs.conf missing C:\Program Files\Splunk\etc/apps/SplunkForwarder/default/server.conf missing C:\Program Files\Splunk\etc/apps/SplunkForwarder/metadata/default.meta missing C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/appserver/static/pages/jquery_scan.js differs C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/appserver/static/pages/python_scan.js differs C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/appserver/static/pages/setting.js differs C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/appserver/static/pages/setting_scan.js differs C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/appserver/static/pages/splunk9x_scan.js differs C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/eura_app_list.py differs C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/eura_check_mongodb_tls_dns_validation.py differs C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/eura_check_search_peer_ssl_config.py differs C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/eura_email_notification_switch_scripted_input.py differs C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/eura_remote_latest_report.py differs C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/eura_remote_scan_scripted_input.py differs C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/eura_scan_apps.py differs C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/eura_scan_deployment.py differs C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/eura_scan_process.py differs C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/eura_send_email.py differs C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/eura_telemetry.py differs C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/jura_scan_process.py differs C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/jura_telemetry.py differs C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/libs_py2/pura_libs_utils/pura_consts.py differs C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/libs_py2/pura_libs_utils/pura_logger_manager.py differs C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/libs_py2/pura_libs_utils/pura_skynet_log_manager.py differs C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/libs_py2/pura_libs_utils/pura_utils.py differs C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/libs_py2/pura_libs_utils/splunkbaseapps.csv differs C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/libs_py3/pura_libs_utils/pura_consts.py differs C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/libs_py3/pura_libs_utils/pura_logger_manager.py differs C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/libs_py3/pura_libs_utils/pura_skynet_log_manager.py differs C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/libs_py3/pura_libs_utils/pura_utils.py differs C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/libs_py3/pura_libs_utils/splunkbaseapps.csv differs C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/pura_remote_latest_report.py differs C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/pura_telemetry.py differs C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/scan_process.py differs C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/default/data/ui/nav/default.env_cloud.xml missing C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/default/data/ui/nav/default.xml differs C:\Program Files\Splunk\etc/system/local/README missing

Gregski11 · ‎01-10-2023

got this advice via the Splunk Documentation feedback: "This issue sounds like a bug in the code. Have you raised the issue with Splunk Support, so that they can investigate?" so today I opened a support case, and will keep you posted on what they find

Gregski11 · ‎01-06-2023

root node is Splunkish for the very first stupid line of any code so in the case below it's the <form> tag <form> <label>Accounts accessed by a specific device</label> <fieldset submitButton="true"> <input type="time" token="field1"> <label>Enter Date & Time Range</label> <default> so make it look like this <form version="1.1"> and yes this is the dumbest thing we have to do ever

Gregski11 · ‎01-06-2023

sorry about that Rick I am usually more better than this, so the error shows up in the Splunkd.log on one of our ver 9.0.0 Windows platform Search Heads right after we restart Splunk: INFO loader [0 MainThread] - win-service: Starting as a Windows service: will run various system checks first... INFO loader [0 MainThread] - win-service: Splunk starting as a local administrator INFO loader [0 MainThread] - Automatic migration of modular inputs WARN ApplicationManager [0 MainThread] - Cannot import non-existent application: __globals__ INFO loader [0 MainThread] - win-service: Command pre-flight-checks ran successfully. INFO loader [0 MainThread] - win-service: Command check-xml-files ran successfully.

Gregski11 · ‎01-06-2023

appreciate you, thank you so much

Gregski11 · ‎01-06-2023

awesome, thank you, that did it

Gregski11 · ‎01-06-2023

this was a no dice for me with the error: F - [main] Fatal Assertion 50755 at src\mongo\util\net\ssl_manager_windows.cpp 1609 F - [main] \n\n***aborting after fassert() failure\n\n

Gregski11 · ‎01-06-2023

I mean I don't even know where to start with this Error, lol Of course you can not import something that does not exist, it's like me saying I can not eat the cake that does not exist on my table. Anyway how do I go about find this application, looks like it gave me a name of it, does it look defaultish to you all or is this something we rolled ourselves? WARN ApplicationManager [0 MainThread] - Cannot import non-existent application: __globals__

Gregski11 · ‎01-05-2023

thank you splunkd.log 01-05-2023 23:03:47.881 -0800 WARN SSLOptions [0 MainThread] - server.conf/[sslConfig]/sslRootCAPath (C:\Program Files\Splunk\etc\auth\myCertificateChain.pem): does not exist; may not be able to use SSL the actual path is: C:\Program Files\Splunk\etc\auth\mycerts\myCertificateChain.pem

Gregski11 · ‎01-05-2023

in related news: also I noticed if I drop a file on the Search Head Captain in the system\local subdirectory it does not get automagically replicated to the other 3 members, however if I modify the banner using the Web UI and that creates a file in the system\local subdirectory on the Captain then it does get automagically replicated to the other members, perhaps that is how it is supposed to work I don't know

Gregski11 · ‎01-05-2023

thanks Sanjay I changed the mode like you suggested and repushed the bundle with a \local subdirectory in it with a .conf file in that, and no dice, the local subdirectory does not get created on the members, it is driving me crazy

Gregski11 · ‎01-05-2023

thank you Sanjay for offering to help, I tried what you just said and the Splunk Restart just sits at the following command Waiting for web server at https://127.0.0.1:8000 to be available.... What I don't understand is how or why the "Server" certificate impacts Splunk Web from starting when I am using a second and completely separate SSL web certificate for Splunk Web, that doesn't make any sense to me also as mentioned before I have 21 other Splunk servers where I did the exact precise same setup/configuration and those work fine, here's what lurks inside all of our server.conf files, they are all the same on purpose to make our configuration and administration consistent and uniform (but yes the passwords use each servers unique salt etc to hash the passwords) [sslConfig] serverCert = C:\Program Files\Splunk\etc\auth\myServerCertificate.pem sslRootCAPath = C:\Program Files\Splunk\etc\auth\myCertificateChain.pem sslPassword = $7$AfXR3titQZ6t0CxjunksAndStuffZN8dJunkbHStuffvyU7lX4= cliVerifyServerName = true

Gregski11 · ‎01-05-2023

so I configured a 4 member search head cluster successfully with a captain and a deployer server however when I got to deploy a simple app such as the GlobalBanner it aint working per this doc Use the deployer to distribute apps and configuration updates I create the following subdirectory and conf file on my Deployer server C:\Program Files\Splunk\etc\shcluster\apps\GlobalBanner\local\global-banner.conf and I push the bundle using this command from the Deployer server splunk apply shcluster-bundle -target https://MySearchHeadCaptain.MyDomain.net:8089 and the file ends up in the following directory on the search head cluster members C:\Program Files\Splunk\etc\apps\GlobalBanner\default\gloval-banner.conf Why does it end up in the default sub folder? My Deployer server is setup with the push mode set to full like so [shclustering] pass4SymmKey = $7$SoMeJuNk= shcluster_label = shcluster1 deployer_push_mode = full 1,000 thanks to anyone who can set me straight

Gregski11 · ‎01-05-2023

Failed to contact license manager: reason='Unable to connect to license manager=https://SplunkInstance01.MyDomain.net:8089 Error connecting:error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed - please check the output of the `openssl verify` command for the certificates involved; note that if certificate verification is enabled (requireClientCert or sslVerifyServerCert set to "true"), the CA certificate and the server certificate should not have the same Common Name.', first failure time= [...] first mea culpa for asking this as I am sure it has been asked before but I couldn't quite understand how to use the openssl verify command, when I try to run it I get this error: C:\Program Files\Splunk\etc\auth\distServerKeys>openssl verify -CAfile private.pem trusted.pem WARNING: can't open config file: C:\\jnkns\\workspace\\build-home/ssl/openssl.cnf Error loading file private.pem I also tried to run it from the bin subdirectory, home of the openssl utility C:\Program Files\Splunk\bin>openssl verify -CAfile "C:\Program Files\Splunk\etc\auth\distServerKeys\private.pem" "C:\Program Files\Splunk\etc\auth\distServerKeys\trusted.pem" WARNING: can't open config file: C:\\jnkns\\workspace\\build-home/ssl/openssl.cnf Error loading file C:\Program Files\Splunk\etc\auth\distServerKeys\private.pem I suspect this private \ public key pair combination may still be the stale default self signed combination cause Splunk to frown upon it, however what is perplexing to me is that it works on the other 20 plus servers, so I throw yourself upon your mercy for help please note we have over 2 dozen Splunk servers running version 9.0.0 all on Windows platforms and this is the only server getting this error, all servers use our own Microsoft CA internal Enterprise certificates based on a two tier (RootCA \ IntermediateCA architecture) so I think I know what I am doing ha ha at least in terms of certing

Gregski11 · ‎01-05-2023

thanks Rich I think this link is more my speed, now just to interpret it from Greek to English, lol Managing Indexers and Clusters of Indexers Configure multi-cluster search for multisite indexer clusters A search head can search across multiple multisite clusters or a combination of single-site and multisite clusters. To configure this, you need to specify the search head's site attribute when connecting it to a multisite cluster. By editing server.conf To configure multi-cluster search for a multisite cluster, you need to set two multisite-specific attributes: site and multisite. The locations of these attributes vary, depending on a few factors. If the search head will be searching across only multisite clusters, and the search head is on the same site in each cluster, put the site attribute under the [general] stanza and the multisite attribute under each [clustermanager] stanza: If the search head will be searching across only multisite clusters, and the search head is on a different site in each cluster, put both the site and the multisite attributes under the [clustermanager] stanzas: If the search head will be searching across a combination of single-site and multisite clusters, put both the site and the multisite attributes under the [clustermanager] stanza for any multisite clusters. In this example, the search head searches across two clusters, only one of which is multisite:

Gregski11 · ‎01-05-2023

I took over an established Splunk ecosystem when the main support admin retired. I noticed that not all of our stand alone Search Heads, and both Deployment Servers are setup to forward to all 12 of our Indexers in a single multi site Indexer Cluster (see table below) Some Search Heads in their outputs.conf only list the six Indexers that are assigned to Site 1, while the other Search Heads in their outputs.conf only list the six Indexers assigned to Site 2 However all Search Heads in their server.conf file have this stanza: [clustering] multisite = true So the question is should all of our Splunk Instances aka Search Heads, Cluster Master, and Deployment Servers have all 12 Indexers defined in their outputs.conf ? Site 1 Site 2 Indexer01 Indexer07 Indexer02 Indexer08 Indexer03 Indexer09 Indexer04 Indexer10 Indexer05 Indexer11 Indexer06 Indexer12

Gregski11 · ‎01-04-2023

also under my User directory I only have two subdirectories and no local subdirectory: C:\Program Files\Splunk\etc\users\greg\splunk_monitoring_console history metadata

Gregski11 · ‎01-04-2023

thank you Rich, I followed your instructions and checked under Settings \ User Interfaces \ Views and changed the owner to me, but I do not see my Dashboard there, I must be doing something wrong

Gregski11 · ‎01-04-2023

guys and gals let me start off by saying that my Search Game is weak, lol In version 9.0.0 on a Winderz platform I was in Settings \ Monitoring Console \ Forwarders: Deployment and under the Status and Configuration section I clicked the Sherlock Holmes round glass thingie to Open in Search, so far so good, I then made a few tiny mods to the Search, and got the results I needed then I clicked on Save As \ New Dashboard and this is where things get interesting, after saving my Dashboard I can't find it, it went into some black hole I guess? now I aint totally dumb so I booked marked that page first, but that is the only way I can get back to my Dashboard, so where did it get saved? I did see in the address bar that it's under the App https://MyClusterMaster.MyDomain.net:8000/en-US/app/splunk_monitoring_console/version_900_servers

Gregski11 · ‎01-03-2023

this is verbatim what I said in our staff meeting this morning, great minds, great minds indeed

Gregski11 · ‎01-03-2023

our main Splunk administrator retired and we since disabled his Active Directory account which he used to create and manage hundreds of Splunk searches, now listed as Orphaned under Settings \ All Configurations \ Reassign Knowledge Objects \ Orphaned we have the option of reassigning these searches to other Domain Accounts belonging to regular Splunk non admin users, or to the built in default Splunk admin account which is a local account on the box with no Domain permissions, so the question is should we do that since there is also this Warning: Knowledge object ownership changes can have side effects such as giving saved searches access to previously inaccessible data or making previously available knowledge objects unavailable. Review your knowledge objects before you reassign them. Running Splunk version 9.0.0 on the Microsoft Windows platform

Gregski11 · ‎01-03-2023

thank you so much for that direct link, you read my mind

Posts	147
Solutions	3
Karma Given	53
Karma Received	12
Member Since	‎03-04-2021

Online Status	Offline
Date Last Visited	‎10-12-2024 10:54 AM

Index Volume Location On Search Heads

How To Collect Event Viewer Logs On Deployment Ser...

Can't Log In To Splunk Web After Forklift Upgrade

Why does Splunk upgrade to version 9.1.0.1 end pre...

Is it possible to have alerts and reports assigned...

Are Alerts and Reports Global or specific to that ...

Can you send the same event log to two different I...

Why does field not populate in report but does whe...

How to see the forwarder management list of client...

Is there a way to tell what is populating a LOOKUP...

Re: File Integrity checks found files that did not...

Re: File Integrity checks found files that did not...

Re: File Integrity checks found files that did not...

File Integrity checks found files that did not mat...

Re: Simple Global Banner Search Head Cluster Bundl...

Re: This dashboard version is missing. Update the ...

Re: Cannot import non-existent application:

Re: Why is KV Store certificate renewal not workin...

Re: kvstore certificate renewal is not working...

Re: How to resolve issues with mongod startup such...

Cannot import non-existent application:

Re: Failed to contact license manager: reason='Una...

Re: Simple Global Banner Search Head Cluster Bundl...

Re: Simple Global Banner Search Head Cluster Bundl...

Re: Failed to contact license manager: reason='Una...

Simple Global Banner Search Head Cluster Bundle Pu...

Failed to contact license manager: reason='Unable ...

Re: Do we need to Forward to all Indexers in a two...

Do we need to forward to all indexers in a two sit...

Re: Where did my new Dashboard get saved?

Re: Where did my new Dashboard get saved?

Where did my new dashboard get saved?

Re: Should we reassing orphaned searches to the Sp...

Should we reassign orphaned searches to the Splunk...

Re: Monitoring Console has no separate pane for De...

Are you a member of the Splunk Community?