It failed with the method mentioned by @isoutamo  as well as @maraman_splunk  ... View more

systemctl stop splunkd //run as root works fine update splunk using rpm and tgz method //run a root works fine chown -R splunk:splunk /opt/splunk/ works fine changed the content of splunkd.service file based on latest content mentioned on site plus the two lines mentioned above (as splunk) /...path/to/splunk/bin/splunk start --accept-license --answer-yes Splunk Software License Agreement 10.21.2019 Failed with below message: This appears to be an upgrade of Splunk. --------------------------------------------------------------------------------) Splunk has detected an older version of Splunk installed on this machine. To finish upgrading to the new version, Splunk's installer will automatically update and alter your current configuration files. Deprecated configuration files will be renamed with a .deprecated extension. You can choose to preview the changes that will be made to your configuration files before proceeding with the migration and upgrade: If you want to migrate and upgrade without previewing the changes that will be made to your existing configuration files, choose 'y'. If you want to see what changes will be made before you proceed with the upgrade, choose 'n'. Perform migration and upgrade without previewing configuration changes? [y/n] y -- Migration information is being logged to '/opt/splunk/var/log/splunk/migration.log.2020-10-23.19-53-25' -- Migrating to: VERSION=8.0.2.1 BUILD=f002026bad55 PRODUCT=splunk PLATFORM=Linux-x86_64 Copying '/opt/splunk/etc/myinstall/splunkd.xml' to '/opt/splunk/etc/myinstall/splunkd.xml-migrate.bak'. Checking saved search compatibility... Handling deprecated files... Checking script configuration... Copying '/opt/splunk/etc/myinstall/splunkd.xml.cfg-default' to '/opt/splunk/etc/myinstall/splunkd.xml'. Deleting '/opt/splunk/etc/system/local/field_actions.conf'. The following apps might contain lookup table files that are not exported to other apps: splunk_monitoring_console Such lookup table files could only be used within their source app. To export them globally and allow other apps to access them, add the following stanza to each /opt/splunk/etc/apps/<app_name>/metadata/local.meta file: [lookups] export = system For more information, see http://docs.splunk.com/Documentation/Splunk/latest/AdvancedDev/SetPermissions#Make_objects_globally_available. An error occurred: Failed to run splunkd rest: stdout: -- stderr:splunkd: /opt/splunk/src/util/HttpClientRequest.cpp:1760: void HttpClientTransaction::_handleProxyConnect(): Assertion `_poolp->hasSslContext()' failed. Dying on signal #6 (si_code=-6), sent by PID 1657 (UID 1002). Attempting to clean up pidfile -- First-time run failed! Method 2 1)(as root) Stop it: systemctl stop splunkd 2) /opt/splunk/bin/splunk disable boot-start failed with below message /opt/splunk/bin/splunk disable boot-start error reading information on service splunk: No such file or directory Disabled. the splunkd.service file still exsists not deleted though. 3) performed the upgrade as mentioned with method above 4) /opt/splunk/bin/splunk enable boot-start -systemd-managed 1 -user splunk failed with error message below Perform migration and upgrade without previewing configuration changes? [y/n] y -- Migration information is being logged to '/opt/splunk/var/log/splunk/migration.log.2020-10-23.19-47-04' -- Migrating to: VERSION=8.0.2.1 BUILD=f002026bad55 PRODUCT=splunk PLATFORM=Linux-x86_64 Copying '/opt/splunk/etc/myinstall/splunkd.xml' to '/opt/splunk/etc/myinstall/splunkd.xml-migrate.bak'. Checking saved search compatibility... Handling deprecated files... Checking script configuration... Copying '/opt/splunk/etc/myinstall/splunkd.xml.cfg-default' to '/opt/splunk/etc/myinstall/splunkd.xml'. Deleting '/opt/splunk/etc/system/local/field_actions.conf'. The following apps might contain lookup table files that are not exported to other apps: splunk_monitoring_console Such lookup table files could only be used within their source app. To export them globally and allow other apps to access them, add the following stanza to each /opt/splunk/etc/apps/<app_name>/metadata/local.meta file: [lookups] export = system For more information, see http://docs.splunk.com/Documentation/Splunk/latest/AdvancedDev/SetPermissions#Make_objects_globally_available. An error occurred: Failed to run splunkd rest: stdout: -- stderr:splunkd: /opt/splunk/src/util/HttpClientRequest.cpp:1760: void HttpClientTransaction::_handleProxyConnect(): Assertion `_poolp->hasSslContext()' failed. Dying on signal #6 (si_code=-6), sent by PID 1572 (UID 1002). Attempting to clean up pidfile -- Also tried to delete the splunkd.file manually and recreate based on content mentioned but failed again. Also after failure removes the splunk.service file, which i created manually. ... View more

8 apps ... View more

This is not my query here. ... View more

No, i am trying to overide the sourcetype using regex, as available in below documentation. https://docs.splunk.com/Documentation/Splunk/latest/Data/Advancedsourcetypeoverrides The source type is override based on the regex which i have written in tranform.conf file. ... View more

Plus this is the new file content which i created while the splunk is in stop state (on splunk system 8.0) #This unit file replaces the traditional start-up script for systemd #configurations, and is used when enabling boot-start for Splunk on #systemd-based Linux distributions. [Unit] Description=Systemd service file for Splunk, generated by 'splunk enable boot-start' After=network.target [Service] Type=simple Restart=always ExecStart=/opt/splunk/bin/splunk _internal_launch_under_systemd KillMode=mixed KillSignal=SIGINT TimeoutStopSec=360 LimitNOFILE=65536 SuccessExitStatus=51 52 RestartPreventExitStatus=51 RestartForceExitStatus=52 Delegate=true CPUShares=1024 [Install] WantedBy=multi-user.target ... View more

My exsisting splunkd.service file looks like this: [Unit] Description=Splunk Enterprise 7.0.0 After=network.target Wants=network.target [Service] Type=forking Restart=always RestartSec=30s User=splunk Group=splunk LimitNOFILE=65536 LimitNPROC=16384 TimeoutSec=300 ExecStart=/opt/splunk/bin/splunk start --accept-license --answer-yes --no-prompt ExecStop=/opt/splunk/bin/splunk stop ExecReload=/opt/splunk/bin/splunk restart PIDFile=/opt/splunk/var/run/splunk/splunkd.pid [Install] WantedBy=multi-user.target # If you want to use $(systemctl [start|stop|restart] splunk) instead of splunkd ... Alias=splunk.service ... View more

One thing which i see in my web_Service logs are  root:730 - CONFIG: error_page.default (method): <bound method ErrorController.handle_error of <splunk.appserver.mrsparkle.controllers.error.ErrorController object at 0x7fb7fdef6e10>>   Other thing seems ok. From splunkd log file i have error messages related to my remote license server: ERROR LMTracker - failed to send rows, reason='Unable to connect to license master=XXX This is intentially kept like that because i want my server's to run without a license server.These are test servers and i don't want them to connect to a license server. Another thing is an warning related to searchead: WARN DistributedPeer - Peer:https://searchhead.splunk.test:8089 Failed to get server info from https://searchhead.splunk.test:8089/services/server/info response code=401 I tried to telnet from my searchead to cluster master on port 8089 and the connectivity looks ok.   I can upload the complete files if required. ... View more

I guess that is the problem with the events which i am trying to parse. Not every event include this function_name which makes it difficult to dynamically asign the source type. In my case the the events starts which a message 2020-08-31T05:27:58 : Log forwarding initializing for job=job::/test/function-name  It ends with  2020-08-31T05:28: job.id: XX Killed In between there are logs which has no reference for this so called function-name   So the query remains can be use this initializing line for creating a sourcetype and assign logs to the same sourcetype till it recived a message Killed in the events.   Or will process each and every event based on the conf files props.conf and tranform.conf.   Since in this case i am using func_name as a variable and it can comeup with multiple values.   ... View more

Works like a charm. Thanks for help  ... View more

Still the same now it gives an error on No package matching fatal: [default]: FAILED! => {"changed": false, "msg": "No package matching 'https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=8.0.5&product=splunk&filename=splunk-8.0.5-a1a6394cc5ae-linux-2.6-x86_64.rpm&wget=true' found available, installed or updated", "rc": 126, "results": ["No package matching 'https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=8.0.5&product=splunk&filename=splunk-8.0.5-a1a6394cc5ae-linux-2.6-x86_64.rpm&wget=true' found available, installed or updated"]}   PS: In this case i tried installing splunk rpm it gave the error same for UF ... View more

Splunk is a secure source and i have the code to verify the download. Anyways it still does not address my query, is this URL a Download-ble one ? ... View more

I don't think the latest version 8 is configured as init.d, coz my current version 7 is configured as systemd. But thanks for the comment. ... View more

Sorry to say but this seems like a situation where the response seems been there , done that, kind of case :). I have splunk installed as a service and service splunk stop works and all the process stops at once the output for ps -ef | grep -i splunk is none. I am more keen to know more about this article here which describe my situation https://docs.splunk.com/Documentation/Splunk/8.0.3/Admin/RunSplunkassystemdservice#Upgrade_considerations_for_systemd Does anyone faced or elaborate on this section of the article. ... View more

Done that 🙂 ... View more