Splunk web down

[corti77]

Hi,

recently we had an issue with the LUN drive where data is stored and after fixing it, a new problem came up.

splunk services starts normally but the web access does not work anymore.

the output of the splunk start command is the following

\bin>splunk.exe start

Splunk> Map. Reduce. Recycle.

Checking prerequisites...
Checking mgmt port [8089]: open
Checking configuration... Done.
Checking critical directories... Done
Checking indexes...
(skipping validation of index paths because not running as LocalSystem)
Validated: _configtracker _introspection _metrics _metrics_rollup _thefishbucket anomaly_detection autek azure cim_modactions cisco citrix email eusc_apps firedalerts ftp hyper-v infraops itsi_grouped_alerts itsi_im_meta itsi_im_metrics itsi_import_objects itsi_notable_archive itsi_notable_audit itsi_summary itsi_summary_metrics itsi_tracked_alerts kubernetes metrics_sc4s msad msexchange netauth netfw netops netproxy os osnix pan_logs perfmon rancher_k8sca rancher_k8scc rancher_k8scs rancherprod sample snmptrapd sns symantec sysmon test thor windefender windows wineventlog winevents
Done


Bypassing local license checks since this instance is configured with a remote license master.

Checking filesystem compatibility... Done
Checking conf files for problems...
Bad regex value: '(::)?...', of param: props.conf / [(::)?...]; why: this regex is likely to apply to all data and may break summary indexing, among other Splunk features.
One or more regexes in your configuration are not valid. For details, please see btool.log or directly above.
Done
Checking default conf files for edits...
Validating installed files against hashes from 'C:\Program Files\Splunk\splunk-9.0.8-4fb5067d40d2-windows-64-manifest'
All installed files intact.
Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...

Splunkd: Starting (pid 38432)
Done

extract of btool.log 

05-06-2024 11:07:35.039 WARN  ConfMetrics - single_action=BASE_INITIALIZE took wallclock_ms=1014
05-06-2024 11:17:25.445 WARN  IConfCache - Stanza has an expansion [script://C:\Program Files\Splunk\etc\apps\TA-Exchange-ClientAccess\bin\exchangepowershell.cmd v14 read-audit-logs_2010_2013.ps1], ignoring alternate expansion [script://C:\Program Files\Splunk\etc\apps\TA-Exchange-Mailbox\bin\exchangepowershell.cmd v14 read-audit-logs_2010_2013.ps1] in inputs.conf
05-06-2024 11:17:25.445 WARN  IConfCache - Stanza has an expansion [script://C:\Program Files\Splunk\etc\apps\TA-Exchange-ClientAccess\bin\exchangepowershell.cmd v15 read-audit-logs_2010_2013.ps1], ignoring alternate expansion [script://C:\Program Files\Splunk\etc\apps\TA-Exchange-Mailbox\bin\exchangepowershell.cmd v15 read-audit-logs_2010_2013.ps1] in inputs.conf
05-06-2024 13:00:58.310 WARN  IConfCache - Stanza has an expansion [script://C:\Program Files\Splunk\etc\apps\TA-Exchange-ClientAccess\bin\exchangepowershell.cmd v14 read-audit-logs_2010_2013.ps1], ignoring alternate expansion [script://C:\Program Files\Splunk\etc\apps\TA-Exchange-Mailbox\bin\exchangepowershell.cmd v14 read-audit-logs_2010_2013.ps1] in inputs.conf
05-06-2024 13:00:58.310 WARN  IConfCache - Stanza has an expansion [script://C:\Program Files\Splunk\etc\apps\TA-Exchange-ClientAccess\bin\exchangepowershell.cmd v15 read-audit-logs_2010_2013.ps1], ignoring alternate expansion [script://C:\Program Files\Splunk\etc\apps\TA-Exchange-Mailbox\bin\exchangepowershell.cmd v15 read-audit-logs_2010_2013.ps1] in inputs.conf
05-06-2024 13:00:58.373 WARN  btool-support - Bad regex value: '(::)?...', of param: props.conf / [(::)?...]; why: this regex is likely to apply to all data and may break summary indexing, among other Splunk features.
05-06-2024 13:19:36.176 WARN  ConfMetrics - single_action=BASE_INITIALIZE took wallclock_ms=1234
05-06-2024 14:44:42.912 WARN  IConfCache - Stanza has an expansion [script://C:\Program Files\Splunk\etc\apps\TA-Exchange-ClientAccess\bin\exchangepowershell.cmd v14 read-audit-logs_2010_2013.ps1], ignoring alternate expansion [script://C:\Program Files\Splunk\etc\apps\TA-Exchange-Mailbox\bin\exchangepowershell.cmd v14 read-audit-logs_2010_2013.ps1] in inputs.conf
05-06-2024 14:44:42.912 WARN  IConfCache - Stanza has an expansion [script://C:\Program Files\Splunk\etc\apps\TA-Exchange-ClientAccess\bin\exchangepowershell.cmd v15 read-audit-logs_2010_2013.ps1], ignoring alternate expansion [script://C:\Program Files\Splunk\etc\apps\TA-Exchange-Mailbox\bin\exchangepowershell.cmd v15 read-audit-logs_2010_2013.ps1] in inputs.conf
05-06-2024 14:44:42.975 WARN  btool-support - Bad regex value: '(::)?...', of param: props.conf / [(::)?...]; why: this regex is likely to apply to all data and may break summary indexing, among other Splunk features.
05-06-2024 14:44:51.022 WARN  IConfCache - Stanza has an expansion [script://C:\Program Files\Splunk\etc\apps\TA-Exchange-ClientAccess\bin\exchangepowershell.cmd v14 read-audit-logs_2010_2013.ps1], ignoring alternate expansion [script://C:\Program Files\Splunk\etc\apps\TA-Exchange-Mailbox\bin\exchangepowershell.cmd v14 read-audit-logs_2010_2013.ps1] in inputs.conf
05-06-2024 14:44:51.022 WARN  IConfCache - Stanza has an expansion [script://C:\Program Files\Splunk\etc\apps\TA-Exchange-ClientAccess\bin\exchangepowershell.cmd v15 read-audit-logs_2010_2013.ps1], ignoring alternate expansion [script://C:\Program Files\Splunk\etc\apps\TA-Exchange-Mailbox\bin\exchangepowershell.cmd v15 read-audit-logs_2010_2013.ps1] in inputs.conf
05-06-2024 14:44:51.084 WARN  btool-support - Bad regex value: '(::)?...', of param: props.conf / [(::)?...]; why: this regex is likely to apply to all data and may break summary indexing, among other Splunk features.
05-06-2024 16:36:21.051 WARN  IConfCache - Stanza has an expansion [script://C:\Program Files\Splunk\etc\apps\TA-Exchange-ClientAccess\bin\exchangepowershell.cmd v14 read-audit-logs_2010_2013.ps1], ignoring alternate expansion [script://C:\Program Files\Splunk\etc\apps\TA-Exchange-Mailbox\bin\exchangepowershell.cmd v14 read-audit-logs_2010_2013.ps1] in inputs.conf
05-06-2024 16:36:21.051 WARN  IConfCache - Stanza has an expansion [script://C:\Program Files\Splunk\etc\apps\TA-Exchange-ClientAccess\bin\exchangepowershell.cmd v15 read-audit-logs_2010_2013.ps1], ignoring alternate expansion [script://C:\Program Files\Splunk\etc\apps\TA-Exchange-Mailbox\bin\exchangepowershell.cmd v15 read-audit-logs_2010_2013.ps1] in inputs.conf
05-06-2024 16:36:21.114 WARN  btool-support - Bad regex value: '(::)?...', of param: props.conf / [(::)?...]; why: this regex is likely to apply to all data and may break summary indexing, among other Splunk features.
05-06-2024 16:36:29.661 WARN  IConfCache - Stanza has an expansion [script://C:\Program Files\Splunk\etc\apps\TA-Exchange-ClientAccess\bin\exchangepowershell.cmd v14 read-audit-logs_2010_2013.ps1], ignoring alternate expansion [script://C:\Program Files\Splunk\etc\apps\TA-Exchange-Mailbox\bin\exchangepowershell.cmd v14 read-audit-logs_2010_2013.ps1] in inputs.conf
05-06-2024 16:36:29.661 WARN  IConfCache - Stanza has an expansion [script://C:\Program Files\Splunk\etc\apps\TA-Exchange-ClientAccess\bin\exchangepowershell.cmd v15 read-audit-logs_2010_2013.ps1], ignoring alternate expansion [script://C:\Program Files\Splunk\etc\apps\TA-Exchange-Mailbox\bin\exchangepowershell.cmd v15 read-audit-logs_2010_2013.ps1] in inputs.conf
05-06-2024 16:36:29.723 WARN  btool-support - Bad regex value: '(::)?...', of param: props.conf / [(::)?...]; why: this regex is likely to apply to all data and may break summary indexing, among other Splunk features.

I already checked the /etc/system/local/web.conf and everything seems fine.

[settings]
enableSplunkWebSSL = 1
httpport = 443

system/default/web.conf

[default]


[settings]

# enable/disable the appserver
startwebserver = 1

# First party apps:
splunk_dashboard_app_name = splunk-dashboard-studio

# enable/disable splunk dashboard app feature
enable_splunk_dashboard_app_feature = true

# port number tag is missing or 0 the server will NOT start an http listener
# this is the port used for both SSL and non-SSL (we only have 1 port now).
httpport = 8000

# this determines whether to start SplunkWeb in http or https.
enableSplunkWebSSL = false

# location of splunkd; don't include http[s]:// in this anymore.
mgmtHostPort = 127.0.0.1:8089

# list of ports to start python application servers on (although usually
# one port is enough)
#
# In the past a special value of "0" could be passed here to disable
# the modern UI appserver infrastructure, but that is no longer supported.
appServerPorts = 8065

any suggestion?

many thanks.

jose

[kiran_panchavat]

@corti77 

Is Splunk web running on the default port (8000)?

netstat -ano | findstr 8000

Are there any firewalls or network configurations blocking access to port 8000?

If the above solution helps, an upvote is appreciated.

[kiran_panchavat]

@corti77 

Check the _internal index for the logs in web_service.log.
Do you see anything prior to the stopping ?

Location: $SPLUNK_HOME/var/log/splunk/web_service.log 

If the above solution helps, an upvote is appreciated. 

[corti77 · Accepted solution]

Thanks for your reply.

At the end, the solution was about to just disable the splunk light forwarder via CLI.

./splunk disable app SplunkLightForwarder 

after this change I restarted splunk service and it worked fine back again.