Monitoring Splunk

Splunk 7.0.0 – GUI changes are not reflected in .conf files, deprecated paremters are returning

dfollis
Engager

Wondering if anyone else has seen issues where you modify the changes in the GUI and the applicable .conf files are not updated or modified incorrectly? This is Splunk 7 Enterprise running on Windows server 2016.

First example, spent far too much time trying to get LDAP authentication to work and gave up because we don't have a CA and Splunk didn't like the DC self-signed certs we are using, despite us importing them as trusted to the Windows Server and Splunk Trusted Root stores. Decided to use SAML via Okta instead, but while trying to figure that out, we noticed these errors in the log:

11-22-2017 18:09:35.434 -0800 WARN SSLOptions - authentication.conf/[saml]/sslKeysfile: deprecated; use 'clientCert' instead
11-22-2017 18:09:35.434 -0800 WARN SSLOptions - authentication.conf/[saml]/sslKeysfilePassword: deprecated; use 'sslPassword' instead

We modified as instructed, but every time we had to make a change in the GUI, it changed the suggested parameters back to the depreciated ones. We finally got it working, but it was a major PITA.

More recently we are using a Universal Forwarder install on a dedicated Win2016 VM to ingest Windows Forwarded Events. GPO configures PCs to forward events to Win2016 VM running Universal Forwarder. The events are forwarded to Splunk but they always go to the "main" index. We've tried everything to get them to go to a dedicated index like our other numerous SYSLOG sources. It is very frustrating and I suspect this might be a related issue where despite us seeing the Win2016 host as a new data source, selecting Forwarded Events, and choosing a new index, those settings don't get updated in the proper conf file (we aren't sure where that is for the UF inputs). This is despite the GUI showing all indications that is how it is configured.

We have twice removed UF and all configs on Splunk and the source server and tried to recreate to no avail. Splunk GUI shows that data source is associated with index "wef" but searching index=wef shows nothing. Searching index=main shows the events.

As a last result we tried to manually update the inputs.conf file but that didn't work after a Splunkd restart.

We are opening a ticket, but super frustrating.

0 Karma

dfollis
Engager

@ravitejaj

I think this is more of a log cluttering issue than performance impacting. Not sure if you are running Splunk on nix or Windows. We are running it on Win Svr 2016. The fix is to modify those fields in the authentication.conf file in /local. Make sure you edit the correct one as there are multiple copies of it in multiple locations.

The odd behavior is that if you edit the file directly to the proper parameter names and then attempt to make a change in the GUI, it modifies the .conf file back to the depreciated parameters. You should be able to modify the .conf file directly and then restart Splunk to see if it works.

Are you using OKTA? We found the following to be most helpful.

  1. Use Splunk to monitor "index=_*" real-time when making authentication attempts to see which errors are being generated.
  2. Use a SAML debugger such as https://chrome.google.com/webstore/detail/saml-message-decoder/mpabchoaimgbdbbjjieoaeiibojelbhm?hl=e... with Chrome. This gives you insight as to why the attempt might be failing.

In our case with OKTA, using the above tool allowed us to identify that we had the IDP string mis-configured and what Splunk was expecting was not what OKTA was providing.

0 Karma

ravitejaj
Explorer

Can you please let me know, how you were able to manage the below errors? I'm currently stuck up with the same.

11-22-2017 18:09:35.434 -0800 WARN SSLOptions - authentication.conf/[saml]/sslKeysfile: deprecated; use 'clientCert' instead
11-22-2017 18:09:35.434 -0800 WARN SSLOptions - authentication.conf/[saml]/sslKeysfilePassword: deprecated; use 'sslPassword' instead

0 Karma

hardikJsheth
Motivator

Is this standalone environment or clustered environment?

0 Karma

dfollis
Engager

Stand Alone

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...