Splunk Enterprise Security

DomainTools App and TA for Splunk: Why am I unable to save credentials?

panovattack
Communicator

We use Splunk Enterprise Security (which uses SA-DomainTools) for whois. Our API license and key is therefore already in the credential store. The Splunk TA-DomainTools is not allowing me to save the credential:

"Encountered the following error while trying to update: Error while posting to url=/servicesNS/nobody/TA-domaintools/storage/passwords/" 

How do we get the DomainTools App for Splunk and DomainTools TA for Splunk (TA-DomainTools and SA-DomainTools) to work together?

0 Karma
1 Solution

markkendrick
Path Finder

Hi - happy to help with this.

First, you only need the TA-DomainTools app - you don't want the SA-DomainTools app installed at all, or they will conflict. We followed Splunk's lead in switching from an SA to a TA a few versions back; hence the confusion. That's what I would recommend doing first, and please make sure you are using the latest version of our TA and dashboard app.

If that doesn't solve it, be sure you are saving or accessing the credentials as a user with admin privileges. If that works, take a look at the readme file for the latest version of the TA we released - we included a new role which you can give to users so they can access the credential store without full admin privileges.

Then, the next thing to look for is a corrupt credential store. There's a bug in Splunk where that underlying credential store will become corrupted, and when that happens everything that tries to use it will fail. We've submitted that to Splunk for review, but until that's resolved, we've got a work-around that's helped some of our customer. Here's the gist of that.

Diagnose the credential store by running the following curl command:

curl -k -u <username> -p https://<search head hostname/IP>:8089/services/storage/passwords

Credentials should be those of a user on the search head with admin privileges.

If the encrypted credentials system is broken, the response you'll get will be an incomplete XML document. It will abruptly end at a line that looks something like this:

<s:key name="clear_password">^\x01

If you see something like the above, you have a corrupted passwords.conf somewhere. Scroll up a bit in the response to see what app it's in:

<link href="/servicesNS/nobody/Splunk_TA_microsoft-cloudservices/storage/passwords/__REST_CREDENTIAL__%23Splunk_TA_microsoft-cloudservices%23configs%252Fconf-splunk_ta_ms_o365_server_ucc_system_snapshot%23forwarder%3Ausername%3A" rel="remove"/>

In this example, the app just happens to be Splunk_TA_microsoft-cloudservices (it could be anything, including our app). Now just delete $SPLUNK_HOME/etc/apps//local/passwords.conf (where matches what you saw in the href path) and restart Splunk.

Lastly, if all that doesn't work, contact us directly and I'll send you a version that doesn't use the encrypted credential store at all. Some installs never get it to work properly.

Let me know how this goes so I can update the answer if any of it is wrong. And thanks for trying out the app!

View solution in original post

0 Karma

markkendrick
Path Finder

Hi - happy to help with this.

First, you only need the TA-DomainTools app - you don't want the SA-DomainTools app installed at all, or they will conflict. We followed Splunk's lead in switching from an SA to a TA a few versions back; hence the confusion. That's what I would recommend doing first, and please make sure you are using the latest version of our TA and dashboard app.

If that doesn't solve it, be sure you are saving or accessing the credentials as a user with admin privileges. If that works, take a look at the readme file for the latest version of the TA we released - we included a new role which you can give to users so they can access the credential store without full admin privileges.

Then, the next thing to look for is a corrupt credential store. There's a bug in Splunk where that underlying credential store will become corrupted, and when that happens everything that tries to use it will fail. We've submitted that to Splunk for review, but until that's resolved, we've got a work-around that's helped some of our customer. Here's the gist of that.

Diagnose the credential store by running the following curl command:

curl -k -u <username> -p https://<search head hostname/IP>:8089/services/storage/passwords

Credentials should be those of a user on the search head with admin privileges.

If the encrypted credentials system is broken, the response you'll get will be an incomplete XML document. It will abruptly end at a line that looks something like this:

<s:key name="clear_password">^\x01

If you see something like the above, you have a corrupted passwords.conf somewhere. Scroll up a bit in the response to see what app it's in:

<link href="/servicesNS/nobody/Splunk_TA_microsoft-cloudservices/storage/passwords/__REST_CREDENTIAL__%23Splunk_TA_microsoft-cloudservices%23configs%252Fconf-splunk_ta_ms_o365_server_ucc_system_snapshot%23forwarder%3Ausername%3A" rel="remove"/>

In this example, the app just happens to be Splunk_TA_microsoft-cloudservices (it could be anything, including our app). Now just delete $SPLUNK_HOME/etc/apps//local/passwords.conf (where matches what you saw in the href path) and restart Splunk.

Lastly, if all that doesn't work, contact us directly and I'll send you a version that doesn't use the encrypted credential store at all. Some installs never get it to work properly.

Let me know how this goes so I can update the answer if any of it is wrong. And thanks for trying out the app!

0 Karma

panovattack
Communicator

Thanks! Removing SA-DomainTools and recreating the credentials seemed to fix the DomainTools for Splunk App. Does the TA-DomainTools app, now being properly configured, automatically integrate with the Splunk Enterprise Security App (for the Domain Analysis Dashboards)? If not, what are the appropriate setting for the WhoIs-Management input in ES?

0 Karma

markkendrick
Path Finder

Great - glad that worked. Let's see if we can get that dashboard populating for you.

Our app populates the same whois index that ES is looking at, although we also add other fields from the Whois data and our own domain risk score (if you have access to that through our API). That means once we get the populating search working for you, the index should start filling up with data you can use in your own queries, and in that ES dashboard.

You can check to see if data is coming into the whois index with a simple search like "index=whois". If you don't see anything in there it is probably because our populating search isn't finding the right things to enrich for you.

You can see how we're selecting data by going to Settings > Searches, reports & alerts, then setting App context to the one that mentions our TA, and then looking at the "Domain Analysis Datamodel Populator". Take a look at the first part of that search and see if it's likely to find the data on your network that will have domain names in it (by default, we're looking for " | datamodel Web Web search "). That works in most environments that have CIM compliant proxy logs, but we know that's not going to work for everyone, and sometimes you have to tweak that search to make it work.

Let me know if that gets it working for you - I have some tips for other things you can do with Whois data beyond just that dashboard and I want to share those with you once we have this solved.

0 Karma

panovattack
Communicator

I think they are working, though I must admit the dashboard "New Domain Analysis" is generally confusing. For "Newly Registered" the top panels never populate - that only seems to work for "Newly Seen."

I can understand the use of "Web" - is there an easy to way to change that to "Network_Resolution"

Happy to hear the extra tips!

0 Karma

markkendrick
Path Finder

The "Newly Seen" panel is quite helpful because it alerts you to new domain names that are active on your network but that have not been active in the past. It does that with data internally in Splunk and does not use our Whois data for it. The "Newly Registered" is based off the create date in domain name Whois records, and it does need our data for that to work. The fact that that panel is still empty tells me our populating search isn't working yet.

Right now, the best suggestion we know of is for you to change our populating search manually to find data in the Network_Resolution model. I'd suggest editing the first part of our populating search to find your data and rename the column in the manner we're doing, using a basic search to troubleshoot it and be sure it's behaving properly. Focus on editing this part of our search command, being certain that the output has data in a column called "domain":
| datamodel Web Web search | rename "Web.dest" AS domain
Let me know if that works. I'd also be happy to hear ideas from you or anyone reading this of ways we could make this easier for people to adjust without having to edit the search directly.

0 Karma

panovattack
Communicator

Also, I aware of : https://www.domaintools.com/support/splunk/docs

Install TA to indexer(s)

  1. Copy TA-domaintools from SA-domaintools/appserver/addons
  2. Install TA-domaintools to all indexers
  3. Restart indexers

Does that mean copy TA-domaintools OVER SA-domaintools?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...