This was a writeup that I did for this   Backup Splunk Stop and Backup the entire Splunk folder if able. /opt/splunk/bin/splunk stop   tar -zcvf splunk_pre_secret.tar.gz /opt/splunk/etc   Find encrypted passwords find /opt/splunk/etc -name '*.conf' -exec grep -inH '\$[0-9]\$' {} \;   Record the context (file location, stanza, parameter) Can decrypt the hashed passwords with the following /opt/splunk/bin/splunk show-decrypted --value 'PASSWORDHASH'   Updating the splunk.secret Copy the splunk.secret file from 192.168.70.2 to /opt/splunk/etc/auth/splunk.secret on the target system. cp /home/dapslunk/splunk.secret /opt/splunk/etc/auth/splunk.secret   Ensure the permissions are correct 400 ll /opt/splunk/etc/auth/splunk.secret   Update all of the password sections Use the following to find any missed passwords that have not been corrected. find /opt/splunk/etc -name '*.conf' -exec grep -inH '\$[0-9]\$' {} \;   Restart Splunk /opt/splunk/bin/splunk restart   Verify Access to Splunk GUI If any splunk commands that require authentication work Connection to license master /cluster/ deployment server If any inputs have data coming in If LDAP authentication works If all passwords are encrypted. Use the command from before. ... View more

I'm sorry I still have not found the resolution to my issue. I was able to dig deeper at one point and saw timeout messages in the Splunk internal logs. I would also see timeouts when going to the Splunk webpage or any other tools webpage. For some reason though if I move the deployment server to sit outside of our firewall and change the physical ip address of the deployment server to the NAT that was being used on the firewall. All of the Splunk agents can suddenly connect. I believe this to be a network issue, but we have yet to figure out what it is. When I pulled some pcaps I saw every other line was one of the following. TCP Dup ACK, TCP Retransmissions, or TCP Out-of-Order ... View more

Thanks for the reply! I took a look at some of those searches to look for additional messages. I don't think it would be the firewall, because if I change the interval to 30 it can eventually connect to the DS and shows up in the Forwarder Management. I still double checked though and see no blocks and the port is also added in firewalld. For the deployment server side I didn't get any messages from that that search. For the client side I saw the following messages - Attempted handshake xxx times. Will try to re-subscribe to handshake reply - Phonehome thread start, intervals: handshakeRetry=60 phonehome=300.0 - channel=deploymentServer/phoneHome/default Will retry sending phonehome to DS; err=not_connected - channel=tenantService/handshake Will retry handshake message to DS; err=not_connected Also I saw some messages that look related. HTTPPubSubConnection - Unable to parse message from PubSubSvr: Could no obtain connection, will retry after=xxx.xxx seconds. I did a tcpdump and made two different pcaps to look at in wireshark and I kinda wanna say this looks like the client is sending resets before the TLS connection could be finished? Is that what is happening here? interval set to 300 (Bad connection to the ds) client SYN ds SYN, ACK client ACK client TLSv1.2 Client Hello ds ACK ds Server Hello, Certificate, Server Hello Done client ACK client TLSv1.2 Client Key Exchange, Change Cipher Spec, Encrypted Handshake Message ds TLSv1.2 New Session Ticket, Change Cipher Spec, Encrypted Handshake Message client TLSv1.2 Application Data ds ACK client FIN, ACK ds ACK ds TLSv1.2 Application Data client RST ds TLSv1.2 Application Data client RST ds TLSv1.2 Encrypted Alert client RST ds FIN, ACK client RST interval set to 30 (Good connection to ds) client SYN ds SYN, ACK client ACK client TLSv1.2 Client Hello ds ACK ds Server Hello, Certificate, Server Hello Done client ACK client TLSv1.2 Client Key Exchange, Change Cipher Spec, Encrypted Handshake Message ds TLSv1.2 New Session Ticket, Change Cipher Spec, Encrypted Handshake Message client TLSv1.2 Application Data ds TLSv1.2 Application Data ds TLSv1.2 Application Data client ACK client FIN, ACK ds TLSv1.2 Encrypted Alert client RST ds FIN, ACK client RST After all that I went through and started verifying the cipherSuites and sslVersions between the client and ds for web.conf and server.conf which both are using splunks default values. Verified also the date on each server because I saw that could be another issue when dealing with TLS connections. ... View more

I haven't actually built a solution for the Windows side, but Tripwire might be something you may want to look into. The upside to this is it is also available for Linux so this might be useful if you want to only use one solution instead of using both Tripwire and AIDE. I used AIDE for Linux only because it came pre-installed on our systems. ... View more

References: https://docs.splunk.com/Documentation/DBX/3.1.4/DeployDBX/Distributeddeployment jdk1.1.8.0_131 is a Linux RPM and would be installed on the system like so. Keep in mind that the java rpm would need to be install on all instance that is running dbconnect. rpm -i filename.rpm I have only done single instance installs of dbconnect and not on a SHC, but this is what I've found about installing it on a cluster. 1. It looks like the db connect first needs to be setup on the search head deployer. 2. Download the latest version of dbconnect 3. Download and install jdk as mentioned above. 4. Download and install any db drivers required for your deployment. https://docs.splunk.com/Documentation/DBX/3.1.4/DeployDBX/Installdatabasedrivers 5. Logon and ensure that the app is working. 6. Create identities and connections. 7. Copy the dbconnect app from /etc/apps to /etc/shcluster/apps on the deployer. There was an additional mention of two files that would need to be copied over manually from the deployer to the SHC nodes. kerberos_client.conf and identity.dat 8. Issue the deployer command splunk apply shcluster-bundle -target <URI>:<management_port> -auth <username>:<password> ... View more

For Linux I used AIDE and ingested those reports with Splunk to monitor file integrity on systems. ... View more

Not only if you deploy the fields.conf in an app but /etc/system/local as well. The field would show up in a search but as soon as you try to search for a specific field value it would return no results. I had to add the export = system if I was deploying it to /etc/system/local ... View more

That sounds right. I was wondering if there was a way though for a user to get access to just share objects with another group/role but not actually have access to their content. ... View more

DalJeanis thank you for your reply! This was a very informative post and I'll keep your suggestions in mind when posting more questions. ... View more

I did pull over the same Bro app that has all of our parsing inside the app from another one of our Splunk instances. I commented out all of the entries in our transforms.conf file in the Bro app on one of our indexers and tried to search the field bro_smtp in verbose mode and what do you know! It works! I guess now I just need to go back through and figure out which one broke that sourcetype. Thanks! ... View more

Thanks for the response. I did actually try to use the AIDE-TA, but the issue was having to run Splunk as root. That of course was not an option. I took into account your consideration about bringing in the log over and over which makes since. You would want to keep seeing the alerts until you fix them. What I ended up doing was creating a TA that would parse all the events and only bring in those that matched the words changed: removed: added: This actually worked well because now we don't have to deal with all the rest of the junk lines. ... View more