If you open up the indexes.conf file (Most likely in C:\Program Files\Splunk\etc\system\local) it will have the names of the indexes in your environment.  Even if you mainly utilize one index, there are other indexes that Splunk uses. (_audit, _internal, for example) Run this search in Splunk | eventcount summarize=f index=* index=_* | dedup index | fields index Everything in the indexes column is a type of index you have.   Example if your index is named "main" then the command you would run is  splunk _internal call /data/indexes/main/roll-hot-buckets –auth admin : password       ... View more

You will need to run the command  splunk _internal call /data/indexes/<index_name>/roll-hot-buckets –auth <admin_username>:<admin_password>  This must be done for every index you have.  This will roll your indexes to warm.  Read the section about Working with the CLI on Windows here https://docs.splunk.com/Documentation/Splunk/8.2.5/Admin/AbouttheCLI     These are the steps to migrate the indexes again  How to move index buckets from one host to another If you want to retire a Splunk Enterprise instance and immediately move the data to another instance, you can move individual buckets of an index between hosts, as long as: When you copy individual bucket files, you must make sure that no bucket IDs conflict on the new system. Otherwise, Splunk Enterprise does not start. You might need to rename individual bucket directories after you move them from the source system to the target system. 1. Roll any hot buckets on the source host from hot to warm. 2. Review indexes.conf on the old host to get a list of the indexes on that host. 3. On the target host, create indexes that are identical to the ones on the source system. 4. Copy the index buckets from the source host to the target host. Restart Splunk Enterprise.    Your indexes.conf (mentioned in step 2) is an example of one of the conf files you need to migrate over. This might be in your C:\Program Files\Splunk\etc\system\local directory or it could be elsewhere. Taking an inventory of the conf files and their locations will help you in the long run.   And for your question:  IF it is......for the $SPLUNK_HOME/var/lib/splunk (just the highlighted portion will be copied?) or will I be able to go one directory higher (copy the entire lib folder over and replace)   Sure you can copy the entire lib folder if you wish. On my installation the only folder under 'lib' is 'splunk' 🙂     ... View more

It is active.   How do I obtain the Splunk Add-on for Splunk UBA? The Splunk Add-on for UBA is not available for download on Splunkbase. The add-on is installed by default with Splunk Enterprise Security (ES). If you find that the Splunk Add-on for UBA is not installed, run the Splunk Enterprise Security Post-Install Configuration again and ensure that Splunk_TA_ueba is selected for installation. See Install Splunk Enterprise Security in the Splunk Enterprise Security Installation and Upgrade manual. ... View more

On the page you took that screenshot there should be a "Usage Report" button. This will take you to a set of dashboards that will let you look at how much you've ingested over time. On one or more days within the last 30-60 days you ingested more than 1024 MB of data.  You have a "No enforcement" License meaning after 30 days your searches will not be disabled but if you are constantly having this error message it might be time to look at increasing your license.  ... View more

1. Yes those apps that come with Splunk Enterprise Security will be updated after you install a new version of Enterprise Security and then run through the Configuration pages.   2. Yes. Your best bet to find out if an app is built into Splunk by looking at the Version number. The Version number is the same as the version number of your Splunk Enterprise installation. ... View more

It's saying you're ingesting more data than your license allows you to.  From Splunk: If you exceed license capacity 45 or more times in a rolling 60-day period and have a license stack volume less than 100 GB on Splunk Enterprise version 8.1.0 and above, the software will disable your search functionality. In cases where license pools have strict quota enabled, search is disabled for the offending license pool member(s) after 45 warnings over a rolling 60-day window for the license pool.     ... View more

That should be fairly simple.  Depending on your original installation you may have some conf files placed in different directories.    Sound's like it might be less of a hassle if you freshly install the same version of Splunk you are running currently onto the D drive, let it start up to create the default conf files, and then stop splunk and copy over your configurations/apps. I'd start by copying everything from C:\Program Files\Splunk\etc\apps\* to D:\Splunk\etc\apps  C:\Program Files\Splunk\etc\licenses\* to D:\Splunk\etc\licenses and copy over C:\Program Files\Splunk\etc\users\* to D:\Splunk\etc\users   After those three directories are copied over, you will need to copy the conf files in C:\Program Files\Splunk\etc\system\local  ... You can not do a straight copy over as some of these files are specific to the directory they're installed in.  My suggestion is to open each one and bring over the configurations that do not mention the old Server name or a password. Ensure that the configurations to connect to your Indexer are brought over. ... View more

These are the instructions I followed from Splunk when I did my upgrade:   Spot check Search Head functionality    Migrate Core Splunk Search Head             Stop Core Splunk SH             Straight copy Splunk folder to new Core Splunk SH (This is the C:\Program Files\Splunk folder)             Ensure licensing configuration was migrated             Modify server.conf -                         under [general] stanza                         Set servername to new server.                         clear text on pass4symmkey {default is changeme}                         under [sslConfig] stanza                         clear text sslPassword {default is password}                         under [clustering]                         clear text on pass4symmkey             Insert latest Web Console certificates (Prepare certificates)             Start new Core Splunk SH             Validate SH functionality   Some of it  might not apply for your environment such as the clustering stanza.    We had a indexer cluster so migrating an Indexer was easy. But since you have only one indexer the steps here https://docs.splunk.com/Documentation/Splunk/8.2.5/Installation/MigrateaSplunkinstance seem easy enough  Roll any hot buckets on the source host from hot to warm. Review indexes.conf on the old host to get a list of the indexes on that host. On the target host, create indexes that are identical to the ones on the source system. Copy the index buckets from the source host to the target host. Restart Splunk Enterprise.      Hope this helps and gives you a bit more guidance. ... View more

Yes the C:\Program Files\Splunk folder is essentially  your "Home" folder.    We migrated the Search Heads, Cluster Master, Deployment Server, and Indexers to new servers by coping over the Splunk folder. (Albeit, ours was Linux so '/opt/splunk')   Personally we decided to upgrade our version of Splunk to the latest version so that we would be able to receive assistance from Splunk if need be.     How many servers are you migrating and what are their roles (Search Head, Indexer, Cluster Master, etc)?   ... View more

Are your IP addresses internal? The iplocation command utilizes information from 3rd party databases to tell it where the IPs are originating from.  Your internal IPs will not work with the IP location command.   Try this command. Its an example from Splunk's Documentation | makeresults | eval myip="2001:4860:4860::8888" | iplocation myip If it returns the Country as United Stats and the latitude/longitude then the iplocation command works in your environment.   https://docs.splunk.com/Documentation/Splunk/8.2.5/SearchReference/Iplocation ... View more

Ah okay that helps me out. How many assets are you looking to collect data from? You have the option to install Splunk Forwarder on one windows machine and set it up to collect WMI data remotely from other windows machines.  But if there are a lot of assets, you also have the option to install Splunk Forwarder on every windows machine and collect WMI data that way. https://docs.splunk.com/Documentation/Splunk/latest/Data/MonitorWMIdata Splunk does offer an add-on to automatically set up and collect Windows data. After installing the Splunk Forwarder you would install this app here https://splunkbase.splunk.com/app/742/ and then turn on which items you want it to report back to Splunk.  To manage these configurations (WMI or add-on) Splunk can act as a deployment server which will push out your requested configurations to each forwarder. That way you don't have to make the changes on every server.   Hope this helps, let me know if I need to clarify anything. https://docs.splunk.com/Documentation/Forwarder/8.2.5/Forwarder/HowtoforwarddatatoSplunkEnterprise       ... View more

This search utilizes metrics data.  Do you know if your environment is already collecting metrics data? What about the resource usage (CPU usage, etc)? ... View more

This has me stumped. 🤔 @yj055 wrote: Again " TcpOutputProc" start appearing - reset with 10 seconds  03-28-2022 11:26:11.411 +0100 WARN TcpOutputProc - Applying quarantine to ip=xx.yy.zz.aa  port=xxxx _numberOfFailures=2   Was this one of the correct IP/port combinations for your indexer or are you using indexer discovery? ip=xx.yy.zz.aa port=xxxx      Maybe you can try to completely uninstall the forwarder and reinstall.   ... View more

Thank you that's helpful. Can you read splunkd.log while you restart the splunkforwarder service on the affected machines. What does it say before it starts saying "Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group index-peers has been blocked"? ... View more

I'd reinstall 8.2.5 on top of it and see if it would resolve it? Is this the only server that's affected?  Just to make sure.. can you cd to    /opt/splunk/.splunk   Just to see if that directory exists.     What happens when you do the command    cd $SPLUNK_HOME   Does it take you to /opt/splunk or does it throw an error for /opt/splunk/splunk   ... View more

It looks like it thinks your splunk environment was installed in /opt/splunk/splunk ... Can you check to see what your $SPLUNK_HOME variable is set to?  If you open /opt/splunk/etc/splunk-launch.conf do you see SPLUNK_HOME=/opt/splunk? ... View more

Try this index=app_events_fx4cash_uk_prod source=*STPManager-servicemanagement.20220303-100818.log* |rex (?<Error>\[(.*)) |table Error   I also noticed in your search query, your rex command assigns it to "Error" but your table says "error".  Field names are case-sensitive. ... View more

This sounds more of a network connectivity problem... Are you able to test port 9997 on the affected forwarders to ensure it's open? Could it be that these affected servers are on a different VLAN? Are the indexers listed in your outputs.conf listed as FQDNs or IPs? Can you run the nslookup command on those indexers from your forwarders?   ... View more

@shocko wrote: So I have verified the indexer is listening on TCP 9998 and has my PKI certificate bound to it. I don't understand why the forwarder needs any client certificates to use SSL. I just want to ensure the data is forwarded over SSL. Why would I need a client certificate for this?  For the SSL connection to the indexers the forwarder requires a certificate. The clientCert is used to "turn on" SSL connections. That's my assumption.  You can use the certificate you created for your indexers to use on your forwarders.   @shocko wrote: I'm using deployment server to deploy my apps so I also don't want to specify the password for the client certificate private key in the outputs.conf.  Splunk doesn't support setting up SSL certificates in apps for this very reason anymore. It took me a long time of trial and error before someone @ Splunk told me this. You'll need to place your certificate somewhere in $SPLUNK_HOME/etc/auth/(folder) and your outputs.conf in $SPLUNK_HOME/etc/system/local       Touching back on your error you received on the splunkd.log on your forwarder, if you restart the indexer do you see where your indexer is successfully accepting SSL? It might say something like  port 9998 is reserved for splunk 2 splunk (SSL)       ... View more

@shocko  I had similar problems with my set up for SSL.  Are you able to run the command: >openssl.exe rsa -in "C:\Program Files\Splunk\etc\auth\mycert\my.pem" -text   Try following the steps listed here if you haven't https://docs.splunk.com/Documentation/Splunk/8.2.5/Security/Troubleshootyouforwardertoindexerauthentication ... View more

My settings for the Network traffic data model is the same as the screenshot above. The only difference is after I used the wildcard to include all indexes, waited to see what it detected, then I modified the settings to only include the indexes it was aligning to the data model using the search: (`cim_Network_Traffic_indexes`)   As for your tag whitelist. I removed everything. I didn't understand why those specific tags were there when if you go to the Data Model page in Splunk and look at the Data Models, those tags did not align with the tags it wants. For example the "All Traffic" search for the Network Traffic data model is  (`cim_Network_Traffic_indexes`) tag=network tag=communicate   Those two tags weren't in the tag whitelist in the CIM settings. Try removing them too. Just make sure you have a screenshot or a backup before you do, incase you need them back.   ... View more

Interesting.  I had the same problem with my CIM for Enterprise Security. The index whitelist was blank and I wasn't getting events anymore.  In the Indexes Whitelist field, can you put * ? Here's an example of mine.   Make that change and let it sit for a few minutes to run. Then try searching that datamodel using a search like  (`cim_Authentication_indexes`)         After verifying data is coming in, you can manually specify the indexes in the Indexes Whitelist by what's showing up using the wildcard.     Let me know how this works for you. ... View more