Thank you for the response! The queries are all pretty basic for the most part, one would be: index="wineventlog" EventID=4740 My current workaround is two alerts, one for zero results, one for more than zero. Do you think I could do it in one?   ... View more

I have a Splunk Cloud trial. I have followed the instructions to install Splunk Enterprise as a Deployment Server on Server 2019 and 5 total Universal Forwarders, also Server 2019 machines. I have deployed the add on for Windows and UniversalForwarders to all clients. The clients all show up in "Settings > Forwarder management" on the deployment server and appear to be talking to it via the logs. At the moment I only have logs from the deployment server showing up. I'm trying to get windows event log data from all clients into the Splunk Cloud instance. The command "splunk list forward-server" on any of the client machines will not get a response, it simply hangs. On the deployment server, that command returns the cloud instance. ... View more

I am not. I wasn't sure how to go about getting the license for a Cloud free trial. Didn't seem to be geared toward Cloud trials. Thank you for all of your help today. I do appreciate it. ... View more

See below for the section and the link where I found it. I only made use of the server and DomainController labels but I'm not even sure if they're doing anything. When I do that search on the cloud instance, I find millions of Splunk cloud entries. If I do it on my local server, I get the same 11K data entries I've had for a bit now. If I exclude cloud data from the search, I get the same limited deployment server data that I have locally. No data from the forwarders in my network. https://docs.splunk.com/Documentation/SplunkCloud/8.2.2202/Admin/WindowsGDI Configure and customize the Windows data collection add-ons Navigate to Windows > Program Files > Splunk > etc > deployment-apps. Make copies of the Splunk_TA_windows folder for each of the types of Windows instances that you want to get data from. Rename each of the folders so that they represent your different Windows servers. For this example, create the following folders: Splunk_TA_windows_DomainController Splunk_TA_windows_server Splunk_TA_windows_client Splunk_TA_windows_GlobalCatalogServer ... View more

Unfortunately nothing changed with the data. I was able to get the inputs list tool to work though. I think there's something in how I did the deployment server / indexes that is fouled up. When I deployed the Windows app it had me create multiple folders like "Splunk_TA_windows_server" and "Splunk_TA_DomainController" so I have inputs.conf everywhere and no data from the forwarders in Splunk. I have a list of inputs that I printed to PDF to attach here as well.     ... View more

I may be misunderstanding your instructions a bit. You're saying to take the contents of the "deployment-apps" folder on the deployment server and put them on one of the universal forwarder servers in the "apps" folder. This is essentially what the deployment server was meant to do but we're doing it manually? Just want to make sure I'm doing the right process. Thank you! ... View more

Well, the inputs.conf has thoroughly confused me. I've edited an inputs.conf for apps and in the local folder of deployed apps and I'm not sure if any of them are doing anything. In C:\Program Files\Splunk\etc\deployment-apps\Splunk_TA_windows\local\inputs.conf (and other deployed apps) I have:   $SPLUNK_HOME/etc/apps/Splunk_TA_windows/local. ## To make changes, copy the section/stanza you want to change from $SPLUNK_HOME/etc/apps/Splunk_TA_windows/default ## into ../local and edit there. ## ###### OS Logs ###### [WinEventLog://Application] disabled = 0 start_from = oldest current_only = 0 checkpointInterval = 5 renderXml=true index=wineventlog [WinEventLog://Security] disabled = 0 start_from = oldest current_only = 0 evt_resolve_ad_obj = 1 checkpointInterval = 5 blacklist1 = EventCode="4662" Message="Object Type:(?!\s*groupPolicyContainer)" blacklist2 = EventCode="566" Message="Object Type:(?!\s*groupPolicyContainer)" renderXml=true index=wineventlog [WinEventLog://System] disabled = 0 start_from = oldest current_only = 0 checkpointInterval = 5 renderXml=true index=wineventlog     ... View more

Below is what I get in the logs now. 05-10-2022 11:12:17.140 -0400 INFO DC:DeploymentClient [9488 MainThread] - Starting phonehome thread. 05-10-2022 11:12:17.140 -0400 INFO DS_DC_Common [9488 MainThread] - Deployment Client initialized. 05-10-2022 11:12:17.140 -0400 INFO ServerRoles [9488 MainThread] - Declared role=deployment_client. 05-10-2022 11:12:17.140 -0400 INFO DS_DC_Common [9488 MainThread] - Deployment Server not available on a dedicated forwarder. 05-10-2022 11:12:17.140 -0400 INFO DC:PhonehomeThread [8536 PhonehomeThread] - Phonehome thread start, intervals: handshakeRetry=12.0 phonehome=60.0. 05-10-2022 11:12:17.140 -0400 INFO ClusteringMgr [9488 MainThread] - initing clustering with: ht=60.000 rf=3 sf=2 ct=60.000 st=60.000 rt=60.000 rct=5.000 rst=5.000 rrt=10.000 rmst=600.000 rmrt=600.000 icps=25 sfrt=600.000 pe=1 im=0 ip=0 mob=5 mor=5 mosr=5 pb=5 rep_port= pptr=10 pptrl=100 fznb=10 Empty/Default cluster pass4symmkey=false allow Empty/Default cluster pass4symmkey=true rrt=restart dft=180 abt=600 sbs=1 05-10-2022 11:12:17.140 -0400 INFO DC:DeploymentClient [8536 PhonehomeThread] - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected 05-10-2022 11:12:17.140 -0400 INFO ClusteringMgr [9488 MainThread] - clustering disabled 05-10-2022 11:12:17.140 -0400 WARN SHCConfig [9488 MainThread] - Default pass4symkey is being used. Please change to a random one. 05-10-2022 11:12:17.140 -0400 INFO SHClusterMgr [9488 MainThread] - initing shpooling with: ht=60.000 rf=3 ct=60.000 st=60.000 rt=60.000 rct=5.000 rst=5.000 rrt=10.000 rmst=600.000 rmrt=600.000 pe=1 im=0 is=0 mor=5 pb=5 rep_port= pptr=10 05-10-2022 11:12:17.140 -0400 INFO SHClusterMgr [9488 MainThread] - shpooling disabled 05-10-2022 11:12:17.140 -0400 INFO WorkloadManager [9488 MainThread] - Workload management cannot be enabled on this system because the feature is not supported. Check the status of workload management preflight checks for additional information. 05-10-2022 11:12:17.155 -0400 INFO loader [9488 MainThread] - win-service: Windows service is now in running state. 05-10-2022 11:12:17.155 -0400 INFO ApplicationLicense [12132 AppLicenseThread] - app license disabled by conf setting. 05-10-2022 11:12:17.233 -0400 INFO loader [9488 MainThread] - SAML cert db registration with KVStore failed 05-10-2022 11:12:17.233 -0400 INFO CertStorageProvider [9488 MainThread] - Updating status from unknown to unknown 05-10-2022 11:12:17.233 -0400 INFO loader [9488 MainThread] - Auth cert db registration with KVStore failed 05-10-2022 11:12:17.233 -0400 INFO CertStorageProvider [9488 MainThread] - Updating status from unknown to unknown 05-10-2022 11:12:17.233 -0400 INFO Rsa2FA [9488 MainThread] - Could not find [externalTwoFactorAuthSettings] in authentication stanza. 05-10-2022 11:12:17.233 -0400 INFO loader [9488 MainThread] - JsonWebToken Manager registration with KVStore failed. 05-10-2022 11:12:17.233 -0400 INFO IndexerInit [11632 SplunkdSpecificInitThread] - running splunkd specific init 05-10-2022 11:12:17.249 -0400 INFO IntrospectionGenerator:disk_objects [11632 SplunkdSpecificInitThread] - Enabled: disk_objects=false indexes=false volumes=false dispatch=false fishbucket=true partitions=false summaries=false distributedIndexes=false 05-10-2022 11:12:17.249 -0400 INFO IntrospectionGenerator:disk_objects [11632 SplunkdSpecificInitThread] - I-data gathering (Disk Objects) starting; period=600.000s 05-10-2022 11:12:17.249 -0400 INFO loader [9488 MainThread] - Initializing from configuration 05-10-2022 11:12:17.249 -0400 INFO ChunkedLBProcessor [14548 parsing] - Initializing the chunked line breaking processor 05-10-2022 11:12:17.249 -0400 INFO TcpOutputProc [14548 parsing] - Initializing with fwdtype=lwf 05-10-2022 11:12:17.249 -0400 INFO TcpOutputProc [14548 parsing] - found Whitelist forwardedindex.0.whitelist , RE : .* 05-10-2022 11:12:17.249 -0400 INFO TcpOutputProc [14548 parsing] - found Blacklist forwardedindex.1.blacklist , RE : _.* 05-10-2022 11:12:17.249 -0400 INFO TcpOutputProc [14548 parsing] - found Whitelist forwardedindex.2.whitelist , RE : (_audit|_introspection|_internal|_telemetry) 05-10-2022 11:12:17.249 -0400 INFO TcpOutputProc [14548 parsing] - Initializing connection for non-ssl forwarding to 192.168.0.2:9997 05-10-2022 11:12:17.249 -0400 INFO TcpOutputProc [14548 parsing] - tcpout group group1 using Auto load balanced forwarding 05-10-2022 11:12:17.249 -0400 INFO AutoLoadBalancedConnectionStrategy [14548 parsing] - Group group1 initialized with maxQueueSize=512000 in bytes. 05-10-2022 11:12:17.249 -0400 INFO AutoLoadBalancedConnectionStrategy [14548 parsing] - Group group1 initialized with autoLBFrequency=30.000 05-10-2022 11:12:29.153 -0400 INFO DC:DeploymentClient [8536 PhonehomeThread] - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected 05-10-2022 11:12:41.165 -0400 INFO DC:DeploymentClient [8536 PhonehomeThread] - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected 05-10-2022 11:12:47.133 -0400 INFO AutoLoadBalancedConnectionStrategy [3212 TcpOutEloop] - Connected to idx=192.168.0.2:9997, pset=0, reuse=0. 05-10-2022 11:12:47.242 -0400 INFO ScheduledViewsReaper [12532 DispatchReaper] - Scheduled views reaper run complete. Reaped count=0 scheduled views 05-10-2022 11:12:47.242 -0400 INFO CascadingReplicationManager [12532 DispatchReaper] - Using value for property max_replication_threads=2. 05-10-2022 11:12:47.242 -0400 INFO CascadingReplicationManager [12532 DispatchReaper] - Using value for property max_replication_jobs=5. 05-10-2022 11:12:47.242 -0400 INFO FileAndDirectoryEliminator [12532 DispatchReaper] - Enabled 05-10-2022 11:12:53.178 -0400 INFO DC:DeploymentClient [8536 PhonehomeThread] - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected 05-10-2022 11:13:05.191 -0400 INFO DC:DeploymentClient [8536 PhonehomeThread] - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected 05-10-2022 11:13:06.441 -0400 INFO ProxyConfig [14352 HttpClientPollingThread_C7BD701A-F102-461C-8FA1-9B5D6DC14779] - Failed to initialize http_proxy from server.conf for splunkd. Please make sure that the http_proxy property is set as http_proxy=http://host:port in case HTTP proxying needs to be enabled. 05-10-2022 11:13:06.441 -0400 INFO ProxyConfig [14352 HttpClientPollingThread_C7BD701A-F102-461C-8FA1-9B5D6DC14779] - Failed to initialize https_proxy from server.conf for splunkd. Please make sure that the https_proxy property is set as https_proxy=http://host:port in case HTTP proxying needs to be enabled. 05-10-2022 11:13:06.441 -0400 INFO ProxyConfig [14352 HttpClientPollingThread_C7BD701A-F102-461C-8FA1-9B5D6DC14779] - Failed to initialize the proxy_rules setting from server.conf for splunkd. Please provide a valid set of proxy_rules in case HTTP proxying needs to be enabled. 05-10-2022 11:13:06.441 -0400 INFO ProxyConfig [14352 HttpClientPollingThread_C7BD701A-F102-461C-8FA1-9B5D6DC14779] - Failed to initialize the no_proxy setting from server.conf for splunkd. Please provide a valid set of no_proxy rules in case HTTP proxying needs to be enabled. 05-10-2022 11:13:06.456 -0400 INFO HttpPubSubConnection [14352 HttpClientPollingThread_C7BD701A-F102-461C-8FA1-9B5D6DC14779] - SSL connection with id: connection_192.168.0.1_8089_SERVER.domain.com_SERVER_C7BD701A-F102-461C-8FA1-9B5D6DC14779 05-10-2022 11:13:06.456 -0400 INFO HttpPubSubConnection [14352 HttpClientPollingThread_C7BD701A-F102-461C-8FA1-9B5D6DC14779] - Running phone uri=/services/broker/phonehome/connection_192.168.0.1_8089_SERVER.domain.com_SERVER_C7BD701A-F102-461C-8FA1-9B5D6DC14779 05-10-2022 11:13:16.985 -0400 INFO AutoLoadBalancedConnectionStrategy [3212 TcpOutEloop] - Found currently active indexer. Connected to idx=192.168.0.2:9997, reuse=1. 05-10-2022 11:13:17.204 -0400 INFO HttpPubSubConnection [14352 HttpClientPollingThread_C7BD701A-F102-461C-8FA1-9B5D6DC14779] - Running phone uri=/services/broker/phonehome/connection_192.168.0.1_8089_SERVER.domain.com_SERVER_C7BD701A-F102-461C-8FA1-9B5D6DC14779 05-10-2022 11:13:17.204 -0400 INFO DC:HandshakeReplyHandler [14352 HttpClientPollingThread_C7BD701A-F102-461C-8FA1-9B5D6DC14779] - Handshake done. ... View more

I apologize for any confusion of terms. I have a deployment server acting as an indexer and three universal forwarders in addition. I am not well versed in Splunk terms and wasn't aware that I didn't need a deployment server. As far as multiple inputs.conf, I was moreso referring to them being in many locations, not having duplicate entries in any one location. The command did not return any data. Thank you for your help.       ... View more

  # Version 8.2.6 # # This file contains an example outputs.conf. Use this file to configure # forwarding in a distributed set up. # # To use one or more of these configurations, copy the configuration block into # outputs.conf in $SPLUNK_HOME/etc/system/local/. You must restart Splunk to # enable configurations. # # To learn more about configuration files (including precedence) please see the # documentation located at # http://docs.splunk.com/Documentation/Splunk/latest/Admin/Aboutconfigurationfiles # Specify a target group for an IP:PORT which consists of a single receiver. # This is the simplest possible configuration; it sends data to the host at # 10.1.1.197 on port 9997. [tcpout:group1] server=192.168.0.2:9997   The above is what I've got. I simply took the first example and put it in. Server IP is a pretty generic internal IP so no worries there. Good luck getting to it. 🙂 Thank you very much for your help. It means a lot. Also, when I attempt the CLI command, it just hangs. Never completes.   ... View more

@venky1544 wrote: 3) check if the path are correctly configured in the monitor stanza in inputs.conf .if you are just uploading the file then don't need to look into inputs.conf  and do check the index as well by playing with alltime options  I'm unsure of the part above but I did verify that 9997 was configured on the Splunk deployment server for receiving data. It was already there, I didn't add it. I have messed with so many inputs.conf files that I'm not sure which are the effective ones. I have tried a test_index and the wineventlog index but nothing from the Server 2019 servers is making it into the deployment server or cloud. Thanks for your help!  ... View more

I put the simple one liner outputs.conf in $SPLUNK_HOME/etc/system/local/ Restarted Splunk and all the forwarders have this entry, which I've anonymized a bit, replacing SERVER_IP and SERVER in place of its name. So this would appear to be successful phoning home? To be clear though, still not getting data from any hosts others than the deployment server. It's odd. Thank you for your help.   05-10-2022 10:19:21.447 -0400 INFO HttpPubSubConnection [6008 HttpClientPollingThread_873E6E32-D1FD-427B-A82D-C1D92C0D4E1E] - Running phone uri=/services/broker/phonehome/connection_SERVER_IP_8089_SERVER.DOMAIN.COM_SERVER_873E6E32-D1FD-427B-A82D-C1D92C0D4E1E ... View more