Good Morning,
I'm trialing Splunk Cloud in anticipation of a purchase. I have installed Splunk Enterprise as the deployment server and universal forwarders on three servers. My clients are showing up in "Forwarder Management" but I can't seem to get event logs from any servers except the deployment server. I have enabled firewall ports outbound 8089 and inbound 9997 on the deployment server. These are all Server 2019 machines.
I have verified inputs.conf is pointing event logs to index:wineventlog but that index locally has 0 results and about 112,000 results on the cloud server.
I'm sure it's something simple I'm missing with all the moving parts. Thank you in advance!
Hi @theitgui it seems there is lot of confusion with your terminology let take a step at a time
1) are you using splunk cloud ?? or you have installed splunk enterprise on EC2 instance in aws or Azure amd you are calling it as splunk cloud ?? please clarify
I have a Splunk Cloud trial. I have followed the instructions to install Splunk Enterprise as a Deployment Server on Server 2019 and 5 total Universal Forwarders, also Server 2019 machines.
I have deployed the add on for Windows and UniversalForwarders to all clients. The clients all show up in "Settings > Forwarder management" on the deployment server and appear to be talking to it via the logs.
At the moment I only have logs from the deployment server showing up. I'm trying to get windows event log data from all clients into the Splunk Cloud instance.
The command "splunk list forward-server" on any of the client machines will not get a response, it simply hangs. On the deployment server, that command returns the cloud instance.
Hi @theitgui ,
I'm assuming the deployment server is acting as your indexer as well?
When setting up your forwarders, did you give them an outputs.conf to tell them to send log data to your deployment server?
https://docs.splunk.com/Documentation/Splunk/8.2.6/Admin/Outputsconf
Also, check your logs in C:\Program Files\SplunkUniversalForwarder\var\log\splunkd.log on your server 2019 machines. There should be a clue as to not having a connection to send logs to.
Hope that helps!
Hi @theitgui
you probably have to do some traceback steps
1) you have enabled inbound 9997 on deployment server i assume you did it on the VPC but did you configure in your splunk server under settings >>Forwarding and recieving >> Recieve data
you have to add 9997
2) i guess outputs.conf is correct thats why they are showing up in the forwarder maangement
3) check if the path are correctly configured in the monitor stanza in inputs.conf .if you are just uploading the file then don't need to look into inputs.conf and do check the index as well by playing with alltime options
Thanks
venky
if it helps karma points are appreciated/if it resolves acceptance of solution is appreciated
@venky1544 wrote:3) check if the path are correctly configured in the monitor stanza in inputs.conf .if you are just uploading the file then don't need to look into inputs.conf and do check the index as well by playing with alltime options
I'm unsure of the part above but I did verify that 9997 was configured on the Splunk deployment server for receiving data. It was already there, I didn't add it. I have messed with so many inputs.conf files that I'm not sure which are the effective ones. I have tried a test_index and the wineventlog index but nothing from the Server 2019 servers is making it into the deployment server or cloud. Thanks for your help!
firstly can you clarify do you have seperate indexer and deployment server ?? or is it one splunk standalone server ?? where you have everything in one server thats bit confusing when you say deployment server
secondly if if you have setup the servers then it can't be already there it always need to be configured please see the screenshot and configure 9997
and if you have multiple inputs.conf try to delete the duplicates or you should use the btool command to see which inputs.conf is taking the precedence and probably thats the culprit where you might have to change the index name
./splunk btool <configFileName> list --debug
I apologize for any confusion of terms. I have a deployment server acting as an indexer and three universal forwarders in addition. I am not well versed in Splunk terms and wasn't aware that I didn't need a deployment server.
As far as multiple inputs.conf, I was moreso referring to them being in many locations, not having duplicate entries in any one location. The command did not return any data. Thank you for your help.
I put the simple one liner outputs.conf in $SPLUNK_HOME/etc/system/local/
Restarted Splunk and all the forwarders have this entry, which I've anonymized a bit, replacing SERVER_IP and SERVER in place of its name. So this would appear to be successful phoning home?
To be clear though, still not getting data from any hosts others than the deployment server. It's odd. Thank you for your help.
05-10-2022 10:19:21.447 -0400 INFO HttpPubSubConnection [6008 HttpClientPollingThread_873E6E32-D1FD-427B-A82D-C1D92C0D4E1E] - Running phone uri=/services/broker/phonehome/connection_SERVER_IP_8089_SERVER.DOMAIN.COM_SERVER_873E6E32-D1FD-427B-A82D-C1D92C0D4E1E
@theitgui wrote:I put the simple one liner outputs.conf in $SPLUNK_HOME/etc/system/local/
Could you post an anonymized version of your outputs.conf?
Maybe using the CLI would be easier
./splunk add forward-server <host name or ip address of your splunk server>:9997
To run CLI commands in Splunk Enterprise on Windows, use PowerShell or the command prompt as an administrator.
1. Open a PowerShell window or command prompt as an administrator.
2. Change to the Splunk Forwarder bin directory.
3. Run a Splunk command by typing in splunk followed by the subcommand and any required arguments.
# Version 8.2.6
#
# This file contains an example outputs.conf. Use this file to configure
# forwarding in a distributed set up.
#
# To use one or more of these configurations, copy the configuration block into
# outputs.conf in $SPLUNK_HOME/etc/system/local/. You must restart Splunk to
# enable configurations.
#
# To learn more about configuration files (including precedence) please see the
# documentation located at
# http://docs.splunk.com/Documentation/Splunk/latest/Admin/Aboutconfigurationfiles
# Specify a target group for an IP:PORT which consists of a single receiver.
# This is the simplest possible configuration; it sends data to the host at
# 10.1.1.197 on port 9997.
[tcpout:group1]
server=192.168.0.2:9997
The above is what I've got. I simply took the first example and put it in. Server IP is a pretty generic internal IP so no worries there. Good luck getting to it. 🙂 Thank you very much for your help. It means a lot.
Also, when I attempt the CLI command, it just hangs. Never completes.
Interesting,
Okay can you add this to your outputs.conf and then restart?
[tcpout]
defaultGroup = group1
[tcpout:group1]
server=192.168.0.2:9997
And then restart your Splunk Forwarder.
Could you then review splunkd.log for any errors? It might say something like "TCPoutput paused data flow" or something like that if I remember correctly.
Below is what I get in the logs now.
05-10-2022 11:12:17.140 -0400 INFO DC:DeploymentClient [9488 MainThread] - Starting phonehome thread.
05-10-2022 11:12:17.140 -0400 INFO DS_DC_Common [9488 MainThread] - Deployment Client initialized.
05-10-2022 11:12:17.140 -0400 INFO ServerRoles [9488 MainThread] - Declared role=deployment_client.
05-10-2022 11:12:17.140 -0400 INFO DS_DC_Common [9488 MainThread] - Deployment Server not available on a dedicated forwarder.
05-10-2022 11:12:17.140 -0400 INFO DC:PhonehomeThread [8536 PhonehomeThread] - Phonehome thread start, intervals: handshakeRetry=12.0 phonehome=60.0.
05-10-2022 11:12:17.140 -0400 INFO ClusteringMgr [9488 MainThread] - initing clustering with: ht=60.000 rf=3 sf=2 ct=60.000 st=60.000 rt=60.000 rct=5.000 rst=5.000 rrt=10.000 rmst=600.000 rmrt=600.000 icps=25 sfrt=600.000 pe=1 im=0 ip=0 mob=5 mor=5 mosr=5 pb=5 rep_port= pptr=10 pptrl=100 fznb=10 Empty/Default cluster pass4symmkey=false allow Empty/Default cluster pass4symmkey=true rrt=restart dft=180 abt=600 sbs=1
05-10-2022 11:12:17.140 -0400 INFO DC:DeploymentClient [8536 PhonehomeThread] - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
05-10-2022 11:12:17.140 -0400 INFO ClusteringMgr [9488 MainThread] - clustering disabled
05-10-2022 11:12:17.140 -0400 WARN SHCConfig [9488 MainThread] - Default pass4symkey is being used. Please change to a random one.
05-10-2022 11:12:17.140 -0400 INFO SHClusterMgr [9488 MainThread] - initing shpooling with: ht=60.000 rf=3 ct=60.000 st=60.000 rt=60.000 rct=5.000 rst=5.000 rrt=10.000 rmst=600.000 rmrt=600.000 pe=1 im=0 is=0 mor=5 pb=5 rep_port= pptr=10
05-10-2022 11:12:17.140 -0400 INFO SHClusterMgr [9488 MainThread] - shpooling disabled
05-10-2022 11:12:17.140 -0400 INFO WorkloadManager [9488 MainThread] - Workload management cannot be enabled on this system because the feature is not supported. Check the status of workload management preflight checks for additional information.
05-10-2022 11:12:17.155 -0400 INFO loader [9488 MainThread] - win-service: Windows service is now in running state.
05-10-2022 11:12:17.155 -0400 INFO ApplicationLicense [12132 AppLicenseThread] - app license disabled by conf setting.
05-10-2022 11:12:17.233 -0400 INFO loader [9488 MainThread] - SAML cert db registration with KVStore failed
05-10-2022 11:12:17.233 -0400 INFO CertStorageProvider [9488 MainThread] - Updating status from unknown to unknown
05-10-2022 11:12:17.233 -0400 INFO loader [9488 MainThread] - Auth cert db registration with KVStore failed
05-10-2022 11:12:17.233 -0400 INFO CertStorageProvider [9488 MainThread] - Updating status from unknown to unknown
05-10-2022 11:12:17.233 -0400 INFO Rsa2FA [9488 MainThread] - Could not find [externalTwoFactorAuthSettings] in authentication stanza.
05-10-2022 11:12:17.233 -0400 INFO loader [9488 MainThread] - JsonWebToken Manager registration with KVStore failed.
05-10-2022 11:12:17.233 -0400 INFO IndexerInit [11632 SplunkdSpecificInitThread] - running splunkd specific init
05-10-2022 11:12:17.249 -0400 INFO IntrospectionGenerator:disk_objects [11632 SplunkdSpecificInitThread] - Enabled: disk_objects=false indexes=false volumes=false dispatch=false fishbucket=true partitions=false summaries=false distributedIndexes=false
05-10-2022 11:12:17.249 -0400 INFO IntrospectionGenerator:disk_objects [11632 SplunkdSpecificInitThread] - I-data gathering (Disk Objects) starting; period=600.000s
05-10-2022 11:12:17.249 -0400 INFO loader [9488 MainThread] - Initializing from configuration
05-10-2022 11:12:17.249 -0400 INFO ChunkedLBProcessor [14548 parsing] - Initializing the chunked line breaking processor
05-10-2022 11:12:17.249 -0400 INFO TcpOutputProc [14548 parsing] - Initializing with fwdtype=lwf
05-10-2022 11:12:17.249 -0400 INFO TcpOutputProc [14548 parsing] - found Whitelist forwardedindex.0.whitelist , RE : .*
05-10-2022 11:12:17.249 -0400 INFO TcpOutputProc [14548 parsing] - found Blacklist forwardedindex.1.blacklist , RE : _.*
05-10-2022 11:12:17.249 -0400 INFO TcpOutputProc [14548 parsing] - found Whitelist forwardedindex.2.whitelist , RE : (_audit|_introspection|_internal|_telemetry)
05-10-2022 11:12:17.249 -0400 INFO TcpOutputProc [14548 parsing] - Initializing connection for non-ssl forwarding to 192.168.0.2:9997
05-10-2022 11:12:17.249 -0400 INFO TcpOutputProc [14548 parsing] - tcpout group group1 using Auto load balanced forwarding
05-10-2022 11:12:17.249 -0400 INFO AutoLoadBalancedConnectionStrategy [14548 parsing] - Group group1 initialized with maxQueueSize=512000 in bytes.
05-10-2022 11:12:17.249 -0400 INFO AutoLoadBalancedConnectionStrategy [14548 parsing] - Group group1 initialized with autoLBFrequency=30.000
05-10-2022 11:12:29.153 -0400 INFO DC:DeploymentClient [8536 PhonehomeThread] - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
05-10-2022 11:12:41.165 -0400 INFO DC:DeploymentClient [8536 PhonehomeThread] - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
05-10-2022 11:12:47.133 -0400 INFO AutoLoadBalancedConnectionStrategy [3212 TcpOutEloop] - Connected to idx=192.168.0.2:9997, pset=0, reuse=0.
05-10-2022 11:12:47.242 -0400 INFO ScheduledViewsReaper [12532 DispatchReaper] - Scheduled views reaper run complete. Reaped count=0 scheduled views
05-10-2022 11:12:47.242 -0400 INFO CascadingReplicationManager [12532 DispatchReaper] - Using value for property max_replication_threads=2.
05-10-2022 11:12:47.242 -0400 INFO CascadingReplicationManager [12532 DispatchReaper] - Using value for property max_replication_jobs=5.
05-10-2022 11:12:47.242 -0400 INFO FileAndDirectoryEliminator [12532 DispatchReaper] - Enabled
05-10-2022 11:12:53.178 -0400 INFO DC:DeploymentClient [8536 PhonehomeThread] - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
05-10-2022 11:13:05.191 -0400 INFO DC:DeploymentClient [8536 PhonehomeThread] - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
05-10-2022 11:13:06.441 -0400 INFO ProxyConfig [14352 HttpClientPollingThread_C7BD701A-F102-461C-8FA1-9B5D6DC14779] - Failed to initialize http_proxy from server.conf for splunkd. Please make sure that the http_proxy property is set as http_proxy=http://host:port in case HTTP proxying needs to be enabled.
05-10-2022 11:13:06.441 -0400 INFO ProxyConfig [14352 HttpClientPollingThread_C7BD701A-F102-461C-8FA1-9B5D6DC14779] - Failed to initialize https_proxy from server.conf for splunkd. Please make sure that the https_proxy property is set as https_proxy=http://host:port in case HTTP proxying needs to be enabled.
05-10-2022 11:13:06.441 -0400 INFO ProxyConfig [14352 HttpClientPollingThread_C7BD701A-F102-461C-8FA1-9B5D6DC14779] - Failed to initialize the proxy_rules setting from server.conf for splunkd. Please provide a valid set of proxy_rules in case HTTP proxying needs to be enabled.
05-10-2022 11:13:06.441 -0400 INFO ProxyConfig [14352 HttpClientPollingThread_C7BD701A-F102-461C-8FA1-9B5D6DC14779] - Failed to initialize the no_proxy setting from server.conf for splunkd. Please provide a valid set of no_proxy rules in case HTTP proxying needs to be enabled.
05-10-2022 11:13:06.456 -0400 INFO HttpPubSubConnection [14352 HttpClientPollingThread_C7BD701A-F102-461C-8FA1-9B5D6DC14779] - SSL connection with id: connection_192.168.0.1_8089_SERVER.domain.com_SERVER_C7BD701A-F102-461C-8FA1-9B5D6DC14779
05-10-2022 11:13:06.456 -0400 INFO HttpPubSubConnection [14352 HttpClientPollingThread_C7BD701A-F102-461C-8FA1-9B5D6DC14779] - Running phone uri=/services/broker/phonehome/connection_192.168.0.1_8089_SERVER.domain.com_SERVER_C7BD701A-F102-461C-8FA1-9B5D6DC14779
05-10-2022 11:13:16.985 -0400 INFO AutoLoadBalancedConnectionStrategy [3212 TcpOutEloop] - Found currently active indexer. Connected to idx=192.168.0.2:9997, reuse=1.
05-10-2022 11:13:17.204 -0400 INFO HttpPubSubConnection [14352 HttpClientPollingThread_C7BD701A-F102-461C-8FA1-9B5D6DC14779] - Running phone uri=/services/broker/phonehome/connection_192.168.0.1_8089_SERVER.domain.com_SERVER_C7BD701A-F102-461C-8FA1-9B5D6DC14779
05-10-2022 11:13:17.204 -0400 INFO DC:HandshakeReplyHandler [14352 HttpClientPollingThread_C7BD701A-F102-461C-8FA1-9B5D6DC14779] - Handshake done.
Okay looks like we got a connection now.
So now, what inputs.confs do you have installed on your forwarders? Have you installed any apps to your forwarders?
If so, can you post an example of it?
Well, the inputs.conf has thoroughly confused me. I've edited an inputs.conf for apps and in the local folder of deployed apps and I'm not sure if any of them are doing anything.
In C:\Program Files\Splunk\etc\deployment-apps\Splunk_TA_windows\local\inputs.conf (and other deployed apps) I have:
$SPLUNK_HOME/etc/apps/Splunk_TA_windows/local.
## To make changes, copy the section/stanza you want to change from $SPLUNK_HOME/etc/apps/Splunk_TA_windows/default
## into ../local and edit there.
##
###### OS Logs ######
[WinEventLog://Application]
disabled = 0
start_from = oldest
current_only = 0
checkpointInterval = 5
renderXml=true
index=wineventlog
[WinEventLog://Security]
disabled = 0
start_from = oldest
current_only = 0
evt_resolve_ad_obj = 1
checkpointInterval = 5
blacklist1 = EventCode="4662" Message="Object Type:(?!\s*groupPolicyContainer)"
blacklist2 = EventCode="566" Message="Object Type:(?!\s*groupPolicyContainer)"
renderXml=true
index=wineventlog
[WinEventLog://System]
disabled = 0
start_from = oldest
current_only = 0
checkpointInterval = 5
renderXml=true
index=wineventlog
Could you move everything from C:\Program Files\SplunkUniversalForwarder\etc\deployment-apps\Splunk_TA_windows\* to C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\* then restart the forwarder service again.
The deployment-apps folder is for the Deployment Server to tell forwarders which apps it needs to download from the Deployment Server. Those apps then get installed into the "apps" folder on the forwarders.
Try that and see if you start seeing logs, if so, We can set up your deployment server correctly. 🙂
I may be misunderstanding your instructions a bit. You're saying to take the contents of the "deployment-apps" folder on the deployment server and put them on one of the universal forwarder servers in the "apps" folder. This is essentially what the deployment server was meant to do but we're doing it manually? Just want to make sure I'm doing the right process. Thank you!
That's correct. My apologies.
Take the Splunk_TA_windows that you have on the deployment server (Including the inputs.conf and everything) and put it on the forwarder in the apps folder.
Unfortunately nothing changed with the data. I was able to get the inputs list tool to work though. I think there's something in how I did the deployment server / indexes that is fouled up. When I deployed the Windows app it had me create multiple folders like "Splunk_TA_windows_server" and "Splunk_TA_DomainController" so I have inputs.conf everywhere and no data from the forwarders in Splunk. I have a list of inputs that I printed to PDF to attach here as well.
I'm not familiar with the Splunk_TA_DomainController, where did you see to create that?
Looking at your inputs.conf I'm not seeing anything out of the ordinary.
What happens when you search
index=_internal
Do you see events from your forwarders?
See below for the section and the link where I found it. I only made use of the server and DomainController labels but I'm not even sure if they're doing anything.
When I do that search on the cloud instance, I find millions of Splunk cloud entries. If I do it on my local server, I get the same 11K data entries I've had for a bit now. If I exclude cloud data from the search, I get the same limited deployment server data that I have locally. No data from the forwarders in my network.
https://docs.splunk.com/Documentation/SplunkCloud/8.2.2202/Admin/WindowsGDI