Getting Data In

How to get Windows data into Splunk Cloud?

theitgui
Path Finder

Good Morning,

I'm trialing Splunk Cloud in anticipation of a purchase. I have installed Splunk Enterprise as the deployment server and universal forwarders on three servers. My clients are showing up in "Forwarder Management" but I can't seem to get event logs from any servers except the deployment server. I have enabled firewall ports outbound 8089 and inbound 9997 on the deployment server. These are all Server 2019 machines.

I have verified inputs.conf is pointing event logs to index:wineventlog but that index locally has 0 results and about 112,000 results on the cloud server.

I'm sure it's something simple I'm missing with all the moving parts. Thank you in advance!

Labels (3)
Tags (2)
0 Karma

venky1544
Contributor

Hi @theitgui  it seems there is lot of confusion with your terminology let take a step at a time

1) are you using splunk cloud ?? or you have installed splunk enterprise on EC2 instance in aws or Azure amd you are calling it as splunk cloud  ?? please clarify

 

theitgui
Path Finder

I have a Splunk Cloud trial. I have followed the instructions to install Splunk Enterprise as a Deployment Server on Server 2019 and 5 total Universal Forwarders, also Server 2019 machines.

I have deployed the add on for Windows and UniversalForwarders to all clients. The clients all show up in "Settings > Forwarder management" on the deployment server and appear to be talking to it via the logs.

At the moment I only have logs from the deployment server showing up. I'm trying to get windows event log data from all clients into the Splunk Cloud instance.

The command "splunk list forward-server" on any of the client machines will not get a response, it simply hangs. On the deployment server, that command returns the cloud instance.

0 Karma

Stefanie
Builder

Hi @theitgui ,

I'm assuming the deployment server is acting as your indexer as well? 

When setting up your forwarders, did you give them an outputs.conf to tell them to send log data to your deployment server? 

https://docs.splunk.com/Documentation/Splunk/8.2.6/Admin/Outputsconf

 

Also, check your logs in C:\Program Files\SplunkUniversalForwarder\var\log\splunkd.log on your server 2019 machines. There should be a clue as to not having a connection to send logs to. 

 

Hope that helps!

 

venky1544
Contributor

Hi @theitgui 

you probably have to do some traceback steps

1) you have enabled inbound 9997 on deployment server i assume you did it on the VPC but did you configure in your splunk server under settings >>Forwarding and recieving >> Recieve data

venky1544_0-1652192603784.png

you have to add 9997

2)  i guess outputs.conf is correct thats why they are showing up in the forwarder maangement 

3) check if the path are correctly configured in the monitor stanza in inputs.conf .if you are just uploading the file then don't need to look into inputs.conf  and do check the index as well by playing with alltime options 

Thanks

venky

if it helps karma points are appreciated/if it resolves acceptance of solution is appreciated 

 

 

theitgui
Path Finder

@venky1544 wrote:

3) check if the path are correctly configured in the monitor stanza in inputs.conf .if you are just uploading the file then don't need to look into inputs.conf  and do check the index as well by playing with alltime options 


I'm unsure of the part above but I did verify that 9997 was configured on the Splunk deployment server for receiving data. It was already there, I didn't add it. I have messed with so many inputs.conf files that I'm not sure which are the effective ones. I have tried a test_index and the wineventlog index but nothing from the Server 2019 servers is making it into the deployment server or cloud. Thanks for your help! 

0 Karma

venky1544
Contributor

firstly can you clarify do you have seperate indexer and deployment server ?? or is it one splunk standalone server ?? where you have everything in one server thats bit confusing when you say deployment server 

secondly if if you have setup the servers then it can't be already there it always need to be configured please see the screenshot and configure 9997 

 

venky1544_0-1652193747795.png

and if you have multiple inputs.conf  try to delete the duplicates or  you should use the btool command to see which inputs.conf is taking the precedence and probably thats the culprit where you might have to change the index name

./splunk btool <configFileName> list --debug

 

 

theitgui
Path Finder

I apologize for any confusion of terms. I have a deployment server acting as an indexer and three universal forwarders in addition. I am not well versed in Splunk terms and wasn't aware that I didn't need a deployment server.

As far as multiple inputs.conf, I was moreso referring to them being in many locations, not having duplicate entries in any one location. The command did not return any data. Thank you for your help.

 

theitgui_0-1652194661810.pngtheitgui_1-1652194685952.png

theitgui_2-1652195149893.png

 

 

0 Karma

theitgui
Path Finder

I put the simple one liner outputs.conf in $SPLUNK_HOME/etc/system/local/

Restarted Splunk and all the forwarders have this entry, which I've anonymized a bit, replacing SERVER_IP and SERVER in place of its name. So this would appear to be successful phoning home?

To be clear though, still not getting data from any hosts others than the deployment server. It's odd. Thank you for your help.

 

05-10-2022 10:19:21.447 -0400 INFO HttpPubSubConnection [6008 HttpClientPollingThread_873E6E32-D1FD-427B-A82D-C1D92C0D4E1E] - Running phone uri=/services/broker/phonehome/connection_SERVER_IP_8089_SERVER.DOMAIN.COM_SERVER_873E6E32-D1FD-427B-A82D-C1D92C0D4E1E

0 Karma

Stefanie
Builder

 


@theitgui wrote:

I put the simple one liner outputs.conf in $SPLUNK_HOME/etc/system/local/


Could you post an anonymized version of your outputs.conf?

 

Maybe using the CLI would be easier 

./splunk add forward-server <host name or ip address of your splunk server>:9997
To run CLI commands in Splunk Enterprise on Windows, use PowerShell or the command prompt as an administrator.

1. Open a PowerShell window or command prompt as an administrator.
2. Change to the Splunk Forwarder bin directory.
3. Run a Splunk command by typing in splunk followed by the subcommand and any required arguments.

 

theitgui
Path Finder

 

# Version 8.2.6
#
# This file contains an example outputs.conf. Use this file to configure
# forwarding in a distributed set up.
#
# To use one or more of these configurations, copy the configuration block into
# outputs.conf in $SPLUNK_HOME/etc/system/local/. You must restart Splunk to
# enable configurations.
#
# To learn more about configuration files (including precedence) please see the
# documentation located at
# http://docs.splunk.com/Documentation/Splunk/latest/Admin/Aboutconfigurationfiles


# Specify a target group for an IP:PORT which consists of a single receiver.
# This is the simplest possible configuration; it sends data to the host at
# 10.1.1.197 on port 9997.

[tcpout:group1]
server=192.168.0.2:9997

 

The above is what I've got. I simply took the first example and put it in. Server IP is a pretty generic internal IP so no worries there. Good luck getting to it. 🙂 Thank you very much for your help. It means a lot.

Also, when I attempt the CLI command, it just hangs. Never completes.

theitgui_0-1652194494671.png

 

0 Karma

Stefanie
Builder

Interesting,

Okay can you add this to your outputs.conf and then restart? 

[tcpout]
defaultGroup = group1

[tcpout:group1]
server=192.168.0.2:9997

 

And then restart your Splunk Forwarder. 

Could you then review splunkd.log for any errors? It might say something like "TCPoutput paused data flow" or something like that if I remember correctly.

theitgui
Path Finder

Below is what I get in the logs now.

05-10-2022 11:12:17.140 -0400 INFO  DC:DeploymentClient [9488 MainThread] - Starting phonehome thread.
05-10-2022 11:12:17.140 -0400 INFO  DS_DC_Common [9488 MainThread] - Deployment Client initialized.
05-10-2022 11:12:17.140 -0400 INFO  ServerRoles [9488 MainThread] - Declared role=deployment_client.
05-10-2022 11:12:17.140 -0400 INFO  DS_DC_Common [9488 MainThread] - Deployment Server not available on a dedicated forwarder.
05-10-2022 11:12:17.140 -0400 INFO  DC:PhonehomeThread [8536 PhonehomeThread] - Phonehome thread start, intervals: handshakeRetry=12.0 phonehome=60.0.
05-10-2022 11:12:17.140 -0400 INFO  ClusteringMgr [9488 MainThread] - initing clustering with: ht=60.000 rf=3 sf=2 ct=60.000 st=60.000 rt=60.000 rct=5.000 rst=5.000 rrt=10.000 rmst=600.000 rmrt=600.000 icps=25 sfrt=600.000 pe=1 im=0 ip=0 mob=5 mor=5 mosr=5 pb=5 rep_port= pptr=10 pptrl=100 fznb=10 Empty/Default cluster pass4symmkey=false allow Empty/Default cluster pass4symmkey=true rrt=restart dft=180 abt=600 sbs=1
05-10-2022 11:12:17.140 -0400 INFO  DC:DeploymentClient [8536 PhonehomeThread] - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
05-10-2022 11:12:17.140 -0400 INFO  ClusteringMgr [9488 MainThread] - clustering disabled
05-10-2022 11:12:17.140 -0400 WARN  SHCConfig [9488 MainThread] - Default pass4symkey is being used. Please change to a random one.
05-10-2022 11:12:17.140 -0400 INFO  SHClusterMgr [9488 MainThread] - initing shpooling with: ht=60.000 rf=3 ct=60.000 st=60.000 rt=60.000 rct=5.000 rst=5.000 rrt=10.000 rmst=600.000 rmrt=600.000 pe=1 im=0 is=0 mor=5 pb=5 rep_port= pptr=10
05-10-2022 11:12:17.140 -0400 INFO  SHClusterMgr [9488 MainThread] - shpooling disabled
05-10-2022 11:12:17.140 -0400 INFO  WorkloadManager [9488 MainThread] - Workload management cannot be enabled on this system because the feature is not supported. Check the status of workload management preflight checks for additional information.
05-10-2022 11:12:17.155 -0400 INFO  loader [9488 MainThread] - win-service: Windows service is now in running state.
05-10-2022 11:12:17.155 -0400 INFO  ApplicationLicense [12132 AppLicenseThread] - app license disabled by conf setting.
05-10-2022 11:12:17.233 -0400 INFO  loader [9488 MainThread] - SAML cert db registration with KVStore failed
05-10-2022 11:12:17.233 -0400 INFO  CertStorageProvider [9488 MainThread] - Updating status from unknown to unknown
05-10-2022 11:12:17.233 -0400 INFO  loader [9488 MainThread] - Auth cert db registration with KVStore failed
05-10-2022 11:12:17.233 -0400 INFO  CertStorageProvider [9488 MainThread] - Updating status from unknown to unknown
05-10-2022 11:12:17.233 -0400 INFO  Rsa2FA [9488 MainThread] - Could not find [externalTwoFactorAuthSettings] in authentication stanza.
05-10-2022 11:12:17.233 -0400 INFO  loader [9488 MainThread] - JsonWebToken Manager registration with KVStore failed.
05-10-2022 11:12:17.233 -0400 INFO  IndexerInit [11632 SplunkdSpecificInitThread] - running splunkd specific init
05-10-2022 11:12:17.249 -0400 INFO  IntrospectionGenerator:disk_objects [11632 SplunkdSpecificInitThread] - Enabled: disk_objects=false indexes=false volumes=false dispatch=false fishbucket=true partitions=false summaries=false distributedIndexes=false
05-10-2022 11:12:17.249 -0400 INFO  IntrospectionGenerator:disk_objects [11632 SplunkdSpecificInitThread] - I-data gathering (Disk Objects) starting; period=600.000s
05-10-2022 11:12:17.249 -0400 INFO  loader [9488 MainThread] - Initializing from configuration
05-10-2022 11:12:17.249 -0400 INFO  ChunkedLBProcessor [14548 parsing] - Initializing the chunked line breaking processor
05-10-2022 11:12:17.249 -0400 INFO  TcpOutputProc [14548 parsing] - Initializing with fwdtype=lwf
05-10-2022 11:12:17.249 -0400 INFO  TcpOutputProc [14548 parsing] - found Whitelist forwardedindex.0.whitelist , RE : .*
05-10-2022 11:12:17.249 -0400 INFO  TcpOutputProc [14548 parsing] - found Blacklist forwardedindex.1.blacklist , RE : _.*
05-10-2022 11:12:17.249 -0400 INFO  TcpOutputProc [14548 parsing] - found Whitelist forwardedindex.2.whitelist , RE : (_audit|_introspection|_internal|_telemetry)
05-10-2022 11:12:17.249 -0400 INFO  TcpOutputProc [14548 parsing] - Initializing connection for non-ssl forwarding to 192.168.0.2:9997
05-10-2022 11:12:17.249 -0400 INFO  TcpOutputProc [14548 parsing] - tcpout group group1 using Auto load balanced forwarding
05-10-2022 11:12:17.249 -0400 INFO  AutoLoadBalancedConnectionStrategy [14548 parsing] - Group group1 initialized with maxQueueSize=512000 in bytes.
05-10-2022 11:12:17.249 -0400 INFO  AutoLoadBalancedConnectionStrategy [14548 parsing] - Group group1 initialized with autoLBFrequency=30.000
05-10-2022 11:12:29.153 -0400 INFO  DC:DeploymentClient [8536 PhonehomeThread] - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
05-10-2022 11:12:41.165 -0400 INFO  DC:DeploymentClient [8536 PhonehomeThread] - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
05-10-2022 11:12:47.133 -0400 INFO  AutoLoadBalancedConnectionStrategy [3212 TcpOutEloop] - Connected to idx=192.168.0.2:9997, pset=0, reuse=0.
05-10-2022 11:12:47.242 -0400 INFO  ScheduledViewsReaper [12532 DispatchReaper] - Scheduled views reaper run complete. Reaped count=0 scheduled views
05-10-2022 11:12:47.242 -0400 INFO  CascadingReplicationManager [12532 DispatchReaper] - Using value for property max_replication_threads=2.
05-10-2022 11:12:47.242 -0400 INFO  CascadingReplicationManager [12532 DispatchReaper] - Using value for property max_replication_jobs=5.
05-10-2022 11:12:47.242 -0400 INFO  FileAndDirectoryEliminator [12532 DispatchReaper] - Enabled
05-10-2022 11:12:53.178 -0400 INFO  DC:DeploymentClient [8536 PhonehomeThread] - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
05-10-2022 11:13:05.191 -0400 INFO  DC:DeploymentClient [8536 PhonehomeThread] - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
05-10-2022 11:13:06.441 -0400 INFO  ProxyConfig [14352 HttpClientPollingThread_C7BD701A-F102-461C-8FA1-9B5D6DC14779] - Failed to initialize http_proxy from server.conf for splunkd. Please make sure that the http_proxy property is set as http_proxy=http://host:port in case HTTP proxying needs to be enabled.
05-10-2022 11:13:06.441 -0400 INFO  ProxyConfig [14352 HttpClientPollingThread_C7BD701A-F102-461C-8FA1-9B5D6DC14779] - Failed to initialize https_proxy from server.conf for splunkd. Please make sure that the https_proxy property is set as https_proxy=http://host:port in case HTTP proxying needs to be enabled.
05-10-2022 11:13:06.441 -0400 INFO  ProxyConfig [14352 HttpClientPollingThread_C7BD701A-F102-461C-8FA1-9B5D6DC14779] - Failed to initialize the proxy_rules setting from server.conf for splunkd. Please provide a valid set of proxy_rules in case HTTP proxying needs to be enabled.
05-10-2022 11:13:06.441 -0400 INFO  ProxyConfig [14352 HttpClientPollingThread_C7BD701A-F102-461C-8FA1-9B5D6DC14779] - Failed to initialize the no_proxy setting from server.conf for splunkd. Please provide a valid set of no_proxy rules in case HTTP proxying needs to be enabled.
05-10-2022 11:13:06.456 -0400 INFO  HttpPubSubConnection [14352 HttpClientPollingThread_C7BD701A-F102-461C-8FA1-9B5D6DC14779] - SSL connection with id: connection_192.168.0.1_8089_SERVER.domain.com_SERVER_C7BD701A-F102-461C-8FA1-9B5D6DC14779
05-10-2022 11:13:06.456 -0400 INFO  HttpPubSubConnection [14352 HttpClientPollingThread_C7BD701A-F102-461C-8FA1-9B5D6DC14779] - Running phone uri=/services/broker/phonehome/connection_192.168.0.1_8089_SERVER.domain.com_SERVER_C7BD701A-F102-461C-8FA1-9B5D6DC14779
05-10-2022 11:13:16.985 -0400 INFO  AutoLoadBalancedConnectionStrategy [3212 TcpOutEloop] - Found currently active indexer. Connected to idx=192.168.0.2:9997, reuse=1.
05-10-2022 11:13:17.204 -0400 INFO  HttpPubSubConnection [14352 HttpClientPollingThread_C7BD701A-F102-461C-8FA1-9B5D6DC14779] - Running phone uri=/services/broker/phonehome/connection_192.168.0.1_8089_SERVER.domain.com_SERVER_C7BD701A-F102-461C-8FA1-9B5D6DC14779
05-10-2022 11:13:17.204 -0400 INFO  DC:HandshakeReplyHandler [14352 HttpClientPollingThread_C7BD701A-F102-461C-8FA1-9B5D6DC14779] - Handshake done.
0 Karma

Stefanie
Builder

Okay looks like we got a connection now.

So now, what inputs.confs do you have installed on your forwarders? Have you installed any apps to your forwarders? 
If so, can you post an example of it?

theitgui
Path Finder

Well, the inputs.conf has thoroughly confused me. I've edited an inputs.conf for apps and in the local folder of deployed apps and I'm not sure if any of them are doing anything.

In C:\Program Files\Splunk\etc\deployment-apps\Splunk_TA_windows\local\inputs.conf (and other deployed apps) I have:

 

$SPLUNK_HOME/etc/apps/Splunk_TA_windows/local.
## To make changes, copy the section/stanza you want to change from $SPLUNK_HOME/etc/apps/Splunk_TA_windows/default
## into ../local and edit there.
##



###### OS Logs ######
[WinEventLog://Application]
disabled = 0
start_from = oldest
current_only = 0
checkpointInterval = 5
renderXml=true
index=wineventlog

[WinEventLog://Security]
disabled = 0
start_from = oldest
current_only = 0
evt_resolve_ad_obj = 1
checkpointInterval = 5
blacklist1 = EventCode="4662" Message="Object Type:(?!\s*groupPolicyContainer)"
blacklist2 = EventCode="566" Message="Object Type:(?!\s*groupPolicyContainer)"
renderXml=true
index=wineventlog

[WinEventLog://System]
disabled = 0
start_from = oldest
current_only = 0
checkpointInterval = 5
renderXml=true
index=wineventlog

 

 

0 Karma

Stefanie
Builder

Could you move everything from  C:\Program Files\SplunkUniversalForwarder\etc\deployment-apps\Splunk_TA_windows\* to C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\* then restart the forwarder service again.

 

The deployment-apps folder is for the Deployment Server to tell forwarders which apps it needs to download from the Deployment Server. Those apps then get installed into the "apps" folder on the forwarders.

Try that and see if you start seeing logs, if so, We can set up your deployment server correctly. 🙂 

 

theitgui
Path Finder

I may be misunderstanding your instructions a bit. You're saying to take the contents of the "deployment-apps" folder on the deployment server and put them on one of the universal forwarder servers in the "apps" folder. This is essentially what the deployment server was meant to do but we're doing it manually? Just want to make sure I'm doing the right process. Thank you!

0 Karma

Stefanie
Builder

That's correct. My apologies.

Take the Splunk_TA_windows that you have on the deployment server (Including the inputs.conf and everything) and put it on the forwarder in the apps folder. 

theitgui
Path Finder

Unfortunately nothing changed with the data. I was able to get the inputs list tool to work though. I think there's something in how I did the deployment server / indexes that is fouled up. When I deployed the Windows app it had me create multiple folders like "Splunk_TA_windows_server" and "Splunk_TA_DomainController" so I have inputs.conf everywhere and no data from the forwarders in Splunk. I have a list of inputs that I printed to PDF to attach here as well.

 

 

0 Karma

Stefanie
Builder

I'm not familiar with the Splunk_TA_DomainController, where did you see to create that? 

Looking at your inputs.conf I'm not seeing anything out of the ordinary. 

What happens when you search 

index=_internal

Do you see events from your forwarders?

theitgui
Path Finder

See below for the section and the link where I found it. I only made use of the server and DomainController labels but I'm not even sure if they're doing anything.

When I do that search on the cloud instance, I find millions of Splunk cloud entries. If I do it on my local server, I get the same 11K data entries I've had for a bit now. If I exclude cloud data from the search, I get the same limited deployment server data that I have locally. No data from the forwarders in my network.

https://docs.splunk.com/Documentation/SplunkCloud/8.2.2202/Admin/WindowsGDI

Configure and customize the Windows data collection add-ons

  1. Navigate to Windows > Program Files > Splunk > etc > deployment-apps.
  2. Make copies of the Splunk_TA_windows folder for each of the types of Windows instances that you want to get data from.
  3. Rename each of the folders so that they represent your different Windows servers. For this example, create the following folders:
    • Splunk_TA_windows_DomainController
    • Splunk_TA_windows_server
    • Splunk_TA_windows_client
    • Splunk_TA_windows_GlobalCatalogServer
0 Karma
Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...