Universal forwarder not forwarding

[shirabendor]

Hello,
I'm trying to forward logs from azLog (Azure log integration) into my splunk indexer.
Both are running on AWS instances.
Everything seems to be configured correctly except that I don't see anything on the indexer.
Here is the investigation that I did so far:

My indexer has a receiver configured and enabled on 9997.
My instance which has the forwarder installed is able to connect there:

> PS C:\Users\Administrator>
> Test-NetConnection xxx.xxx.xxx -Port 9997
> 
> ComputerName     : xxx.xxx.xxx
> RemoteAddress    : xx.xx.xx.xx
> RemotePort       : 9997 
> InterfaceAlias   : Ethernet
> SourceAddress    : xx.xx.xx.xx
> TcpTestSucceeded : True

My inputs file looks like this:

[monitor://C:\Users\azlog\AzureActiveDirectoryJson]
disabled = false
crcSalt = <SOURCE>

[monitor://C:\Users\azlog\AzureResourceManagerJson]
disabled = false
crcSalt = <SOURCE>

[monitor://C:\Users\azlog\AzureSecurityCenterJson]
disabled = false
crcSalt = <SOURCE>

My output file looks like this:

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
disabled = false
server = xxx.xxx.xxx:9997

[tcpout-server://xxx.xxx.xxx:9997]

spunkd is running. Splunk list monitor shows the correct list of files.
Looking at the log for a specific file that should be forwarded I see :

05-29-2018 08:21:10.878 +0000 DEBUG TailReader - tailreader0 waiting for jobs
05-29-2018 08:21:13.878 +0000 DEBUG TailingProcessor - Returning disposition: 1
05-29-2018 08:21:13.878 +0000 DEBUG TailingProcessor - ****************************************
05-29-2018 08:21:13.878 +0000 DEBUG TailingProcessor - File state notification for path='C:\Users\azlog\AzureResourceManagerJson'.
05-29-2018 08:21:13.878 +0000 DEBUG TailingProcessor - Returning disposition: 1
05-29-2018 08:21:13.878 +0000 DEBUG TailingProcessor - ****************************************
05-29-2018 08:21:13.878 +0000 DEBUG TailingProcessor - File state notification for path='C:\Users\azlog\AzureResourceManagerJson\20180529T082113_3468468.0000000035.af2ac63e-756c-4c64-ad6d-b7dca46a0ceb.json' (first time).
05-29-2018 08:21:13.878 +0000 DEBUG TailingProcessor - Returning disposition: 1
05-29-2018 08:21:13.878 +0000 DEBUG TailReader - Enqueued file=C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log in tailreader0
05-29-2018 08:21:13.878 +0000 DEBUG TailReader - Enqueued file=C:\Users\azlog\AzureResourceManagerJson in tailreader0
05-29-2018 08:21:13.878 +0000 DEBUG TailReader - Enqueued file=C:\Users\azlog\AzureResourceManagerJson\20180529T082113_3468468.0000000035.af2ac63e-756c-4c64-ad6d-b7dca46a0ceb.json in tailreader0
05-29-2018 08:21:13.878 +0000 DEBUG TailReader - Start reading file="C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log" in tailreader0 thread
05-29-2018 08:21:13.878 +0000 DEBUG WatchedFile -   Reading for plain initCrc...
05-29-2018 08:21:13.878 +0000 DEBUG WatchedFile -   Preserving seekptr and initcrc.
05-29-2018 08:21:13.893 +0000 DEBUG TailReader - Finished reading file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log' in tailreader0 thread, disposition=NO_DISPOSITION, deferredBy=3.000
05-29-2018 08:21:13.893 +0000 DEBUG TailReader - Defering notification for file=C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log by 3.000ms
05-29-2018 08:21:13.893 +0000 DEBUG TailReader - Start reading file="C:\Users\azlog\AzureResourceManagerJson" in tailreader0 thread
05-29-2018 08:21:13.893 +0000 DEBUG TailReader -   Have seen this item before (since splunkd was restarted).
05-29-2018 08:21:13.893 +0000 DEBUG TailReader - Finished reading file='C:\Users\azlog\AzureResourceManagerJson' in tailreader0 thread, disposition=RECURSE_INTO_THIS_DIRECTORY, deferredBy=0.000
05-29-2018 08:21:13.893 +0000 DEBUG TailReader - Returning disposition=RECURSE_INTO_THIS_DIRECTORY for file=C:\Users\azlog\AzureResourceManagerJson
05-29-2018 08:21:13.893 +0000 DEBUG TailReader - Start reading file="C:\Users\azlog\AzureResourceManagerJson\20180529T082113_3468468.0000000035.af2ac63e-756c-4c64-ad6d-b7dca46a0ceb.json" in tailreader0 thread
05-29-2018 08:21:13.893 +0000 DEBUG TailingProcessor -   Skipping itemPath='C:\Users\azlog\AzureResourceManagerJson\20180529T082113_3468468.0000000035.af2ac63e-756c-4c64-ad6d-b7dca46a0ceb.json', does not match path='C:\Users\azlog\AzureSecurityCenterJson' :Not a directory :Not a symlink
05-29-2018 08:21:13.893 +0000 DEBUG TailingProcessor -   Item 'C:\Users\azlog\AzureResourceManagerJson\20180529T082113_3468468.0000000035.af2ac63e-756c-4c64-ad6d-b7dca46a0ceb.json' matches stanza: C:\Users\azlog\AzureResourceManagerJson.
05-29-2018 08:21:13.893 +0000 DEBUG TailingProcessor -   Storing config 'C:\Users\azlog\AzureResourceManagerJson'.
05-29-2018 08:21:13.893 +0000 DEBUG TailingProcessor -   Will use CRC salt='C:\Users\azlog\AzureResourceManagerJson\20180529T082113_3468468.0000000035.af2ac63e-756c-4c64-ad6d-b7dca46a0ceb.json' for this source.
05-29-2018 08:21:13.893 +0000 DEBUG TailingProcessor -   Entry is associated with 1 configuration(s).
05-29-2018 08:21:13.893 +0000 DEBUG TailReader -   Will attempt to read file: C:\Users\azlog\AzureResourceManagerJson\20180529T082113_3468468.0000000035.af2ac63e-756c-4c64-ad6d-b7dca46a0ceb.json.
05-29-2018 08:21:13.940 +0000 DEBUG TailReader -   Got classified_sourcetype='json-6' and classified_charset='AUTO'.
05-29-2018 08:21:13.940 +0000 DEBUG WatchedFile - Storing pending metadata for file=C:\Users\azlog\AzureResourceManagerJson\20180529T082113_3468468.0000000035.af2ac63e-756c-4c64-ad6d-b7dca46a0ceb.json, sourcetype=json-6, charset=AUTO
05-29-2018 08:21:13.940 +0000 DEBUG WatchedFile - setting trailing nulls to true via 'auto'
05-29-2018 08:21:13.940 +0000 DEBUG WatchedFile -   Loading state from fishbucket.
05-29-2018 08:21:13.940 +0000 DEBUG WatchedFile -   Attempting to load indexed extractions config from conf=source::C:\Users\azlog\AzureResourceManagerJson\20180529T082113_3468468.0000000035.af2ac63e-756c-4c64-ad6d-b7dca46a0ceb.json|host::EC2AMAZ-HOQE95P|json-6|338 ...
05-29-2018 08:21:13.940 +0000 DEBUG WatchedFile -   Reading for plain initCrc...
05-29-2018 08:21:13.940 +0000 DEBUG WatchedFile -   initcrc has changed to: 0x5e4645810867b257.
05-29-2018 08:21:13.940 +0000 DEBUG WatchedFile -   Normal record was not found for initCrc=0x5e4645810867b257.
05-29-2018 08:21:13.940 +0000 DEBUG WatchedFile -   Computed initCrc=5e4645810867b257 (old style).
05-29-2018 08:21:13.940 +0000 DEBUG WatchedFile -   Normal record was not found for initCrc=0x5e4645810867b257.
05-29-2018 08:21:13.940 +0000 DEBUG WatchedFile -   Creating new pipeline input channel with channel id: 339.
05-29-2018 08:21:13.956 +0000 DEBUG WatchedFile -   Attempting to load indexed extractions config from conf=source::C:\Users\azlog\AzureResourceManagerJson\20180529T082113_3468468.0000000035.af2ac63e-756c-4c64-ad6d-b7dca46a0ceb.json|host::EC2AMAZ-HOQE95P|json-6|339 ...
05-29-2018 08:21:13.956 +0000 DEBUG TailReader - About to read data (Opening file: C:\Users\azlog\AzureResourceManagerJson\20180529T082113_3468468.0000000035.af2ac63e-756c-4c64-ad6d-b7dca46a0ceb.json).
05-29-2018 08:21:13.956 +0000 DEBUG WatchedFile - seeking C:\Users\azlog\AzureResourceManagerJson\20180529T082113_3468468.0000000035.af2ac63e-756c-4c64-ad6d-b7dca46a0ceb.json to off=0
05-29-2018 08:21:13.956 +0000 DEBUG WatchedFile -   Reading for plain initCrc...
05-29-2018 08:21:13.956 +0000 DEBUG WatchedFile -   initcrc changed to 0x5e4645810867b257 since file grew past initCrcLen.
05-29-2018 08:21:13.956 +0000 DEBUG WatchedFile -   Applying pending meta data
05-29-2018 08:21:13.956 +0000 DEBUG WatchedFile - Clearing pending metadata
05-29-2018 08:21:13.956 +0000 DEBUG WatchedFile - Reached EOF: fname=C:\Users\azlog\AzureResourceManagerJson\20180529T082113_3468468.0000000035.af2ac63e-756c-4c64-ad6d-b7dca46a0ceb.json fishstate=key=0x5e4645810867b257 sptr=12112 scrc=0xc11622e038ef0e51 fnamecrc=0xbe9301895b5e826a modtime=1527582073
05-29-2018 08:21:13.956 +0000 DEBUG TailReader -   Skipping sending done key.
05-29-2018 08:21:13.956 +0000 DEBUG TailReader -   Will doublecheck EOF (in 3000ms)..
05-29-2018 08:21:13.956 +0000 DEBUG TailReader - Finished reading file='C:\Users\azlog\AzureResourceManagerJson\20180529T082113_3468468.0000000035.af2ac63e-756c-4c64-ad6d-b7dca46a0ceb.json' in tailreader0 thread, disposition=NO_DISPOSITION, deferredBy=3.000
05-29-2018 08:21:13.956 +0000 DEBUG TailReader - Defering notification for file=C:\Users\azlog\AzureResourceManagerJson\20180529T082113_3468468.0000000035.af2ac63e-756c-4c64-ad6d-b7dca46a0ceb.json by 3.000ms
05-29-2018 08:21:13.956 +0000 DEBUG TailReader - tailreader0 waiting for jobs
05-29-2018 08:21:14.893 +0000 DEBUG TailingProcessor - ****************************************

But absolutely nothing on the indexer in the main index.

In the internal index I see the log lines : e.g 05-29-2018 08:25:48.948 +0000 DEBUG TailReader - tailreader0 waiting for jobs

Any help with next steps here?

Thanks

[jconger]

I noticed this question was tagged with splunk-cloud. Is your indexer actually a Splunk Cloud instance, or is it an indexer you built yourself in AWS?

If it is a Splunk Cloud indexer, you will need to download and install the credential package for your forwarder https://docs.splunk.com/Documentation/Forwarder/latest/Forwarder/HowtoforwarddatatoSplunkCloud

If it is a Splunk indexer you built in AWS, make sure 9997 is open for the security group associated with your instance. Although, I would expect to see an error message in the _intern index for this case.

[xpac]

Try to run splunk list forward-server - it should show you if any your outputs is actually active (=connected).