Getting Data In

Universal forwarder not forwarding

shirabendor
New Member

Hello,
I'm trying to forward logs from azLog (Azure log integration) into my splunk indexer.
Both are running on AWS instances.
Everything seems to be configured correctly except that I don't see anything on the indexer.
Here is the investigation that I did so far:

My indexer has a receiver configured and enabled on 9997.
My instance which has the forwarder installed is able to connect there:

> PS C:\Users\Administrator>
> Test-NetConnection xxx.xxx.xxx -Port 9997
> 
> ComputerName     : xxx.xxx.xxx
> RemoteAddress    : xx.xx.xx.xx
> RemotePort       : 9997 
> InterfaceAlias   : Ethernet
> SourceAddress    : xx.xx.xx.xx
> TcpTestSucceeded : True

My inputs file looks like this:

[monitor://C:\Users\azlog\AzureActiveDirectoryJson]
disabled = false
crcSalt = <SOURCE>

[monitor://C:\Users\azlog\AzureResourceManagerJson]
disabled = false
crcSalt = <SOURCE>

[monitor://C:\Users\azlog\AzureSecurityCenterJson]
disabled = false
crcSalt = <SOURCE>

My output file looks like this:

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
disabled = false
server = xxx.xxx.xxx:9997

[tcpout-server://xxx.xxx.xxx:9997]

spunkd is running. Splunk list monitor shows the correct list of files.
Looking at the log for a specific file that should be forwarded I see :

05-29-2018 08:21:10.878 +0000 DEBUG TailReader - tailreader0 waiting for jobs
05-29-2018 08:21:13.878 +0000 DEBUG TailingProcessor - Returning disposition: 1
05-29-2018 08:21:13.878 +0000 DEBUG TailingProcessor - ****************************************
05-29-2018 08:21:13.878 +0000 DEBUG TailingProcessor - File state notification for path='C:\Users\azlog\AzureResourceManagerJson'.
05-29-2018 08:21:13.878 +0000 DEBUG TailingProcessor - Returning disposition: 1
05-29-2018 08:21:13.878 +0000 DEBUG TailingProcessor - ****************************************
05-29-2018 08:21:13.878 +0000 DEBUG TailingProcessor - File state notification for path='C:\Users\azlog\AzureResourceManagerJson\20180529T082113_3468468.0000000035.af2ac63e-756c-4c64-ad6d-b7dca46a0ceb.json' (first time).
05-29-2018 08:21:13.878 +0000 DEBUG TailingProcessor - Returning disposition: 1
05-29-2018 08:21:13.878 +0000 DEBUG TailReader - Enqueued file=C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log in tailreader0
05-29-2018 08:21:13.878 +0000 DEBUG TailReader - Enqueued file=C:\Users\azlog\AzureResourceManagerJson in tailreader0
05-29-2018 08:21:13.878 +0000 DEBUG TailReader - Enqueued file=C:\Users\azlog\AzureResourceManagerJson\20180529T082113_3468468.0000000035.af2ac63e-756c-4c64-ad6d-b7dca46a0ceb.json in tailreader0
05-29-2018 08:21:13.878 +0000 DEBUG TailReader - Start reading file="C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log" in tailreader0 thread
05-29-2018 08:21:13.878 +0000 DEBUG WatchedFile -   Reading for plain initCrc...
05-29-2018 08:21:13.878 +0000 DEBUG WatchedFile -   Preserving seekptr and initcrc.
05-29-2018 08:21:13.893 +0000 DEBUG TailReader - Finished reading file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log' in tailreader0 thread, disposition=NO_DISPOSITION, deferredBy=3.000
05-29-2018 08:21:13.893 +0000 DEBUG TailReader - Defering notification for file=C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log by 3.000ms
05-29-2018 08:21:13.893 +0000 DEBUG TailReader - Start reading file="C:\Users\azlog\AzureResourceManagerJson" in tailreader0 thread
05-29-2018 08:21:13.893 +0000 DEBUG TailReader -   Have seen this item before (since splunkd was restarted).
05-29-2018 08:21:13.893 +0000 DEBUG TailReader - Finished reading file='C:\Users\azlog\AzureResourceManagerJson' in tailreader0 thread, disposition=RECURSE_INTO_THIS_DIRECTORY, deferredBy=0.000
05-29-2018 08:21:13.893 +0000 DEBUG TailReader - Returning disposition=RECURSE_INTO_THIS_DIRECTORY for file=C:\Users\azlog\AzureResourceManagerJson
05-29-2018 08:21:13.893 +0000 DEBUG TailReader - Start reading file="C:\Users\azlog\AzureResourceManagerJson\20180529T082113_3468468.0000000035.af2ac63e-756c-4c64-ad6d-b7dca46a0ceb.json" in tailreader0 thread
05-29-2018 08:21:13.893 +0000 DEBUG TailingProcessor -   Skipping itemPath='C:\Users\azlog\AzureResourceManagerJson\20180529T082113_3468468.0000000035.af2ac63e-756c-4c64-ad6d-b7dca46a0ceb.json', does not match path='C:\Users\azlog\AzureSecurityCenterJson' :Not a directory :Not a symlink
05-29-2018 08:21:13.893 +0000 DEBUG TailingProcessor -   Item 'C:\Users\azlog\AzureResourceManagerJson\20180529T082113_3468468.0000000035.af2ac63e-756c-4c64-ad6d-b7dca46a0ceb.json' matches stanza: C:\Users\azlog\AzureResourceManagerJson.
05-29-2018 08:21:13.893 +0000 DEBUG TailingProcessor -   Storing config 'C:\Users\azlog\AzureResourceManagerJson'.
05-29-2018 08:21:13.893 +0000 DEBUG TailingProcessor -   Will use CRC salt='C:\Users\azlog\AzureResourceManagerJson\20180529T082113_3468468.0000000035.af2ac63e-756c-4c64-ad6d-b7dca46a0ceb.json' for this source.
05-29-2018 08:21:13.893 +0000 DEBUG TailingProcessor -   Entry is associated with 1 configuration(s).
05-29-2018 08:21:13.893 +0000 DEBUG TailReader -   Will attempt to read file: C:\Users\azlog\AzureResourceManagerJson\20180529T082113_3468468.0000000035.af2ac63e-756c-4c64-ad6d-b7dca46a0ceb.json.
05-29-2018 08:21:13.940 +0000 DEBUG TailReader -   Got classified_sourcetype='json-6' and classified_charset='AUTO'.
05-29-2018 08:21:13.940 +0000 DEBUG WatchedFile - Storing pending metadata for file=C:\Users\azlog\AzureResourceManagerJson\20180529T082113_3468468.0000000035.af2ac63e-756c-4c64-ad6d-b7dca46a0ceb.json, sourcetype=json-6, charset=AUTO
05-29-2018 08:21:13.940 +0000 DEBUG WatchedFile - setting trailing nulls to true via 'auto'
05-29-2018 08:21:13.940 +0000 DEBUG WatchedFile -   Loading state from fishbucket.
05-29-2018 08:21:13.940 +0000 DEBUG WatchedFile -   Attempting to load indexed extractions config from conf=source::C:\Users\azlog\AzureResourceManagerJson\20180529T082113_3468468.0000000035.af2ac63e-756c-4c64-ad6d-b7dca46a0ceb.json|host::EC2AMAZ-HOQE95P|json-6|338 ...
05-29-2018 08:21:13.940 +0000 DEBUG WatchedFile -   Reading for plain initCrc...
05-29-2018 08:21:13.940 +0000 DEBUG WatchedFile -   initcrc has changed to: 0x5e4645810867b257.
05-29-2018 08:21:13.940 +0000 DEBUG WatchedFile -   Normal record was not found for initCrc=0x5e4645810867b257.
05-29-2018 08:21:13.940 +0000 DEBUG WatchedFile -   Computed initCrc=5e4645810867b257 (old style).
05-29-2018 08:21:13.940 +0000 DEBUG WatchedFile -   Normal record was not found for initCrc=0x5e4645810867b257.
05-29-2018 08:21:13.940 +0000 DEBUG WatchedFile -   Creating new pipeline input channel with channel id: 339.
05-29-2018 08:21:13.956 +0000 DEBUG WatchedFile -   Attempting to load indexed extractions config from conf=source::C:\Users\azlog\AzureResourceManagerJson\20180529T082113_3468468.0000000035.af2ac63e-756c-4c64-ad6d-b7dca46a0ceb.json|host::EC2AMAZ-HOQE95P|json-6|339 ...
05-29-2018 08:21:13.956 +0000 DEBUG TailReader - About to read data (Opening file: C:\Users\azlog\AzureResourceManagerJson\20180529T082113_3468468.0000000035.af2ac63e-756c-4c64-ad6d-b7dca46a0ceb.json).
05-29-2018 08:21:13.956 +0000 DEBUG WatchedFile - seeking C:\Users\azlog\AzureResourceManagerJson\20180529T082113_3468468.0000000035.af2ac63e-756c-4c64-ad6d-b7dca46a0ceb.json to off=0
05-29-2018 08:21:13.956 +0000 DEBUG WatchedFile -   Reading for plain initCrc...
05-29-2018 08:21:13.956 +0000 DEBUG WatchedFile -   initcrc changed to 0x5e4645810867b257 since file grew past initCrcLen.
05-29-2018 08:21:13.956 +0000 DEBUG WatchedFile -   Applying pending meta data
05-29-2018 08:21:13.956 +0000 DEBUG WatchedFile - Clearing pending metadata
05-29-2018 08:21:13.956 +0000 DEBUG WatchedFile - Reached EOF: fname=C:\Users\azlog\AzureResourceManagerJson\20180529T082113_3468468.0000000035.af2ac63e-756c-4c64-ad6d-b7dca46a0ceb.json fishstate=key=0x5e4645810867b257 sptr=12112 scrc=0xc11622e038ef0e51 fnamecrc=0xbe9301895b5e826a modtime=1527582073
05-29-2018 08:21:13.956 +0000 DEBUG TailReader -   Skipping sending done key.
05-29-2018 08:21:13.956 +0000 DEBUG TailReader -   Will doublecheck EOF (in 3000ms)..
05-29-2018 08:21:13.956 +0000 DEBUG TailReader - Finished reading file='C:\Users\azlog\AzureResourceManagerJson\20180529T082113_3468468.0000000035.af2ac63e-756c-4c64-ad6d-b7dca46a0ceb.json' in tailreader0 thread, disposition=NO_DISPOSITION, deferredBy=3.000
05-29-2018 08:21:13.956 +0000 DEBUG TailReader - Defering notification for file=C:\Users\azlog\AzureResourceManagerJson\20180529T082113_3468468.0000000035.af2ac63e-756c-4c64-ad6d-b7dca46a0ceb.json by 3.000ms
05-29-2018 08:21:13.956 +0000 DEBUG TailReader - tailreader0 waiting for jobs
05-29-2018 08:21:14.893 +0000 DEBUG TailingProcessor - ****************************************

But absolutely nothing on the indexer in the main index.

In the internal index I see the log lines : e.g 05-29-2018 08:25:48.948 +0000 DEBUG TailReader - tailreader0 waiting for jobs

Any help with next steps here?

Thanks

0 Karma

jconger
Splunk Employee
Splunk Employee

I noticed this question was tagged with splunk-cloud. Is your indexer actually a Splunk Cloud instance, or is it an indexer you built yourself in AWS?

If it is a Splunk Cloud indexer, you will need to download and install the credential package for your forwarder https://docs.splunk.com/Documentation/Forwarder/latest/Forwarder/HowtoforwarddatatoSplunkCloud

If it is a Splunk indexer you built in AWS, make sure 9997 is open for the security group associated with your instance. Although, I would expect to see an error message in the _intern index for this case.

0 Karma

xpac
SplunkTrust
SplunkTrust

Try to run splunk list forward-server - it should show you if any your outputs is actually active (=connected).

0 Karma