How to activate forwarder server?

[jhl226116]

Hi Guys,

I am struggling to send data from remote machine to Splunk server. I have tried the steps mentioned in the link but still no luck:
https://answers.splunk.com/answers/48760/how-to-activate-forward-server.html

Can anyone tell me how to activate forward server?

Running Splunk server and Forwarder on virtual Ubuntu platform.

Indexer: 10.10.50.49
Universal Forwarder: 10.10.50.18

root@forwarder:/opt/splunkforwarder/bin# ./splunk add forward-server 10.10.50.49:9997
Added forwarding to: 10.10.50.49:9997.

root@forwarder:/opt/splunkforwarder/bin# ./splunk list forward-server
Active forwards:
None
Configured but inactive forwards:
10.10.50.49:9997

Port 9997 has been enabled in the Indexer.

root@indexer: /opt/splunk/bin# ./splunk list forward-server
Active forwards: 
SPsvr:9997
Configured but inactive forwards:
None

I can ping between Indexer =(10.10.50.49) and forwarder(10.10.50.18) vice-versa
I have disabled Ubuntu firewall on both Indexer and Forwarder

root@indexer:/opt/splunk/bin# sudo ufw disable
Firewall stopped and disabled on system startup

root@forwarder:/opt/splunk/bin# sudo ufw disable
Firewall stopped and disabled on system startup

Not sure if my outputs.conf is configured correctly. I checked the document but am not exactly sure. Here is my outputs.conf from the forwarder:

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = 10.10.50.49:9997

[tcpout-server://10.10.50.49:9997]

If someone can tell me what I'm doing wrong or how I can resolve this issue, I would really appreciate it.

I'm almost close to giving up if there's no concrete answer on this. I'd like to at least know what else I can do from here.

Thanks,

[danielwill]

Here's a step-by-step guide to activating a forwarder server:

1) Install Splunk Universal Forwarder

2) Configure Forwarder

3) Start the Forwarder

4) Monitor Forwarder Status

5) Verify Data Forwarding

[MuS]

Hi jhl226116,

this looks wrong:

root@indexer: /opt/splunk/bin# ./splunk list forward-server
Active forwards: 
SPsvr:9997
Configured but inactive forwards:

This tells me you enabled forwarding on the indexer but not receiving. To enable receiving on the indexer run this command:

./splunk enable listen 9997 -auth <username>:<password>

And please remember to disable the forwarding on the indexer before you enable receiving, otherwise you could create a nasty data loop 😉

Hope this helps ...

cheers, MuS

[jhl226116]

It says Failed to create because Configuration for port 9997 already exists. Forwarding is already disabled on the indexer.

root@SPsvr:/opt/splunk/bin# ./splunk list forward-server
Active forwards:
None
Configured but inactive forwards:
SPsvr:9997

root@SPsvr:/opt/splunk/bin# sudo ufw disable
Firewall stopped and disabled on system startup

root@SPsvr:/opt/splunk/bin# ./splunk enable listen 9997 -auth admin:xxxxxxxx
Failed to create. Configuration for port 9997 already exists.

[MuS]

On the indexer run splunk btool outputs list --debug | grep -v default, see what custom outputs.conf you have and remove it on the indexer.

Then run splunk btool inputs list splunktcp --debug | grep -v default, check if everything is correct and also run splunk list inputstatus and check for tcp_cooked:listenerports which should be 9997.

Restart Splunk and it should work

[jhl226116]

I think this proves that my forward server is activated now. Well this part is now sorted but my goal is still far away from my intentions. My goal is to ingest Cisco ASA firewall syslogs data into splunk.
I read many articles but were still failing to make it work.
I better continut to dig in and start a fresh thread to progress further.
Thanks for your help, I apprecaite it.

[MuS]

You're welcome, feel free to up-vote any answer or comment that was useful 😉

[jhl226116]

Sure will do 🙂

I will post new thread on how to to configure forwarder to send Cisco ASA syslogs to the indexer. But before I post anything I will dig a bit more and try to work it myself further.
Thanks again.

[jhl226116]

Additional info:

root@forwarder:/opt/splunkforwarder/bin# ./splunk show deploy-poll
Deployment Server URI is set to "10.10.50.12:8089".

root@forwarder:/opt/splunkforwarder/bin# ./splunk list forward-server
Active forwards:
10.10.50.11:9997
Configured but inactive forwards:
None

root@forwarder:/opt/splunkforwarder/bin# ./splunk show servername
Server name: forwarder

root@forwarder:/opt/splunkforwarder/bin# ./splunk show default-hostname
Default hostname for data inputs: forwarder.

 
 
Search on forwarder:
 
index=_internal host="forwarder"
4/3/17 04-03-2017 13:47:36.366 +1000 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected

1:47:36.366 PM · host = forwarder
· source = /opt/splunkforwarder/var/log/splunk/splunkd.log
· sourcetype = splunkd
4/3/17 04-03-2017 13:47:24.366 +1000 INFO DC:PhonehomeThread - Attempted handshake 120 times. Will try to re-subscribe to handshake reply
1:47:24.366 PM · host = forwarder
· source = /opt/splunkforwarder/var/log/splunk/splunkd.log
· sourcetype = splunkd
4/3/17 04-03-2017 13:47:24.365 +1000 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
1:47:24.365 PM · host = forwarder
· source = /opt/splunkforwarder/var/log/splunk/splunkd.log
· sourcetype = splunkd
4/3/17 04-03-2017 13:47:22.083 +1000 INFO Metrics - group=thruput, name=uncooked_output, instantaneous_kbps=0.000000, instantaneous_eps=0.000000, average_kbps=0.000000, total_k_processed=0.000000, kb=0.000000, ev=0.000000
1:47:22.083 PM · host = forwarder
· source = /opt/splunkforwarder/var/log/splunk/metrics.log
· sourcetype = splunkd
4/3/17 04-03-2017 13:47:22.083 +1000 INFO Metrics - group=thruput, name=thruput, instantaneous_kbps=0.229583, instantaneous_eps=0.580635, average_kbps=0.304443, total_k_processed=434.000000, kb=7.117188, ev=18.000000, load_average=0.810000
1:47:22.083 PM · host = forwarder
· source = /opt/splunkforwarder/var/log/splunk/metrics.log
· sourcetype = splunkd
4/3/17 04-03-2017 13:47:22.083 +1000 INFO Metrics - group=thruput, name=cooked_output, instantaneous_kbps=0.229583, instantaneous_eps=0.548377, average_kbps=0.303742, total_k_processed=433.000000, kb=7.117188, ev=17.000000
1:47:22.083 PM · host = forwarder
· source = /opt/splunkforwarder/var/log/splunk/metrics.log
· sourcetype = splunkd
4/3/17 04-03-2017 13:47:22.083 +1000 INFO Metrics - group=tcpout_connections, name=default-autolb-group:10.10.50.11:9997:0, sourcePort=8089, destIp=10.10.50.11, destPort=9997, _tcp_Bps=297.00, _tcp_KBps=0.29, _tcp_avg_thruput=0.38, _tcp_Kprocessed=502, _tcp_eps=0.53, kb=8.70
1:47:22.083 PM · host = forwarder
· source = /opt/splunkforwarder/var/log/splunk/metrics.log
· sourcetype = splunkd
4/3/17 04-03-2017 13:47:22.083 +1000 INFO Metrics - group=queue, name=tcpin_queue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
1:47:22.083 PM · host = forwarder
· source = /opt/splunkforwarder/var/log/splunk/metrics.log
· sourcetype = splunkd
4/3/17 04-03-2017 13:47:22.083 +1000 INFO Metrics - group=queue, name=structuredparsingqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
1:47:22.083 PM · host = forwarder
· source = /opt/splunkforwarder/var/log/splunk/metrics.log
· sourcetype = splunkd
4/3/17 04-03-2017 13:47:22.083 +1000 INFO Metrics - group=queue, name=parsingqueue, max_size_kb=512, current_size_kb=0, current_size=0, largest_size=2, smallest_size=0
1:47:22.083 PM · host = forwarder
· source = /opt/splunkforwarder/var/log/splunk/metrics.log
· sourcetype = splunkd
4/3/17 04-03-2017 13:47:22.083 +1000 INFO Metrics - group=queue, name=nullqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
1:47:22.083 PM

[jhl226116]

Ok, I manage to wipe out everything so that I can start from scratch.

I installed fresh copy of Ubuntu on a virtual machine and installed clean versions of Splunk.

Finally, here is the output that you requested. There is no result in outputs list but many returned results for inputs list.

Tcp_cooked:listenerports shows 9997.

Now, where to go from here?

root@indexer:/opt/splunk/bin# ./splunk btool outputs list --debug | grep -v default

root@indexer:/opt/splunk/bin# ./splunk btool inputs list splunktcp --debug | grep -v default
/opt/splunk/etc/system/local/inputs.conf host = indexer
/opt/splunk/etc/apps/launcher/local/inputs.conf [splunktcp://9997]
/opt/splunk/etc/apps/launcher/local/inputs.conf connection_host = ip
/opt/splunk/etc/system/local/inputs.conf host = indexer
 
 
 
 
 
root@indexer:/opt/splunk/bin# ./splunk list inputstatus
Cooked:tcp :
    tcp

ExecProcessor:exec commands :
    ./bin/collector.path
        time opened = 2017-03-30T14:31:44+1100

    ./bin/dmc_config.py
        exit status description = exited with code 0
        time closed = 2017-03-30T14:31:47+1100
        time opened = 2017-03-30T14:31:46+1100

    ./bin/instrumentation.py
        exit status description = exited with code 0
        time closed = 2017-04-02T03:06:00+1000
        time opened = 2017-04-02T03:05:00+1000
        total bytes = 97

Raw:tcp :
    tcp

TailingProcessor:FileStatus :
    $SPLUNK_HOME/etc/splunk.version
        file position = 70
        file size = 70
        percent = 100.00
        type = finished reading

    $SPLUNK_HOME/var/log/introspection
        type = directory

    $SPLUNK_HOME/var/log/splunk
        type = directory

    $SPLUNK_HOME/var/log/splunk/license_usage_summary.log
        type = directory

    $SPLUNK_HOME/var/spool/splunk/...stash_new
        type = directory

    /opt/splunk/var/log/introspection/disk_objects.log
        file position = 1110768
        file size = 1110768
        parent = $SPLUNK_HOME/var/log/introspection
        percent = 100.00
        type = open file

    /opt/splunk/var/log/introspection/http_event_collector_metrics.log
        file position = 0
        file size = 0
        parent = $SPLUNK_HOME/var/log/introspection
        percent = 100
        type = finished reading

    /opt/splunk/var/log/introspection/kvstore.log
        file position = 16475557
        file size = 16475557
        parent = $SPLUNK_HOME/var/log/introspection
        percent = 100.00
        type = finished reading

    /opt/splunk/var/log/introspection/kvstore.log.1
        file position = 25003278
        file size = 25003278
        parent = $SPLUNK_HOME/var/log/introspection
        percent = 100.00
        type = finished reading

    /opt/splunk/var/log/introspection/kvstore.log.2
        file position = 25004407
        file size = 25004407
        parent = $SPLUNK_HOME/var/log/introspection
        percent = 100.00
        type = finished reading

    /opt/splunk/var/log/introspection/kvstore.log.3
        file position = 25006419
        file size = 25006419
        parent = $SPLUNK_HOME/var/log/introspection
        percent = 100.00
        type = finished reading

    /opt/splunk/var/log/introspection/resource_usage.log
        file position = 16124806
        file size = 16124806
        parent = $SPLUNK_HOME/var/log/introspection
        percent = 100.00
        type = open file

    /opt/splunk/var/log/introspection/resource_usage.log.1
        file position = 25000795
        file size = 25000795
        parent = $SPLUNK_HOME/var/log/introspection
        percent = 100.00
        type = finished reading

    /opt/splunk/var/log/introspection/resource_usage.log.2
        file position = 25000440
        file size = 25000440
        parent = $SPLUNK_HOME/var/log/introspection
        percent = 100.00
        type = finished reading

    /opt/splunk/var/log/introspection/resource_usage.log.3
        file position = 25000130
        file size = 25000130
        parent = $SPLUNK_HOME/var/log/introspection
        percent = 100.00
        type = finished reading

    /opt/splunk/var/log/splunk/audit.log
        file position = 1785766
        file size = 1785148
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100.03
        type = open file

    /opt/splunk/var/log/splunk/btool.log
        file position = 184036
        file size = 184036
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100.00
        type = open file

    /opt/splunk/var/log/splunk/conf.log
        file position = 296
        file size = 296
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100.00
        type = finished reading

    /opt/splunk/var/log/splunk/django_access.log
        file position = 0
        file size = 0
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100
        type = finished reading

    /opt/splunk/var/log/splunk/django_error.log
        file position = 0
        file size = 0
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100
        type = finished reading

    /opt/splunk/var/log/splunk/django_service.log
        file position = 0
        file size = 0
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100
        type = finished reading

    /opt/splunk/var/log/splunk/export_metrics.log
        file position = 0
        file size = 0
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100
        type = finished reading

    /opt/splunk/var/log/splunk/first_install.log
        file position = 70
        file size = 70
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100.00
        type = finished reading

    /opt/splunk/var/log/splunk/license_usage.log
        file position = 1611
        file size = 1611
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100.00
        type = open file

    /opt/splunk/var/log/splunk/license_usage_summary.log
        file position = 1188
        file size = 1188
        parent = $SPLUNK_HOME/var/log/splunk/license_usage_summary.log
        percent = 100.00
        type = finished reading

    /opt/splunk/var/log/splunk/metrics.log
        file position = 22998327
        file size = 22998327
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100.00
        type = open file

    /opt/splunk/var/log/splunk/metrics.log.1
        file position = 25000091
        file size = 25000091
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100.00
        type = finished reading

    /opt/splunk/var/log/splunk/metrics.log.2
        file position = 25000136
        file size = 25000136
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100.00
        type = finished reading

    /opt/splunk/var/log/splunk/metrics.log.3
        file position = 25000131
        file size = 25000131
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100.00
        type = finished reading

    /opt/splunk/var/log/splunk/mongod.log
        file position = 13073
        file size = 13073
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100.00
        type = open file

    /opt/splunk/var/log/splunk/remote_searches.log
        file position = 0
        file size = 0
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100
        type = finished reading

    /opt/splunk/var/log/splunk/scheduler.log
        file position = 38455
        file size = 38455
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100.00
        type = finished reading

    /opt/splunk/var/log/splunk/searchhistory.log
        file position = 0
        file size = 0
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100
        type = finished reading

    /opt/splunk/var/log/splunk/splunkd-utility.log
        file position = 1397
        file size = 1397
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100.00
        type = finished reading

    /opt/splunk/var/log/splunk/splunkd.log
        file position = 55559
        file size = 55559
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100.00
        type = finished reading

    /opt/splunk/var/log/splunk/splunkd_access.log
        file position = 229248
        file size = 229248
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100.00
        type = open file

    /opt/splunk/var/log/splunk/splunkd_stderr.log
        file position = 67
        file size = 67
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100.00
        type = finished reading

    /opt/splunk/var/log/splunk/splunkd_stdout.log
        file position = 0
        file size = 0
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100
        type = finished reading

    /opt/splunk/var/log/splunk/splunkd_ui_access.log
        file position = 329378
        file size = 329378
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100.00
        type = finished reading

    /opt/splunk/var/log/splunk/web_access.log
        file position = 20717
        file size = 20717
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100.00
        type = open file

    /opt/splunk/var/log/splunk/web_service.log
        file position = 25074
        file size = 25074
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100.00
        type = open file

tcp_cooked:listenerports :
    9997

[jhl226116]

Commands are not working. Is there a typo somewhere?

root@SPsvr:/opt/splunk/bin# splunk btool outputs list --debug | grep -v default
splunk: command not found

root@SPsvr:~# splunk btool outputs list --debug | grep -v default
splunk: command not found

root@SPsvr:/opt/splunk/bin# splunk btool inputs list splunktcp --debug | grep -v default
splunk: command not found

root@SPsvr:/opt/splunk/bin# splunk list inputstatus
splunk: command not found

root@SPsvr:~# splunk list input status
splunk: command not found

[MuS]

use ./splunk instead of splunk

[jhl226116]

Awesome, I can run the commands. See results below:

root@SPsvr:/opt/splunk/bin# ./splunk btool outputs list --debug | grep -v default
/opt/splunk/etc/system/local/outputs.conf [tcpout]
/opt/splunk/etc/system/local/outputs.conf [tcpout-server://SPsvr:9997]
/opt/splunk/etc/system/local/outputs.conf server = SPsvr:9997

root@SPsvr:/opt/splunk/bin# ./splunk btool inputs list splunktcp --debug | grep -v default
/opt/splunk/etc/system/local/inputs.conf host = csoc
/opt/splunk/etc/apps/search/local/inputs.conf [splunktcp://9997]
/opt/splunk/etc/apps/search/local/inputs.conf connection_host = ip
/opt/splunk/etc/system/local/inputs.conf host = csoc

root@SPsvr:/opt/splunk/bin# ./splunk list inputstatus
Cooked:tcp :
9997:127.0.0.1:8089
time opened = 2017-03-22T08:22:38+1100

tcp

ExecProcessor:exec commands :
./bin/collector.path
time opened = 2017-03-22T08:22:45+1100

./bin/dmc_config.py
    exit status description = exited with code 0
    time closed = 2017-03-22T08:22:50+1100
    time opened = 2017-03-22T08:22:50+1100

./bin/instrumentation.py
    exit status description = exited with code 0
    time closed = 2017-03-23T03:06:00+1100
    time opened = 2017-03-23T03:05:00+1100
    total bytes = 305

./bin/scripted_inputs/dependency_manager.py
    exit status description = exited with code 0
    time closed = 2017-03-22T08:22:48+1100
    time opened = 2017-03-22T08:22:48+1100

./bin/scripted_inputs/deploy_splunk_ta_paloalto.py
    exit status description = exited with code 0
    time closed = 2017-03-22T08:22:43+1100
    time opened = 2017-03-22T08:22:43+1100

./bin/scripted_inputs/ftr_lookups.py
    exit status description = exited with code 0
    time closed = 2017-03-22T08:22:40+1100
    time opened = 2017-03-22T08:22:40+1100

./bin/scripted_inputs/update_hosts.py
    exit status description = exited with code 0
    time closed = 2017-03-23T00:00:00+1100
    time opened = 2017-03-23T00:00:00+1100

Raw:tcp :
tcp

TailingProcessor:FileStatus :
$SPLUNK_HOME/etc/splunk.version
file position = 70
file size = 70
percent = 100.00
type = finished reading

$SPLUNK_HOME/var/log/introspection
    type = directory

$SPLUNK_HOME/var/log/splunk/license_usage_summary.log
    type = directory

$SPLUNK_HOME/var/spool/splunk/...stash_new
    type = directory

/opt/splunk/var/log/introspection/kvstore.log.1
    file position = 10616832
    file size = 25005470
    parent = $SPLUNK_HOME/var/log/introspection
    percent = 42.46
    type = reading (batch)

/opt/splunk/var/log/introspection/kvstore.log.2
    file position = 24970463
    file size = 25006411
    parent = $SPLUNK_HOME/var/log/introspection
    percent = 99.86
    type = open file

/opt/splunk/var/log/introspection/kvstore.log.4
    file position = 0
    file size = 25004216
    parent = $SPLUNK_HOME/var/log/introspection
    percent = 0.00
    type = batch processing(toRead=25004216)

tcp_cooked:listenerports :
9997

UDP:listenerports :
514

[MuS]

This is still not good:

root@SPsvr:/opt/splunk/bin# ./splunk btool outputs list --debug | grep -v default
/opt/splunk/etc/system/local/outputs.conf [tcpout]
/opt/splunk/etc/system/local/outputs.conf [tcpout-server://SPsvr:9997]
/opt/splunk/etc/system/local/outputs.conf server = SPsvr:9997

Is SPsvr your indexer? If so will tell the Splunk indexer to forward to itself, which is a loop. I would remove the file and restart Splunk.

[jhl226116]

Yes, SPsvr is a Splunk Server Instance which has server roles: Indexer, License Master and Search Head.

Do you think I should just start from scratch with fresh installation of Splunk?

[MuS]

just remove the outputs.conf first and restart Splunk.
If it does not help you can start from scratch 😉

[jhl226116]

I've installed fresh version of Ubuntu and Splunk, started working on it but nothing works properly. Had problems after problems after problems...
Got a headache, I will continue to work on it next week.

[jhl226116]

I'm not sure how to delete outputs.conf completely. Think I'll just refresh everything and start all over again.

/opt/splunk/etc/system/local/outputs.conf

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = SPsvr:9997

[tcpout-server://SPsvr:9997]

[esix_splunk]

You are getting a connection refused message to the indexer. This means its being blocked at the network level. There is a firewall somewhere blocking this. I'd recommend disabling firewalls on both hosts as a test, but you might also have a network level firewall blocking this.

[jhl226116]

Ubuntu firewalls on both hosts has already been disabled.

After seeing your post, I created a new rule in Cisco ASA firewall in the network level to allow necessary Splunk ports to communicate between the indexer and forwarder.

Ports allowed from any source any to any destination within my internal network range.

Ports allowed:
TCP 8000 - Spluk Web
TCP 8080 - Indexer to Indexer Replication
TCP 8088 - mgmt for myself only
TCP 8089 - mgmt
TCP 9997 - Indexing
UDP 514 - Syslog

Also ICMP, domain, http, https has always been enabled already.

Even after creating a new firewall rule to allow any connections between Index and forward server, it still says forwards is inactive.

root@indexer: /opt/splunk/bin# ./splunk list forward-server
Active forwards:
SPsvr:9997
Configured but inactive forwards:

I have restarted splunk and forwarder but no changes.

I can't completely shut off Cisco ASA down because there's other live traffics running on different ports in different network ranges. But this being at a network level and which I have just created a new rule for splunk ports specifically, I'm pretty sure ASA isn't the issue here.

[jhl226116]

I was trying to drill down to where the connection started failing and spotted below error message in the forwarder logs. 03-21-2017 09:07:13.538 +1100 WARN X509Verify - X509 certificate (O=SplunkUser,CN=SplunkServerDefaultCert) should not be used, as it is issued by Splunk's own default Certificate Authority (CA). This pu$

I have no clue what's going on now, it's driving me nuts. I just wanna give up at this point......

nano /opt/splunkforwarder/var/log/splunk/splunkd.log03-21-2017 09:07:13.378 +1100 INFO ChunkedLBProcessor - Initializing the chunked line breaking processor
03-21-2017 09:07:13.378 +1100 INFO TcpOutputProc - Initializing with fwdtype=lwf
03-21-2017 09:07:13.388 +1100 INFO TcpOutputProc - found Whitelist forwardedindex.0.whitelist , RE : .*
03-21-2017 09:07:13.388 +1100 INFO TcpOutputProc - found Blacklist forwardedindex.1.blacklist , RE : _.*
03-21-2017 09:07:13.388 +1100 INFO TcpOutputProc - found Whitelist forwardedindex.2.whitelist , RE : (_audit|_introspection|_internal|_telemetry)
03-21-2017 09:07:13.389 +1100 INFO TcpOutputProc - Initializing connection for non-ssl forwarding to 10.10.50.49:9997
03-21-2017 09:07:13.389 +1100 INFO TcpOutputProc - tcpout group default-autolb-group using Auto load balanced forwarding
03-21-2017 09:07:13.389 +1100 INFO TcpOutputProc - Group default-autolb-group initialized with maxQueueSize=512000 in bytes.
03-21-2017 09:07:13.390 +1100 INFO PipelineComponent - Pipeline merging disabled in default-mode.conf file
03-21-2017 09:07:13.390 +1100 INFO PipelineComponent - Pipeline typing disabled in default-mode.conf file
03-21-2017 09:07:13.390 +1100 INFO PipelineComponent - Pipeline vix disabled in default-mode.conf file
03-21-2017 09:07:13.465 +1100 INFO PipelineComponent - Launching the pipelines for set 0.
03-21-2017 09:07:13.534 +1100 INFO TailingProcessor - TailWatcher initializing...
03-21-2017 09:07:13.534 +1100 INFO TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk.
03-21-2017 09:07:13.534 +1100 INFO TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk/...stash_new.
03-21-2017 09:07:13.535 +1100 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/etc/splunk.version.
03-21-2017 09:07:13.535 +1100 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk.
03-21-2017 09:07:13.535 +1100 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/license_usage_summary.log.
03-21-2017 09:07:13.535 +1100 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/metrics.log.
03-21-2017 09:07:13.535 +1100 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log.
03-21-2017 09:07:13.535 +1100 INFO TailReader - State transitioning from 1 to 0 (initOrResume).
03-21-2017 09:07:13.535 +1100 INFO TailReader - State transitioning from 1 to 0 (initOrResume).
03-21-2017 09:07:13.535 +1100 INFO TailingProcessor - Adding watch on path: /opt/splunkforwarder/etc/splunk.version.
03-21-2017 09:07:13.535 +1100 INFO TailingProcessor - Adding watch on path: /opt/splunkforwarder/var/log/splunk.
03-21-2017 09:07:13.535 +1100 INFO TailingProcessor - Adding watch on path: /opt/splunkforwarder/var/spool/splunk.
03-21-2017 09:07:13.535 +1100 INFO TailReader - Registering metrics callback for: tailreader0
03-21-2017 09:07:13.535 +1100 INFO TailReader - Starting tailreader0 thread
03-21-2017 09:07:13.537 +1100 INFO loader - Limiting REST HTTP server to 21333 sockets
03-21-2017 09:07:13.537 +1100 INFO loader - Limiting REST HTTP server to 658 threads
03-21-2017 09:07:13.538 +1100 WARN X509Verify - X509 certificate (O=SplunkUser,CN=SplunkServerDefaultCert) should not be used, as it is issued by Splunk's own default Certificate Authority (CA). This pu$
03-21-2017 09:07:13.538 +1100 INFO TailReader - Registering metrics callback for: batchreader0
03-21-2017 09:07:13.538 +1100 INFO TailReader - Starting batchreader0 thread
03-21-2017 09:07:13.539 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 09:07:13.539 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 09:07:13.544 +1100 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/license_usage_summary.log'.
03-21-2017 09:07:13.551 +1100 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/remote_searches.log'.
03-21-2017 09:07:13.553 +1100 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/searchhistory.log'.
03-21-2017 09:07:13.556 +1100 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/splunkd_stdout.log'.
03-21-2017 09:07:13.558 +1100 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/splunkd_ui_access.log'.
03-21-2017 09:07:13.561 +1100 INFO WatchedFile - Will begin reading at offset=60531 for file='/opt/splunkforwarder/var/log/splunk/audit.log'.
03-21-2017 09:07:13.565 +1100 INFO WatchedFile - Will begin reading at offset=123 for file='/opt/splunkforwarder/var/log/splunk/splunkd_stderr.log'.
03-21-2017 09:07:13.568 +1100 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/btool.log'.
03-21-2017 09:07:13.576 +1100 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/scheduler.log'.
03-21-2017 09:07:13.599 +1100 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/license_usage.log'.
03-21-2017 09:07:13.602 +1100 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/mongod.log'.
03-21-2017 09:07:13.610 +1100 INFO WatchedFile - Will begin reading at offset=405521 for file='/opt/splunkforwarder/var/log/splunk/metrics.log'.
03-21-2017 09:07:43.225 +1100 WARN UserManagerPro - Can't find [distributedSearch] stanza in distsearch.conf, using default authtoken HTTP timeouts
03-21-2017 09:07:43.227 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 09:07:43.227 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 09:08:13.073 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 09:08:13.074 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 09:08:42.951 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 09:08:42.952 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 09:09:12.805 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 09:09:12.805 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 09:09:42.662 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 09:09:42.663 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 09:10:12.513 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 09:10:12.513 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 09:10:42.371 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 09:10:42.371 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 09:11:12.223 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 09:11:12.224 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 09:11:42.082 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused