Forwarded events not indexed

[andresito123]

Hello!

I am preparing for the architect exam and I have set the following lab:
10.37.129.10 spl-search-head
10.37.129.11 spl-deployment-server
10.37.129.12 spl-indexer1
10.37.129.13 spl-indexer2
10.37.129.14 spl-forwarder1
10.37.129.15 spl-forwarder2
10.37.129.16 spl-forwarder3
10.37.129.17 spl-forwarder4
10.37.129.18 Checkpoint GAIA R77.30

All forwarders talk to the deployment server and I have pushed an app named "sendtoindex" to the forwarders with the following /opt/splunk/etc/deployment-apps/sendtoindexer/default/outputs.conf:

[tcpout: my_LB_indexers]
server=10.37.129.12:9997,10.37.129.13:9997
compressed=true
forceTimebasedAutoLB=true
autoLBFrequency=40
useACK=true

Then, I configured Checkpoint to send SYSLOG UDP 514 to forwarder1 and pushed the app named "syslogcheckpoint" through deployment server to forwarder1 with the following /opt/splunk/etc/deployment-apps/syslogcheckpoint/default/inputs.conf:

[udp://10.37.129.18:514]
host=10.37.129.18
connection_host = ip
sourcetype=syslog
queueSize=900MB
persistentQueueSize=5GB

In forwarder1 I have enabled tcpdump and I see the logs are delivered to forwarder. Moreover, both indexer1 and indexer2 listen to ports 9997. If I run a search to indexers (e.g. indexer1) it seems that logs are delivered to indexer1:

Search: index="_internal"  host="spl-forwarder1" syslog
11-06-2016 13:35:33.053 +0200 INFO  Metrics - group=per_sourcetype_thruput, series="syslog", kbps=0.042025, eps=0.451624, kb=1.302734, ev=14, avg_age=0.000000, max_age=0

What is wrong in my configuration? Do I have to instruct indexers with a props.conf configuration? Why logs are not indexed although sent to indexers through port 9997?

Thank you in advance for your help!

[andresito123 · Accepted solution]

I changed outputs.conf from

[tcpout: my_LB_indexers]
server=10.37.129.12:9997,10.37.129.13:9997
forceTimebasedAutoLB=true
autoLBFrequency=40
compressed=true

to:

[tcpout:my_LB_indexers]
server=10.37.129.12:9997,10.37.129.13:9997
forceTimebasedAutoLB=true
autoLBFrequency=40
compressed=true

(deleted space after "tcpout:") and seems that this typo was the issue!...

Still can't believe it; I am refreshing the page because I can't believe that the indexers are growing!...

[mattymo]

HAHAAH! did startup not balk at that? or any config checks? btool?

[andresito123]

No clues my friend... Personally, I believe there is room for improvement regarding btool!..

Thank you very much for your help! 🙂

[fz]

Hi Andresito123!

I can see that you have been using "compressed=true" attribute in the outputs.conf file in your forwarders.

Does your Indexer also holds the same attribute "compressed=true" in inputs.conf file?

Because you need to enable the compression on the indexer side as well.

[mattymo]

ahhh good eye fz!! That's gotta be it!!!

compressed = [true|false]
* Applies to non-SSL forwarding only. For SSL useClientSSLCompression
  setting is used.
* If true, forwarder sends compressed data.
* If set to true, the receiver port must also have compression turned on (in
  its inputs.conf file).
* Defaults to false.

[fz]

Exactly mmodestino!
Thanks for the compliment.!

[andresito123]

1. I removed compressed from the forwarders and the problem persists...
2. I entered compressed both on forwarders output.conf and indexers' input.conf and the problem persists...

This is crazy.

[mattymo]

Ha! Wow. I was sure that was going to get ya going.

[mattymo]

still battling this?

[andresito123]

Hi mmodestino,

I got lost in this forum!

The latest update is that I "disconnected" forwarder4 from the deployment server and created an ad-hoc connnection with the indexers, just to see if the deployed app from the forwarder had any issues. But no luck.

Then, I opened rsyslog on the forwarder4 and recorded all logs from syslog to /var/log/checkpoint.log. Then I changed the forwarder instead from listening to 514 just to monitor /var/log/checkpoint.log. But still, still, no luck.

So I believe it's a configuration issue on my indexers...

[mattymo]

Great troubleshooting step! Writing to log is best practice anyhow....

So what does ./splunk list inputstatus say about that log?

What about if you grep /opt/splunkforwarder/var/log/metrics.log, you seeing any 'blocked=true'?

Or is it saying its sending your sourcetype?

This one is killing me. I'm so close to sending you a webex to see this for myself LOL

[andresito123]

Haahhahha! I know, imagine myself! I tried to set up a lab for my exam and I got stuck troubleshooting indefinitely!..

So, inputstatus gives the following:

    /var/log/checkpoint.log
        file position = 8283370
        file size = 8283370
        percent = 100.00
        type = open file

"cat /opt/splunkforwarder/var/log/splunk/metrics.log | grep blocked" gives me nothing...

I firmly believe it's something on the indexer... :S

[andresito123]

Today, I took another troubleshooting step that may help you resolving this (hope so!):

Opened UDP port at indexer1 and voila, the index started to grow!...

[horsefez]

Hi andresito123,

while working with splunk I noticed strange behavior before, when I used a port less than 1024 for the receiving.
Also, your inputs.conf stanza does not have a "index=..." configuration.

By no means I think this will solve anything!
Anyway... I wish you good luck!

[andresito123]

You mean port 514 UDP?

[andresito123]

Hello mmodestino! Thank you VERY much for your comment!

Fist of all, I run the search at the indexers and today's error list is the following:

"Detected system time adjusted backwards by 1227ms."
"Detected system time adjusted backwards by 1228ms."
"Detected system time adjusted backwards by 1489ms."
"Detected system time adjusted backwards by 2127ms."
"Either time adjusted forwards by, or event loop was descheduled for 5282268ms."
"Either time adjusted forwards by, or event loop was descheduled for 5278269ms."
"Either time adjusted forwards by, or event loop was descheduled for 5278267ms."
"Either time adjusted forwards by, or event loop was descheduled for 5277353ms."
"Either time adjusted forwards by, or event loop was descheduled for 5277692ms."
"Either time adjusted forwards by, or event loop was descheduled for 7205106ms."
"Either time adjusted forwards by, or event loop was descheduled for 7196218ms."
"Either time adjusted forwards by, or event loop was descheduled for 7196216ms."
"Either time adjusted forwards by, or event loop was descheduled for 7195297ms."
"Either time adjusted forwards by, or event loop was descheduled for 7195488ms."
"Either time adjusted forwards by, or event loop was descheduled for 7196020ms."
"Either time adjusted forwards by, or event loop was descheduled for 7198834ms."
"Either time adjusted forwards by, or event loop was descheduled for 7195032ms."
"Either time adjusted forwards by, or event loop was descheduled for 7195981ms."
"Invalid Phonehome response:"
"Possible duplication of events with channel=source::/opt/splunkforwarder/var/log/splunk/metrics.log|host::spl-forwarder1|splunkd|71, streamId=0, offset=0 on host=10.37.129.12:9997"
"Possible duplication of events with channel=source::/opt/splunkforwarder/var/log/splunk/splunkd.log|host::spl-forwarder1|splunkd|73, streamId=0, offset=0 on host=10.37.129.12:9997"
"Connection to host=10.37.129.12:9997 failed"
"Connect to 10.37.129.12:9997 failed. Connection refused"
"Connection to host=10.37.129.13:9997 failed"
"Connect to 10.37.129.13:9997 failed. Connection refused"
"Can't find [distributedSearch] stanza in distsearch.conf, using default authtoken HTTP timeouts"
"X509 certificate (O=SplunkUser,CN=SplunkServerDefaultCert) should not be used, as it is issued by Splunk's own default Certificate Authority (CA). This puts your Splunk instance at very high-risk of the MITM attack. Either commercial-CA-signed or self-CA-signed certificates must be used; see: <http://docs.splunk.com/Documentation/Splunk/latest/Security/Howtoself-signcertificates>"
"Metric with name thruput:idxSummary already registered"
"Metric with name thruput:thruput already registered"
"Private key file does not exist but is defined in audit.conf - no local event signing will take place. You can create auditTrail keys if necessary by running splunk createssl audit-keys"
"Core file generation disabled"
"The hard limit of 'processes/threads' is lower than the recommended value. The hard limit is: 1899. The recommended value is: 16000."
"Restarting Splunkd..."
"Possible duplication of events with channel=source::/opt/splunkforwarder/var/log/splunk/splunkd.log|host::spl-forwarder1|splunkd|59, streamId=0, offset=0 on host=10.37.129.12:9997"
"Possible duplication of events with channel=source::/opt/splunkforwarder/var/log/splunk/metrics.log|host::spl-forwarder1|splunkd|60, streamId=0, offset=0 on host=10.37.129.12:9997"
"Applying quarantine to ip=10.37.129.12 port=9997 _numberOfFailures=2"
"Applying quarantine to ip=10.37.129.13 port=9997 _numberOfFailures=2"
"Possible duplication of events with channel=source::/opt/splunkforwarder/var/log/splunk/metrics.log|host::spl-forwarder1|splunkd|58, streamId=0, offset=0 on host=10.37.129.13:9997"
"Either time adjusted forwards by, or event loop was descheduled for 635127ms."
"The hard fd limit is lower than the recommended value. The hard limit is '4096' The recommended value is '64000'."
"helper process seems to have died (child killed by signal 15: Terminated)!"
message
"Can't find [distributedSearch] stanza in distsearch.conf, using default authtoken HTTP timeouts"
"X509 certificate (O=SplunkUser,CN=SplunkServerDefaultCert) should not be used, as it is issued by Splunk's own default Certificate Authority (CA). This puts your Splunk instance at very high-risk of the MITM attack. Either commercial-CA-signed or self-CA-signed certificates must be used; see: <http://docs.splunk.com/Documentation/Splunk/latest/Security/Howtoself-signcertificates>"
"Metric with name thruput:idxSummary already registered"
"Metric with name thruput:thruput already registered"
"Private key file does not exist but is defined in audit.conf - no local event signing will take place. You can create auditTrail keys if necessary by running splunk createssl audit-keys"
"Core file generation disabled"
"The hard fd limit is lower than the recommended value. The hard limit is '4096' The recommended value is '64000'."
"The hard limit of 'processes/threads' is lower than the recommended value. The hard limit is: 1899. The recommended value is: 16000."
"Either time adjusted forwards by, or event loop was descheduled for 5877328ms."
"Either time adjusted forwards by, or event loop was descheduled for 22788819ms."
"Possible duplication of events with channel=source::/opt/splunkforwarder/var/log/splunk/splunkd.log|host::spl-forwarder1|splunkd|114, streamId=0, offset=0 on host=10.37.129.13:9997"
"Either time adjusted forwards by, or event loop was descheduled for 7193272ms."
"Either time adjusted forwards by, or event loop was descheduled for 7190869ms."
"Either time adjusted forwards by, or event loop was descheduled for 7190871ms."
"Either time adjusted forwards by, or event loop was descheduled for 7189950ms."
"Either time adjusted forwards by, or event loop was descheduled for 7189872ms."
"Possible duplication of events with channel=source::/opt/splunkforwarder/var/log/splunk/metrics.log|host::spl-forwarder1|splunkd|109, streamId=0, offset=0 on host=10.37.129.12:9997"
"Either time adjusted forwards by, or event loop was descheduled for 613717ms."
"Either time adjusted forwards by, or event loop was descheduled for 609716ms."
"Either time adjusted forwards by, or event loop was descheduled for 608779ms."
"Either time adjusted forwards by, or event loop was descheduled for 609146ms."
"Connection to host=10.37.129.13:9997 failed"
"Connect to 10.37.129.13:9997 failed. Connection refused"
"Possible duplication of events with channel=source::/opt/splunkforwarder/var/log/splunk/splunkd.log|host::spl-forwarder1|splunkd|43, streamId=0, offset=0 on host=10.37.129.13:9997"
"Possible duplication of events with channel=source::/opt/splunkforwarder/var/log/splunk/metrics.log|host::spl-forwarder1|splunkd|44, streamId=0, offset=0 on host=10.37.129.13:9997"
"Applying quarantine to ip=10.37.129.13 port=9997 _numberOfFailures=2"
"Applying quarantine to ip=10.37.129.12 port=9997 _numberOfFailures=2"
"Connection to host=10.37.129.12:9997 failed"
"Connect to 10.37.129.12:9997 failed. Connection refused"
"Possible duplication of events with channel=source::/opt/splunkforwarder/var/log/splunk/splunkd.log|host::spl-forwarder1|splunkd|43, streamId=0, offset=0 on host=10.37.129.12:9997"
"Possible duplication of events with channel=source::/opt/splunkforwarder/var/log/splunk/metrics.log|host::spl-forwarder1|splunkd|44, streamId=0, offset=0 on host=10.37.129.12:9997"
"Restarting Splunkd..."

Regarding index, I have just let the default. My main concern was to just see the events and search them; in a latter phase I would reconfigure indexes. When I search with index=main, I get nothing both on my 2 indexers and on my search head who is configured to implement distributed search.

When I run the command, I get the following:

Fordwarder1:
root@spl-forwarder1:~# splunk list forward-server
Active forwards:
    10.37.129.13:9997
Configured but inactive forwards:
    10.37.129.12:9997

Forwarder2: 
root@spl-forwarder2:~# splunk list forward-server
Active forwards:
    10.37.129.12:9997
Configured but inactive forwards:
    10.37.129.13:9997

Forwarder3:
root@spl-forwarder3:~# /opt/splunkforwarder/bin/splunk list forward-server
Active forwards:
    10.37.129.13:9997
Configured but inactive forwards:
    10.37.129.12:9997

Forwarder4:
root@spl-forwarder4:~# /opt/splunkforwarder/bin/splunk list forward-server
Active forwards:
    10.37.129.12:9997
Configured but inactive forwards:
    10.37.129.13:9997

In forwarder1, if I run ./splunk list inputstatus, I get the following:
root@spl-forwarder1:~# splunk list inputstatus
Cooked:tcp :
tcp

Raw:tcp :
    tcp

TailingProcessor:FileStatus :
    $SPLUNK_HOME/etc/splunk.version
        file position = 70
        file size = 70
        percent = 100.00
        type = finished reading

    $SPLUNK_HOME/var/log/splunk
        type = directory

    $SPLUNK_HOME/var/log/splunk/license_usage_summary.log
        type = directory

    $SPLUNK_HOME/var/log/splunk/metrics.log
        type = directory

    $SPLUNK_HOME/var/log/splunk/splunkd.log
        type = directory

    $SPLUNK_HOME/var/spool/splunk/...stash_new
        type = directory

    /opt/splunkforwarder/var/log/splunk/audit.log
        file position = 137579
        file size = 137579
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100.00
        type = open file

    /opt/splunkforwarder/var/log/splunk/btool.log
        file position = 0
        file size = 0
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100
        type = finished reading

    /opt/splunkforwarder/var/log/splunk/conf.log
        file position = 8075
        file size = 8075
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100.00
        type = finished reading

    /opt/splunkforwarder/var/log/splunk/first_install.log
        file position = 70
        file size = 70
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100.00
        type = finished reading

    /opt/splunkforwarder/var/log/splunk/license_usage.log
        file position = 0
        file size = 0
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100
        type = finished reading

    /opt/splunkforwarder/var/log/splunk/license_usage_summary.log
        file position = 0
        file size = 0
        parent = $SPLUNK_HOME/var/log/splunk/license_usage_summary.log
        percent = 100
        type = finished reading

    /opt/splunkforwarder/var/log/splunk/metrics.log
        file position = 18301609
        file size = 18301609
        parent = $SPLUNK_HOME/var/log/splunk/metrics.log
        percent = 100.00
        type = open file

    /opt/splunkforwarder/var/log/splunk/mongod.log
        file position = 0
        file size = 0
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100
        type = finished reading

    /opt/splunkforwarder/var/log/splunk/remote_searches.log
        file position = 0
        file size = 0
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100
        type = finished reading

    /opt/splunkforwarder/var/log/splunk/scheduler.log
        file position = 0
        file size = 0
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100
        type = finished reading

    /opt/splunkforwarder/var/log/splunk/searchhistory.log
        file position = 0
        file size = 0
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100
        type = finished reading

    /opt/splunkforwarder/var/log/splunk/splunkd-utility.log
        file position = 21963
        file size = 21963
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100.00
        type = finished reading

    /opt/splunkforwarder/var/log/splunk/splunkd.log
        file position = 1369294
        file size = 1369294
        parent = $SPLUNK_HOME/var/log/splunk/splunkd.log
        percent = 100.00
        type = open file

    /opt/splunkforwarder/var/log/splunk/splunkd_access.log
        file position = 12246
        file size = 12246
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100.00
        type = open file

    /opt/splunkforwarder/var/log/splunk/splunkd_stderr.log
        file position = 3325
        file size = 3325
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100.00
        type = finished reading

    /opt/splunkforwarder/var/log/splunk/splunkd_stdout.log
        file position = 9371
        file size = 9371
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100.00
        type = finished reading

    /opt/splunkforwarder/var/log/splunk/splunkd_ui_access.log
        file position = 0
        file size = 0
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100
        type = finished reading

UDP:hosts :
    10.37.129.18

UDP:listenerports :
    514

Yes, forwarder runs as root:

root@spl-forwarder1:~# ps -ef | grep splunk
root      1030     1  0 11:16 ?        00:00:59 splunkd -p 8089 start
root      1033  1030  0 11:16 ?        00:00:00 [splunkd pid=1030] splunkd -p 8089 start [process-runner]
root      1692  1080  0 20:12 pts/0    00:00:00 grep splunk

Netstat gives the following:

root@spl-forwarder1:~# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      465/sshd        
tcp        0      0 0.0.0.0:8089            0.0.0.0:*               LISTEN      1030/splunkd    
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1015/exim4      
tcp        0      0 0.0.0.0:51706           0.0.0.0:*               LISTEN      448/rpc.statd   
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      439/rpcbind     
tcp6       0      0 :::22                   :::*                    LISTEN      465/sshd        
tcp6       0      0 ::1:25                  :::*                    LISTEN      1015/exim4      
tcp6       0      0 :::111                  :::*                    LISTEN      439/rpcbind     
tcp6       0      0 :::46163                :::*                    LISTEN      448/rpc.statd   
udp        0      0 0.0.0.0:37255           0.0.0.0:*                           411/dhclient    
udp        0      0 0.0.0.0:49322           0.0.0.0:*                           448/rpc.statd   
udp        0      0 0.0.0.0:514             0.0.0.0:*                           1030/splunkd    
udp        0      0 0.0.0.0:68              0.0.0.0:*                           411/dhclient    
udp        0      0 0.0.0.0:614             0.0.0.0:*                           439/rpcbind     
udp        0      0 0.0.0.0:111             0.0.0.0:*                           439/rpcbind     
udp        0      0 127.0.0.1:624           0.0.0.0:*                           448/rpc.statd   
udp6       0      0 :::60798                :::*                                448/rpc.statd   
udp6       0      0 :::25151                :::*                                411/dhclient    
udp6       0      0 :::614                  :::*                                439/rpcbind     
udp6       0      0 :::111                  :::*                                439/rpcbind  

Regarding the deployment server's configuration, yes I have instructed to restart the forwarder. On top of that, I remember also restarting it manually...

Thank you very much again!

[mattymo]

looks like you. need ntp!

time is a VERY important aspect when working with Splunk.

I suggest getting NTP set up on all your nodes!

The rest of the config looks good at first glance, will look again closely and let you know if I find anything

While we are at it best practicing, you'll probably want to file these away for a rainy day 😉

http://www.georgestarcher.com/splunk-ulimits-and-you/

https://answers.splunk.com/answers/188875/how-do-i-disable-transparent-huge-pages-thp-and-co.html

[andresito123]

Regarding NTP servers, I believe this is not the issue because:
1. All splunk instances are VMs with Parallels Tools installed which sync with my OSX (hypervisor).
2. I manually type the command date on all instances and no discrepancies are noticed.

Regarding the telnet, everything seems ok....

root@spl-forwarder1:~# nc -vvvn 10.37.129.13 9997
(UNKNOWN) [10.37.129.13] 9997 (?) open


^C sent 2, rcvd 0
root@spl-forwarder1:~# nc -vvvn 10.37.129.12 9997
(UNKNOWN) [10.37.129.12] 9997 (?) open
^C sent 0, rcvd 0

root@spl-forwarder2:~# nc -vvvn 10.37.129.13 9997
(UNKNOWN) [10.37.129.13] 9997 (?) open


^[[A^C
 sent 7, rcvd 0
root@spl-forwarder2:~# 
root@spl-forwarder2:~# nc -vvvn 10.37.129.12 9997
(UNKNOWN) [10.37.129.12] 9997 (?) open


^C sent 2, rcvd 0

root@spl-forwarder3:~# nc -vvvn 10.37.129.13 9997
(UNKNOWN) [10.37.129.13] 9997 (?) open
^C sent 0, rcvd 0
root@spl-forwarder3:~# nc -vvvn 10.37.129.12 9997
(UNKNOWN) [10.37.129.12] 9997 (?) open
^C sent 0, rcvd 0

root@spl-forwarder4:~# nc -vvvn 10.37.129.12 9997
(UNKNOWN) [10.37.129.12] 9997 (?) open


^C sent 2, rcvd 0
root@spl-forwarder4:~# nc -vvvn 10.37.129.13 9997
(UNKNOWN) [10.37.129.13] 9997 (?) open


^C sent 2, rcvd 0

"cat splunkd.log | grep TcpOutputProc" gives the following:

11-07-2016 10:37:42.711 +0200 INFO  TcpOutputProc - Closing stream for idx=10.37.129.12:9997
11-07-2016 10:37:42.711 +0200 INFO  TcpOutputProc - Connected to idx=10.37.129.13:9997 using ACK.
11-07-2016 10:38:22.559 +0200 INFO  TcpOutputProc - Closing stream for idx=10.37.129.13:9997
11-07-2016 10:38:22.560 +0200 INFO  TcpOutputProc - Connected to idx=10.37.129.12:9997 using ACK.
11-07-2016 10:41:02.161 +0200 INFO  TcpOutputProc - Closing stream for idx=10.37.129.12:9997
11-07-2016 10:41:02.161 +0200 INFO  TcpOutputProc - Connected to idx=10.37.129.13:9997 using ACK.
11-07-2016 10:42:21.909 +0200 INFO  TcpOutputProc - Closing stream for idx=10.37.129.13:9997
11-07-2016 10:42:21.909 +0200 INFO  TcpOutputProc - Connected to idx=10.37.129.12:9997 using ACK.
11-07-2016 10:43:01.757 +0200 INFO  TcpOutputProc - Closing stream for idx=10.37.129.12:9997
11-07-2016 10:43:01.758 +0200 INFO  TcpOutputProc - Connected to idx=10.37.129.13:9997 using ACK.
11-07-2016 10:43:41.622 +0200 INFO  TcpOutputProc - Closing stream for idx=10.37.129.13:9997
11-07-2016 10:43:41.623 +0200 INFO  TcpOutputProc - Connected to idx=10.37.129.12:9997 using ACK.
11-07-2016 10:45:41.292 +0200 INFO  TcpOutputProc - Closing stream for idx=10.37.129.12:9997
11-07-2016 10:45:41.293 +0200 INFO  TcpOutputProc - Connected to idx=10.37.129.13:9997 using ACK.
11-07-2016 10:46:21.148 +0200 INFO  TcpOutputProc - Closing stream for idx=10.37.129.13:9997
11-07-2016 10:46:21.149 +0200 INFO  TcpOutputProc - Connected to idx=10.37.129.12:9997 using ACK.
11-07-2016 10:47:00.995 +0200 INFO  TcpOutputProc - Closing stream for idx=10.37.129.12:9997
11-07-2016 10:47:00.996 +0200 INFO  TcpOutputProc - Connected to idx=10.37.129.13:9997 using ACK.
11-07-2016 10:49:00.602 +0200 INFO  TcpOutputProc - Closing stream for idx=10.37.129.13:9997
11-07-2016 10:49:00.603 +0200 INFO  TcpOutputProc - Connected to idx=10.37.129.12:9997 using ACK.
11-07-2016 10:50:20.385 +0200 INFO  TcpOutputProc - Closing stream for idx=10.37.129.12:9997
11-07-2016 10:50:20.386 +0200 INFO  TcpOutputProc - Connected to idx=10.37.129.13:9997 using ACK.
11-07-2016 10:51:00.312 +0200 INFO  TcpOutputProc - Closing stream for idx=10.37.129.13:9997
11-07-2016 10:51:00.312 +0200 INFO  TcpOutputProc - Connected to idx=10.37.129.12:9997 using ACK.
11-07-2016 10:52:59.923 +0200 INFO  TcpOutputProc - Closing stream for idx=10.37.129.12:9997
11-07-2016 10:52:59.923 +0200 INFO  TcpOutputProc - Connected to idx=10.37.129.13:9997 using ACK.
11-07-2016 10:53:39.772 +0200 INFO  TcpOutputProc - Closing stream for idx=10.37.129.13:9997
11-07-2016 10:53:39.773 +0200 INFO  TcpOutputProc - Connected to idx=10.37.129.12:9997 using ACK.

[mattymo]

Awesome, probably just old logs..

Ok, so everything is looking good. Are you certain that your checkpoint is sending events? How often?

[andresito123]

Those often TcpOutputProc logs (connected & closing stream) are ok?

Regarding the logs sent, it seems they arrive quite often:
root@spl-forwarder1:/opt/splunkforwarder/var/log/splunk# tcpdump -n 'udp port 514'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
15:43:11.767797 IP 10.37.129.18.514 > 10.37.129.14.514: SYSLOG user.debug, length: 77
15:43:11.767821 IP 10.37.129.18.514 > 10.37.129.14.514: SYSLOG user.debug, length: 82
15:43:16.207078 IP 10.37.129.18.514 > 10.37.129.14.514: SYSLOG daemon.debug, length: 73
15:43:16.211091 IP 10.37.129.18.514 > 10.37.129.14.514: SYSLOG daemon.debug, length: 49
15:43:16.211123 IP 10.37.129.18.514 > 10.37.129.14.514: SYSLOG daemon.debug, length: 88
15:43:21.212108 IP 10.37.129.18.514 > 10.37.129.14.514: SYSLOG daemon.debug, length: 49
15:43:21.212162 IP 10.37.129.18.514 > 10.37.129.14.514: SYSLOG daemon.debug, length: 88
15:43:26.207075 IP 10.37.129.18.514 > 10.37.129.14.514: SYSLOG daemon.debug, length: 73
15:43:26.213010 IP 10.37.129.18.514 > 10.37.129.14.514: SYSLOG daemon.debug, length: 49
15:43:26.213039 IP 10.37.129.18.514 > 10.37.129.14.514: SYSLOG daemon.debug, length: 88
^C
10 packets captured
10 packets received by filter
0 packets dropped by kernel