topic Forwarded events not indexed in Getting Data In

Forwarded events not indexed

andresito123 — Sun, 06 Nov 2016 11:40:00 GMT

Hello!

I am preparing for the architect exam and I have set the following lab:
10.37.129.10 spl-search-head
10.37.129.11 spl-deployment-server
10.37.129.12 spl-indexer1
10.37.129.13 spl-indexer2
10.37.129.14 spl-forwarder1
10.37.129.15 spl-forwarder2
10.37.129.16 spl-forwarder3
10.37.129.17 spl-forwarder4
10.37.129.18 Checkpoint GAIA R77.30

All forwarders talk to the deployment server and I have pushed an app named "sendtoindex" to the forwarders with the following /opt/splunk/etc/deployment-apps/sendtoindexer/default/outputs.conf:

[tcpout: my_LB_indexers]
server=10.37.129.12:9997,10.37.129.13:9997
compressed=true
forceTimebasedAutoLB=true
autoLBFrequency=40
useACK=true

Then, I configured Checkpoint to send SYSLOG UDP 514 to forwarder1 and pushed the app named "syslogcheckpoint" through deployment server to forwarder1 with the following /opt/splunk/etc/deployment-apps/syslogcheckpoint/default/inputs.conf:

[udp://10.37.129.18:514]
host=10.37.129.18
connection_host = ip
sourcetype=syslog
queueSize=900MB
persistentQueueSize=5GB

In forwarder1 I have enabled tcpdump and I see the logs are delivered to forwarder. Moreover, both indexer1 and indexer2 listen to ports 9997. If I run a search to indexers (e.g. indexer1) it seems that logs are delivered to indexer1:

Search: index="_internal"  host="spl-forwarder1" syslog
11-06-2016 13:35:33.053 +0200 INFO  Metrics - group=per_sourcetype_thruput, series="syslog", kbps=0.042025, eps=0.451624, kb=1.302734, ev=14, avg_age=0.000000, max_age=0

What is wrong in my configuration? Do I have to instruct indexers with a props.conf configuration? Why logs are not indexed although sent to indexers through port 9997?

Thank you in advance for your help!

Re: Forwarded events not indexed

mattymo — Tue, 29 Sep 2020 11:41:46 GMT

Hey Andresito123!

Nice Lab setup!

The experience of working through these items will serve you well in the exam and beyond!

Your config looks good, and the fact that _internal logs are making it to the indexers means your forwarding/receiving setup looks good! Searching index=_internal and making sure all your hosts are present is a great place to start all your forwarder troubleshooting. How about searching index=_internal host=<yourforwarder> error OR warn anything interesting?

Splunk indexers have syslog as a default props, so you should be good there.

Now, Lets work from the forwarder and see what we can discover:

I noticed that your input lacks an index. Are you just trying to send to default index? out of the box, that would be the 'main' index. if you search index=main over all time....you definitely aren't receiving?

The forwarder has some really great debug commands. Try these from /opt/splunkforwarder/bin:

./splunk list foraward-server

this will confirm your active forwards (you already did this by checking _internal, but figured I'd share anyhow as it is very useful)

./splunk list inputstatus

You should see your UDP input there...how does it look?

Can I assume you are running the forwarder as root? You would need root to listen on ports lower than 1024.

How about the output of netstat -tulpn on your forwarder ( I assume you are on *nix)? Is splunkd listening on 514?

When you pushed the app, did you configure it to restart the forwarder? In the Deployment Server, it should show, "after installation - Enable app, restart splunkd". Have you tried restarting the forwarder already manually?

If you check all these and still not seeing anything, lemme know and we'll move along...

Re: Forwarded events not indexed

andresito123 — Sun, 06 Nov 2016 18:15:19 GMT

Hello mmodestino! Thank you VERY much for your comment!

Fist of all, I run the search at the indexers and today's error list is the following:

"Detected system time adjusted backwards by 1227ms."
"Detected system time adjusted backwards by 1228ms."
"Detected system time adjusted backwards by 1489ms."
"Detected system time adjusted backwards by 2127ms."
"Either time adjusted forwards by, or event loop was descheduled for 5282268ms."
"Either time adjusted forwards by, or event loop was descheduled for 5278269ms."
"Either time adjusted forwards by, or event loop was descheduled for 5278267ms."
"Either time adjusted forwards by, or event loop was descheduled for 5277353ms."
"Either time adjusted forwards by, or event loop was descheduled for 5277692ms."
"Either time adjusted forwards by, or event loop was descheduled for 7205106ms."
"Either time adjusted forwards by, or event loop was descheduled for 7196218ms."
"Either time adjusted forwards by, or event loop was descheduled for 7196216ms."
"Either time adjusted forwards by, or event loop was descheduled for 7195297ms."
"Either time adjusted forwards by, or event loop was descheduled for 7195488ms."
"Either time adjusted forwards by, or event loop was descheduled for 7196020ms."
"Either time adjusted forwards by, or event loop was descheduled for 7198834ms."
"Either time adjusted forwards by, or event loop was descheduled for 7195032ms."
"Either time adjusted forwards by, or event loop was descheduled for 7195981ms."
"Invalid Phonehome response:"
"Possible duplication of events with channel=source::/opt/splunkforwarder/var/log/splunk/metrics.log|host::spl-forwarder1|splunkd|71, streamId=0, offset=0 on host=10.37.129.12:9997"
"Possible duplication of events with channel=source::/opt/splunkforwarder/var/log/splunk/splunkd.log|host::spl-forwarder1|splunkd|73, streamId=0, offset=0 on host=10.37.129.12:9997"
"Connection to host=10.37.129.12:9997 failed"
"Connect to 10.37.129.12:9997 failed. Connection refused"
"Connection to host=10.37.129.13:9997 failed"
"Connect to 10.37.129.13:9997 failed. Connection refused"
"Can't find [distributedSearch] stanza in distsearch.conf, using default authtoken HTTP timeouts"
"X509 certificate (O=SplunkUser,CN=SplunkServerDefaultCert) should not be used, as it is issued by Splunk's own default Certificate Authority (CA). This puts your Splunk instance at very high-risk of the MITM attack. Either commercial-CA-signed or self-CA-signed certificates must be used; see: <http://docs.splunk.com/Documentation/Splunk/latest/Security/Howtoself-signcertificates>"
"Metric with name thruput:idxSummary already registered"
"Metric with name thruput:thruput already registered"
"Private key file does not exist but is defined in audit.conf - no local event signing will take place. You can create auditTrail keys if necessary by running splunk createssl audit-keys"
"Core file generation disabled"
"The hard limit of 'processes/threads' is lower than the recommended value. The hard limit is: 1899. The recommended value is: 16000."
"Restarting Splunkd..."
"Possible duplication of events with channel=source::/opt/splunkforwarder/var/log/splunk/splunkd.log|host::spl-forwarder1|splunkd|59, streamId=0, offset=0 on host=10.37.129.12:9997"
"Possible duplication of events with channel=source::/opt/splunkforwarder/var/log/splunk/metrics.log|host::spl-forwarder1|splunkd|60, streamId=0, offset=0 on host=10.37.129.12:9997"
"Applying quarantine to ip=10.37.129.12 port=9997 _numberOfFailures=2"
"Applying quarantine to ip=10.37.129.13 port=9997 _numberOfFailures=2"
"Possible duplication of events with channel=source::/opt/splunkforwarder/var/log/splunk/metrics.log|host::spl-forwarder1|splunkd|58, streamId=0, offset=0 on host=10.37.129.13:9997"
"Either time adjusted forwards by, or event loop was descheduled for 635127ms."
"The hard fd limit is lower than the recommended value. The hard limit is '4096' The recommended value is '64000'."
"helper process seems to have died (child killed by signal 15: Terminated)!"
message
"Can't find [distributedSearch] stanza in distsearch.conf, using default authtoken HTTP timeouts"
"X509 certificate (O=SplunkUser,CN=SplunkServerDefaultCert) should not be used, as it is issued by Splunk's own default Certificate Authority (CA). This puts your Splunk instance at very high-risk of the MITM attack. Either commercial-CA-signed or self-CA-signed certificates must be used; see: <http://docs.splunk.com/Documentation/Splunk/latest/Security/Howtoself-signcertificates>"
"Metric with name thruput:idxSummary already registered"
"Metric with name thruput:thruput already registered"
"Private key file does not exist but is defined in audit.conf - no local event signing will take place. You can create auditTrail keys if necessary by running splunk createssl audit-keys"
"Core file generation disabled"
"The hard fd limit is lower than the recommended value. The hard limit is '4096' The recommended value is '64000'."
"The hard limit of 'processes/threads' is lower than the recommended value. The hard limit is: 1899. The recommended value is: 16000."
"Either time adjusted forwards by, or event loop was descheduled for 5877328ms."
"Either time adjusted forwards by, or event loop was descheduled for 22788819ms."
"Possible duplication of events with channel=source::/opt/splunkforwarder/var/log/splunk/splunkd.log|host::spl-forwarder1|splunkd|114, streamId=0, offset=0 on host=10.37.129.13:9997"
"Either time adjusted forwards by, or event loop was descheduled for 7193272ms."
"Either time adjusted forwards by, or event loop was descheduled for 7190869ms."
"Either time adjusted forwards by, or event loop was descheduled for 7190871ms."
"Either time adjusted forwards by, or event loop was descheduled for 7189950ms."
"Either time adjusted forwards by, or event loop was descheduled for 7189872ms."
"Possible duplication of events with channel=source::/opt/splunkforwarder/var/log/splunk/metrics.log|host::spl-forwarder1|splunkd|109, streamId=0, offset=0 on host=10.37.129.12:9997"
"Either time adjusted forwards by, or event loop was descheduled for 613717ms."
"Either time adjusted forwards by, or event loop was descheduled for 609716ms."
"Either time adjusted forwards by, or event loop was descheduled for 608779ms."
"Either time adjusted forwards by, or event loop was descheduled for 609146ms."
"Connection to host=10.37.129.13:9997 failed"
"Connect to 10.37.129.13:9997 failed. Connection refused"
"Possible duplication of events with channel=source::/opt/splunkforwarder/var/log/splunk/splunkd.log|host::spl-forwarder1|splunkd|43, streamId=0, offset=0 on host=10.37.129.13:9997"
"Possible duplication of events with channel=source::/opt/splunkforwarder/var/log/splunk/metrics.log|host::spl-forwarder1|splunkd|44, streamId=0, offset=0 on host=10.37.129.13:9997"
"Applying quarantine to ip=10.37.129.13 port=9997 _numberOfFailures=2"
"Applying quarantine to ip=10.37.129.12 port=9997 _numberOfFailures=2"
"Connection to host=10.37.129.12:9997 failed"
"Connect to 10.37.129.12:9997 failed. Connection refused"
"Possible duplication of events with channel=source::/opt/splunkforwarder/var/log/splunk/splunkd.log|host::spl-forwarder1|splunkd|43, streamId=0, offset=0 on host=10.37.129.12:9997"
"Possible duplication of events with channel=source::/opt/splunkforwarder/var/log/splunk/metrics.log|host::spl-forwarder1|splunkd|44, streamId=0, offset=0 on host=10.37.129.12:9997"
"Restarting Splunkd..."

Regarding index, I have just let the default. My main concern was to just see the events and search them; in a latter phase I would reconfigure indexes. When I search with index=main, I get nothing both on my 2 indexers and on my search head who is configured to implement distributed search.

When I run the command, I get the following:

Fordwarder1:
root@spl-forwarder1:~# splunk list forward-server
Active forwards:
    10.37.129.13:9997
Configured but inactive forwards:
    10.37.129.12:9997

Forwarder2: 
root@spl-forwarder2:~# splunk list forward-server
Active forwards:
    10.37.129.12:9997
Configured but inactive forwards:
    10.37.129.13:9997

Forwarder3:
root@spl-forwarder3:~# /opt/splunkforwarder/bin/splunk list forward-server
Active forwards:
    10.37.129.13:9997
Configured but inactive forwards:
    10.37.129.12:9997

Forwarder4:
root@spl-forwarder4:~# /opt/splunkforwarder/bin/splunk list forward-server
Active forwards:
    10.37.129.12:9997
Configured but inactive forwards:
    10.37.129.13:9997

In forwarder1, if I run ./splunk list inputstatus, I get the following:
root@spl-forwarder1:~# splunk list inputstatus
Cooked:tcp :
tcp

Raw:tcp :
    tcp

TailingProcessor:FileStatus :
    $SPLUNK_HOME/etc/splunk.version
        file position = 70
        file size = 70
        percent = 100.00
        type = finished reading

    $SPLUNK_HOME/var/log/splunk
        type = directory

    $SPLUNK_HOME/var/log/splunk/license_usage_summary.log
        type = directory

    $SPLUNK_HOME/var/log/splunk/metrics.log
        type = directory

    $SPLUNK_HOME/var/log/splunk/splunkd.log
        type = directory

    $SPLUNK_HOME/var/spool/splunk/...stash_new
        type = directory

    /opt/splunkforwarder/var/log/splunk/audit.log
        file position = 137579
        file size = 137579
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100.00
        type = open file

    /opt/splunkforwarder/var/log/splunk/btool.log
        file position = 0
        file size = 0
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100
        type = finished reading

    /opt/splunkforwarder/var/log/splunk/conf.log
        file position = 8075
        file size = 8075
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100.00
        type = finished reading

    /opt/splunkforwarder/var/log/splunk/first_install.log
        file position = 70
        file size = 70
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100.00
        type = finished reading

    /opt/splunkforwarder/var/log/splunk/license_usage.log
        file position = 0
        file size = 0
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100
        type = finished reading

    /opt/splunkforwarder/var/log/splunk/license_usage_summary.log
        file position = 0
        file size = 0
        parent = $SPLUNK_HOME/var/log/splunk/license_usage_summary.log
        percent = 100
        type = finished reading

    /opt/splunkforwarder/var/log/splunk/metrics.log
        file position = 18301609
        file size = 18301609
        parent = $SPLUNK_HOME/var/log/splunk/metrics.log
        percent = 100.00
        type = open file

    /opt/splunkforwarder/var/log/splunk/mongod.log
        file position = 0
        file size = 0
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100
        type = finished reading

    /opt/splunkforwarder/var/log/splunk/remote_searches.log
        file position = 0
        file size = 0
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100
        type = finished reading

    /opt/splunkforwarder/var/log/splunk/scheduler.log
        file position = 0
        file size = 0
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100
        type = finished reading

    /opt/splunkforwarder/var/log/splunk/searchhistory.log
        file position = 0
        file size = 0
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100
        type = finished reading

    /opt/splunkforwarder/var/log/splunk/splunkd-utility.log
        file position = 21963
        file size = 21963
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100.00
        type = finished reading

    /opt/splunkforwarder/var/log/splunk/splunkd.log
        file position = 1369294
        file size = 1369294
        parent = $SPLUNK_HOME/var/log/splunk/splunkd.log
        percent = 100.00
        type = open file

    /opt/splunkforwarder/var/log/splunk/splunkd_access.log
        file position = 12246
        file size = 12246
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100.00
        type = open file

    /opt/splunkforwarder/var/log/splunk/splunkd_stderr.log
        file position = 3325
        file size = 3325
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100.00
        type = finished reading

    /opt/splunkforwarder/var/log/splunk/splunkd_stdout.log
        file position = 9371
        file size = 9371
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100.00
        type = finished reading

    /opt/splunkforwarder/var/log/splunk/splunkd_ui_access.log
        file position = 0
        file size = 0
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100
        type = finished reading

UDP:hosts :
    10.37.129.18

UDP:listenerports :
    514

Yes, forwarder runs as root:

root@spl-forwarder1:~# ps -ef | grep splunk
root      1030     1  0 11:16 ?        00:00:59 splunkd -p 8089 start
root      1033  1030  0 11:16 ?        00:00:00 [splunkd pid=1030] splunkd -p 8089 start [process-runner]
root      1692  1080  0 20:12 pts/0    00:00:00 grep splunk

Netstat gives the following:

root@spl-forwarder1:~# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      465/sshd        
tcp        0      0 0.0.0.0:8089            0.0.0.0:*               LISTEN      1030/splunkd    
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1015/exim4      
tcp        0      0 0.0.0.0:51706           0.0.0.0:*               LISTEN      448/rpc.statd   
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      439/rpcbind     
tcp6       0      0 :::22                   :::*                    LISTEN      465/sshd        
tcp6       0      0 ::1:25                  :::*                    LISTEN      1015/exim4      
tcp6       0      0 :::111                  :::*                    LISTEN      439/rpcbind     
tcp6       0      0 :::46163                :::*                    LISTEN      448/rpc.statd   
udp        0      0 0.0.0.0:37255           0.0.0.0:*                           411/dhclient    
udp        0      0 0.0.0.0:49322           0.0.0.0:*                           448/rpc.statd   
udp        0      0 0.0.0.0:514             0.0.0.0:*                           1030/splunkd    
udp        0      0 0.0.0.0:68              0.0.0.0:*                           411/dhclient    
udp        0      0 0.0.0.0:614             0.0.0.0:*                           439/rpcbind     
udp        0      0 0.0.0.0:111             0.0.0.0:*                           439/rpcbind     
udp        0      0 127.0.0.1:624           0.0.0.0:*                           448/rpc.statd   
udp6       0      0 :::60798                :::*                                448/rpc.statd   
udp6       0      0 :::25151                :::*                                411/dhclient    
udp6       0      0 :::614                  :::*                                439/rpcbind     
udp6       0      0 :::111                  :::*                                439/rpcbind

Regarding the deployment server's configuration, yes I have instructed to restart the forwarder. On top of that, I remember also restarting it manually...

Thank you very much again!

Re: Forwarded events not indexed

mattymo — Sun, 06 Nov 2016 20:39:41 GMT

looks like you. need ntp!

time is a VERY important aspect when working with Splunk.

I suggest getting NTP set up on all your nodes!

The rest of the config looks good at first glance, will look again closely and let you know if I find anything

While we are at it best practicing, you'll probably want to file these away for a rainy day 😉

http://www.georgestarcher.com/splunk-ulimits-and-you/

https://answers.splunk.com/answers/188875/how-do-i-disable-transparent-huge-pages-thp-and-co.html

Re: Forwarded events not indexed

mattymo — Sun, 06 Nov 2016 22:24:25 GMT

"Connection to host=10.37.129.12:9997 failed"
"Connect to 10.37.129.12:9997 failed. Connection refused"
"Connection to host=10.37.129.13:9997 failed"
"Connect to 10.37.129.13:9997 failed. Connection refused"

"Applying quarantine to ip=10.37.129.13 port=9997 _numberOfFailures=2"
"Applying quarantine to ip=10.37.129.12 port=9997 _numberOfFailures=2"

Can you telnet to your indexers from the forwarders on 9997? whats the timestamp on them things?

splunker@n00b-splkufwd-01:/opt/splunkforwarder/var/log/splunk$ cat splunkd.log | grep TcpOutputProc

Re: Forwarded events not indexed

andresito123 — Mon, 07 Nov 2016 08:57:42 GMT

Regarding NTP servers, I believe this is not the issue because:
1. All splunk instances are VMs with Parallels Tools installed which sync with my OSX (hypervisor).
2. I manually type the command date on all instances and no discrepancies are noticed.

Regarding the telnet, everything seems ok....

root@spl-forwarder1:~# nc -vvvn 10.37.129.13 9997
(UNKNOWN) [10.37.129.13] 9997 (?) open


^C sent 2, rcvd 0
root@spl-forwarder1:~# nc -vvvn 10.37.129.12 9997
(UNKNOWN) [10.37.129.12] 9997 (?) open
^C sent 0, rcvd 0

root@spl-forwarder2:~# nc -vvvn 10.37.129.13 9997
(UNKNOWN) [10.37.129.13] 9997 (?) open


^[[A^C
 sent 7, rcvd 0
root@spl-forwarder2:~# 
root@spl-forwarder2:~# nc -vvvn 10.37.129.12 9997
(UNKNOWN) [10.37.129.12] 9997 (?) open


^C sent 2, rcvd 0

root@spl-forwarder3:~# nc -vvvn 10.37.129.13 9997
(UNKNOWN) [10.37.129.13] 9997 (?) open
^C sent 0, rcvd 0
root@spl-forwarder3:~# nc -vvvn 10.37.129.12 9997
(UNKNOWN) [10.37.129.12] 9997 (?) open
^C sent 0, rcvd 0

root@spl-forwarder4:~# nc -vvvn 10.37.129.12 9997
(UNKNOWN) [10.37.129.12] 9997 (?) open


^C sent 2, rcvd 0
root@spl-forwarder4:~# nc -vvvn 10.37.129.13 9997
(UNKNOWN) [10.37.129.13] 9997 (?) open


^C sent 2, rcvd 0

"cat splunkd.log | grep TcpOutputProc" gives the following:

11-07-2016 10:37:42.711 +0200 INFO  TcpOutputProc - Closing stream for idx=10.37.129.12:9997
11-07-2016 10:37:42.711 +0200 INFO  TcpOutputProc - Connected to idx=10.37.129.13:9997 using ACK.
11-07-2016 10:38:22.559 +0200 INFO  TcpOutputProc - Closing stream for idx=10.37.129.13:9997
11-07-2016 10:38:22.560 +0200 INFO  TcpOutputProc - Connected to idx=10.37.129.12:9997 using ACK.
11-07-2016 10:41:02.161 +0200 INFO  TcpOutputProc - Closing stream for idx=10.37.129.12:9997
11-07-2016 10:41:02.161 +0200 INFO  TcpOutputProc - Connected to idx=10.37.129.13:9997 using ACK.
11-07-2016 10:42:21.909 +0200 INFO  TcpOutputProc - Closing stream for idx=10.37.129.13:9997
11-07-2016 10:42:21.909 +0200 INFO  TcpOutputProc - Connected to idx=10.37.129.12:9997 using ACK.
11-07-2016 10:43:01.757 +0200 INFO  TcpOutputProc - Closing stream for idx=10.37.129.12:9997
11-07-2016 10:43:01.758 +0200 INFO  TcpOutputProc - Connected to idx=10.37.129.13:9997 using ACK.
11-07-2016 10:43:41.622 +0200 INFO  TcpOutputProc - Closing stream for idx=10.37.129.13:9997
11-07-2016 10:43:41.623 +0200 INFO  TcpOutputProc - Connected to idx=10.37.129.12:9997 using ACK.
11-07-2016 10:45:41.292 +0200 INFO  TcpOutputProc - Closing stream for idx=10.37.129.12:9997
11-07-2016 10:45:41.293 +0200 INFO  TcpOutputProc - Connected to idx=10.37.129.13:9997 using ACK.
11-07-2016 10:46:21.148 +0200 INFO  TcpOutputProc - Closing stream for idx=10.37.129.13:9997
11-07-2016 10:46:21.149 +0200 INFO  TcpOutputProc - Connected to idx=10.37.129.12:9997 using ACK.
11-07-2016 10:47:00.995 +0200 INFO  TcpOutputProc - Closing stream for idx=10.37.129.12:9997
11-07-2016 10:47:00.996 +0200 INFO  TcpOutputProc - Connected to idx=10.37.129.13:9997 using ACK.
11-07-2016 10:49:00.602 +0200 INFO  TcpOutputProc - Closing stream for idx=10.37.129.13:9997
11-07-2016 10:49:00.603 +0200 INFO  TcpOutputProc - Connected to idx=10.37.129.12:9997 using ACK.
11-07-2016 10:50:20.385 +0200 INFO  TcpOutputProc - Closing stream for idx=10.37.129.12:9997
11-07-2016 10:50:20.386 +0200 INFO  TcpOutputProc - Connected to idx=10.37.129.13:9997 using ACK.
11-07-2016 10:51:00.312 +0200 INFO  TcpOutputProc - Closing stream for idx=10.37.129.13:9997
11-07-2016 10:51:00.312 +0200 INFO  TcpOutputProc - Connected to idx=10.37.129.12:9997 using ACK.
11-07-2016 10:52:59.923 +0200 INFO  TcpOutputProc - Closing stream for idx=10.37.129.12:9997
11-07-2016 10:52:59.923 +0200 INFO  TcpOutputProc - Connected to idx=10.37.129.13:9997 using ACK.
11-07-2016 10:53:39.772 +0200 INFO  TcpOutputProc - Closing stream for idx=10.37.129.13:9997
11-07-2016 10:53:39.773 +0200 INFO  TcpOutputProc - Connected to idx=10.37.129.12:9997 using ACK.

Re: Forwarded events not indexed

mattymo — Mon, 07 Nov 2016 12:30:57 GMT

Awesome, probably just old logs..

Ok, so everything is looking good. Are you certain that your checkpoint is sending events? How often?

Re: Forwarded events not indexed

andresito123 — Mon, 07 Nov 2016 13:44:57 GMT

Those often TcpOutputProc logs (connected & closing stream) are ok?

Regarding the logs sent, it seems they arrive quite often:
root@spl-forwarder1:/opt/splunkforwarder/var/log/splunk# tcpdump -n 'udp port 514'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
15:43:11.767797 IP 10.37.129.18.514 > 10.37.129.14.514: SYSLOG user.debug, length: 77
15:43:11.767821 IP 10.37.129.18.514 > 10.37.129.14.514: SYSLOG user.debug, length: 82
15:43:16.207078 IP 10.37.129.18.514 > 10.37.129.14.514: SYSLOG daemon.debug, length: 73
15:43:16.211091 IP 10.37.129.18.514 > 10.37.129.14.514: SYSLOG daemon.debug, length: 49
15:43:16.211123 IP 10.37.129.18.514 > 10.37.129.14.514: SYSLOG daemon.debug, length: 88
15:43:21.212108 IP 10.37.129.18.514 > 10.37.129.14.514: SYSLOG daemon.debug, length: 49
15:43:21.212162 IP 10.37.129.18.514 > 10.37.129.14.514: SYSLOG daemon.debug, length: 88
15:43:26.207075 IP 10.37.129.18.514 > 10.37.129.14.514: SYSLOG daemon.debug, length: 73
15:43:26.213010 IP 10.37.129.18.514 > 10.37.129.14.514: SYSLOG daemon.debug, length: 49
15:43:26.213039 IP 10.37.129.18.514 > 10.37.129.14.514: SYSLOG daemon.debug, length: 88
^C
10 packets captured
10 packets received by filter
0 packets dropped by kernel

Re: Forwarded events not indexed

mattymo — Mon, 07 Nov 2016 14:10:20 GMT

yep, behaving as per your outputs.conf.

all is looking well there my friend.

So, lets take a step back.

If you do directly to one of your indexers and search for these logs....anything???

Re: Forwarded events not indexed

andresito123 — Mon, 07 Nov 2016 15:13:25 GMT

No, nothing. I try even "*" but nothing seems to have been indexed...

Re: Forwarded events not indexed

mattymo — Mon, 07 Nov 2016 15:57:01 GMT

what is the output of this search, obviously changing your values accrodingly:

index=_internal source=*metrics.log host="<yourindexers>"  group=per_sourcetype_thruput series="<yourSourcetype>"

Also if you go to settings > indexes .....is your main index growing???

Re: Forwarded events not indexed

andresito123 — Tue, 29 Sep 2020 11:42:33 GMT

Main index is 1MB for both indexers. 😞

Search output is for indexer1:
11/7/16
7:31:36.380 PM

11-07-2016 19:31:36.380 +0200 INFO Metrics - group=per_sourcetype_thruput, series="syslog", kbps=0.048009, eps=0.483869, kb=1.488281, ev=15, avg_age=0.000000, max_age=0

    host =  spl-forwarder1  
    source =    /opt/splunkforwarder/var/log/splunk/metrics.log 
    sourcetype =    splunkd 

    11/7/16
7:31:05.380 PM  
11-07-2016 19:31:05.380 +0200 INFO  Metrics - group=per_sourcetype_thruput, series="syslog", kbps=0.078911, eps=0.838695, kb=2.446289, ev=26, avg_age=0.000000, max_age=0

    host =  spl-forwarder1  
    source =    /opt/splunkforwarder/var/log/splunk/metrics.log 
    sourcetype =    splunkd 

    11/7/16
7:30:34.379 PM  
11-07-2016 19:30:34.379 +0200 INFO  Metrics - group=per_sourcetype_thruput, series="syslog", kbps=0.065022, eps=0.677440, kb=2.015625, ev=21, avg_age=0.000000, max_age=0

    host =  spl-forwarder1  
    source =    /opt/splunkforwarder/var/log/splunk/metrics.log 
    sourcetype =    splunkd 

    11/7/16
7:29:32.380 PM  
11-07-2016 19:29:32.380 +0200 INFO  Metrics - group=per_sourcetype_thruput, series="syslog", kbps=0.060460, eps=0.624364, kb=1.839844, ev=19, avg_age=0.000000, max_age=0

For indexer2:

    11/7/16
7:32:38.380 PM  
11-07-2016 19:32:38.380 +0200 INFO  Metrics - group=per_sourcetype_thruput, series="syslog", kbps=0.058120, eps=0.580630, kb=1.801758, ev=18, avg_age=0.000000, max_age=0

    host =  spl-forwarder1  
    source =    /opt/splunkforwarder/var/log/splunk/metrics.log 
    sourcetype =    splunkd 

    11/7/16
7:32:07.380 PM  
11-07-2016 19:32:07.380 +0200 INFO  Metrics - group=per_sourcetype_thruput, series="syslog", kbps=0.048010, eps=0.483882, kb=1.488281, ev=15, avg_age=0.000000, max_age=0

    host =  spl-forwarder1  
    source =    /opt/splunkforwarder/var/log/splunk/metrics.log 
    sourcetype =    splunkd 

    11/7/16
7:30:03.380 PM  
11-07-2016 19:30:03.380 +0200 INFO  Metrics - group=per_sourcetype_thruput, series="syslog", kbps=0.050372, eps=0.516128, kb=1.561523, ev=16, avg_age=0.000000, max_age=0

It's started to get a little bit crazy... :S

Re: Forwarded events not indexed

mattymo — Tue, 08 Nov 2016 01:00:38 GMT

Hmmm.....back to the drawing board...

How bout this...

Rebuild your inputs.conf to accept from any host, rather than specifying the sender. Let's see if wide open UDP changes anything....

Re: Forwarded events not indexed

andresito123 — Tue, 08 Nov 2016 12:06:27 GMT

I changed the recipient of the logs to forwarder2 and explicitly used the "/opt/splunkforwarder/etc/system/local/inputs.conf" to include:
[udp:514]
sourcetype = syslog

but still no luck on the indexers....

Re: Forwarded events not indexed

mattymo — Tue, 08 Nov 2016 12:16:13 GMT

wow..cant give up now! MUST. KNOW.WHY

lets check btool

./splunk btool inputs list --debug

im going to set it up in my lab. i use rsyslog to write to disk, but lemme set up udp as well

Re: Forwarded events not indexed

mattymo — Tue, 08 Nov 2016 12:24:08 GMT

lets look at btool before I go set it up in my lab. I have a very similar set up, but i catch the logs with rsyslog then tail the file. Try:

splunker@n00b-splkufwd-01:/opt/splunkforwarder/bin$ ./splunk btool inputs list udp --debug
/opt/splunkforwarder/etc/system/default/inputs.conf [udp]
/opt/splunkforwarder/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunkforwarder/etc/system/default/inputs.conf connection_host = ip
/opt/splunkforwarder/etc/system/local/inputs.conf   host = n00b-splkufwd-01
/opt/splunkforwarder/etc/system/default/inputs.conf index = default

Re: Forwarded events not indexed

mattymo — Thu, 10 Nov 2016 16:23:24 GMT

still working on this one?

Re: Forwarded events not indexed

mattymo — Thu, 10 Nov 2016 16:24:54 GMT

still battling this?

Re: Forwarded events not indexed

andresito123 — Thu, 10 Nov 2016 19:24:33 GMT

Hi mmodestino,

I got lost in this forum!

The latest update is that I "disconnected" forwarder4 from the deployment server and created an ad-hoc connnection with the indexers, just to see if the deployed app from the forwarder had any issues. But no luck.

Then, I opened rsyslog on the forwarder4 and recorded all logs from syslog to /var/log/checkpoint.log. Then I changed the forwarder instead from listening to 514 just to monitor /var/log/checkpoint.log. But still, still, no luck.

So I believe it's a configuration issue on my indexers...

Re: Forwarded events not indexed

mattymo — Thu, 10 Nov 2016 21:06:59 GMT

Great troubleshooting step! Writing to log is best practice anyhow....

So what does ./splunk list inputstatus say about that log?

What about if you grep /opt/splunkforwarder/var/log/metrics.log, you seeing any 'blocked=true'?

Or is it saying its sending your sourcetype?

This one is killing me. I'm so close to sending you a webex to see this for myself LOL