Splunk Connect For Syslog - no data indexed

[corti77]

Hi,

I followed the official instructions and deployed syslog connect in Ubuntu using docker.
https://splunk.github.io/splunk-connect-for-syslog/1266/gettingstarted/docker-systemd-general/

after I run 

sudo systemctl start sc4s

and, according to the documentation, I see the correct event appearing in Splunk. so the connectivity to splunk is correct.

then I wanted to start logging from our Citrix NetScaler appliance. I followed the instructions from
https://splunk.github.io/splunk-connect-for-syslog/1266/sources/Citrix/
and created the index netfw in Splunk.

I modified the file splunk_metadata.csv with the two lines

citrix_netscaler,index,netfw
citrix_netscaler,sourcetype,citrix:netscaler:syslog

I was not sure if it was needed but I restarted the container to read the new configuration file.

systemctl restart sc4s

But I checked in Splunk and no data is coming in that index (0 events)

I checked the local FW in the Ubuntu and it is disabled. In fact I run tcpdump and I see syslog packages coming from the netscaler appliance.

:/opt/sc4s/local/context$ sudo tcpdump -nnSX port 514
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
17:11:29.533661 IP 192.168.115.159.54092 > 192.168.105.202.514: SYSLOG local5.info, length: 270
        0x0000:  4500 012a bdc7 0000 ff11 9e40 c0a8 739f  E..*.......@..s.
        0x0010:  c0a8 69ca d34c 0202 0116 9c65 3c31 3734  ..i..L.....e<174
        0x0020:  3e20 3232 2f30 392f 3230 3231 3a31 353a  >.22/09/2021:15:
        0x0030:  3131 3a32 3920 474d 5420 4154 4c48 514d  11:29.GMT.ATLHQM
        0x0040:  564d 534c 4232 2030 2d50 5045 2d30 203a  VMSLB2.0-PPE-0.:
        0x0050:  2064 6566 6175 6c74 2054 4350 2043 4f4e  .default.TCP.CON
        0x0060:  4e5f 5445 524d 494e 4154 4520 3435 3635  N_TERMINATE.4565
        0x0070:  3733 3920 3020 3a20 2053 6f75 7263 6520  739.0.:..Source.
        0x0080:  3139 322e 3136 382e 3130 352e 3535 3a34  192.168.105.55:4
        0x0090:  3433 202d 2044 6573 7469 6e61 7469 6f6e  43.-.Destination
        0x00a0:  2031 3932 2e31 3638 2e31 3035 2e31 3630  .192.168.105.160
        0x00b0:  3a34 3238 3737 202d 2053 7461 7274 2054  :42877.-.Start.T
        0x00c0:  696d 6520 3232 2f30 392f 3230 3231 3a31  ime.22/09/2021:1
        0x00d0:  353a 3131 3a32 3920 474d 5420 2d20 456e  5:11:29.GMT.-.En
        0x00e0:  6420 5469 6d65 2032 322f 3039 2f32 3032  d.Time.22/09/202
        0x00f0:  313a 3135 3a31 313a 3239 2047 4d54 202d  1:15:11:29.GMT.-
        0x0100:  2054 6f74 616c 5f62 7974 6573 5f73 656e  .Total_bytes_sen
        0x0110:  6420 3020 2d20 546f 7461 6c5f 6279 7465  d.0.-.Total_byte
        0x0120:  735f 7265 6376 2031 200a                 s_recv.1..
17:11:29.689892 IP 192.168.115.159.54092 > 192.168.105.202.514: SYSLOG local5.info, length: 271
        0x0000:  4500 012b bdc8 0000 ff11 9e3e c0a8 739f  E..+.......>..s.
        0x0010:  c0a8 69ca d34c 0202 0117 6078 3c31 3734  ..i..L....`x<174
        0x0020:  3e20 3232 2f30 392f 3230 3231 3a31 353a  >.22/09/2021:15:
        0x0030:  3131 3a32 3920 474d 5420 4154 4c48 514d  11:29.GMT.ATLHQM
        0x0040:  564d 534c 4232 2030 2d50 5045 2d30 203a  VMSLB2.0-PPE-0.:
        0x0050:  2064 6566 6175 6c74 2054 4350 2043 4f4e  .default.TCP.CON

could someone point to me what I am missing?

Thanks a lot in advance.

[corti77 · Accepted solution]

Problem solved. The issue was located in the Splunk HEC configuration. the token I configured in syslog connector did not have access to the index netfw. 
I managed to find the issue after I found drop messages in the main index, default index in that HEC token.