Windows splunk forwarder not sending data

[perfecto25]

Hello, I have a win2008 fwd not sending data to a custom index

we have several indexers + 1 searchhead (all centos) and we were getting the data from this fwd up until a week ago, around the time the winbox was rebooted for updates. After that data stopped coming in.

I checked:

1. network - all ports are open and I can confirm data from this fwd is coming in to other indexes, just not this index
2. watched file is updated every few min, so new data is coming in

3. ran btool for syntax check, looks ok

   C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf [monitor://C:\Windows\System32\LogFiles]
   C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf _rcvbuf = 1572864
   C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf alwaysOpenFile = 1
   C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf baseline = 0
   C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf crcSalt =
   C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf disabled = false
   C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf evt_dc_name =
   C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf evt_dns_name =
   C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf evt_resolve_ad_obj = 0
   C:\Program Files\SplunkUniversalForwarder\etc\system\local\inputs.conf host = BICS03
   C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf ignoreOlderThan = 1h
   C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf index = sec-radius
   C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf interval = 60
   C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf recursive = false
   C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf sourcetype = RADIUS
   C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf whitelist = IN.+.log

the watched file is here, just a log looks like this

162.xxx..5.29,xxxxx,04/28/2017,14:20:09,IAS,COMDOM03,4,162.xxx..5.29,31,111.222.333.98,4108,162.xxx..5.29,4116,0,4128,CGY: DDD DOM03,4154,CGY: CGY DDD CRP,4155,1,4129,COMP\xxxxx,4130,COMP\xxxxx,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 8,4127,1,4136,1,4142,0
162.xxx..5.29,xxxxx,04/28/2017,14:20:09,IAS,COMDOM03,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 8,4127,1,4130,COMP\xxxxx,4108,162.xxx..5.29,4116,0,4128,CGY: DDD DOM03,4154,CGY: CGY DDD CRP,4155,1,4129,COMP\xxxxx,4136,3,4142,36
162.xxx..5.29,xxxxx,04/28/2017,14:20:31,IAS,COMDOM03,4,162.xxx..5.29,31,111.222.333.98,4108,162.xxx..5.29,4116,0,4128,CGY: DDD DOM03,4154,CGY: CGY DDD CRP,4155,1,4129,COMP\xxxxx,4130,COMP\xxxxx,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 9,4127,1,4136,1,4142,0
162.xxx..5.29,xxxxx,04/28/2017,14:20:31,IAS,COMDOM03,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 9,4127,1,4130,COMP\xxxxx,4108,162.xxx..5.29,4116,0,4128,CGY: DDD DOM03,4154,CGY: CGY DDD CRP,4155,1,4129,COMP\xxxxx,4136,3,4142,36
162.xxx..5.29,xxxxx,04/28/2017,14:20:36,IAS,COMDOM03,4,162.xxx..5.29,31,111.222.333.98,4108,162.xxx..5.29,4116,0,4128,CGY: DDD DOM03,4154,CGY: CGY DDD CRP,4155,1,4129,COMP\xxxxx,4130,COMP\xxxxx,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 10,4127,1,4136,1,4142,0
162.xxx..5.29,xxxxx,04/28/2017,14:20:36,IAS,COMDOM03,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 10,4127,1,4130,COMP\xxxxx,4108,162.xxx..5.29,4116,0,4128,CGY: DDD DOM03,4154,CGY: CGY DDD CRP,4155,1,4129,COMP\xxxxx,4136,3,4142,36
162.xxx..5.29,xxxxx,04/28/2017,14:21:24,IAS,COMDOM03,4,162.xxx..5.29,31,111.222.333.98,4108,162.xxx..5.29,4116,0,4128,CGY: DDD DOM03,4154,CGY: CGY DDD CRP,4155,1,4129,COMP\xxxxx,4127,1,4149,CGY: CGY VPN DUO NP,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 11,8136,1,8153,0,8111,0,4130,COMP.local/COM-Users/Employees/Calgary/Greg xxxxx,4136,1,4142,0
162.xxx..5.29,xxxxx,04/28/2017,14:21:24,IAS,COMDOM03,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 11,8153,0,8111,0,4130,COMP.local/COM-Users/Employees/Calgary/Greg xxxxx,4108,162.xxx..5.29,4116,0,4128,CGY: DDD DOM03,4154,CGY: CGY DDD CRP,4155,1,4129,COMP\xxxxx,4127,1,4149,CGY: CGY VPN DUO NP,8136,1,7,1,6,2,4294967210,50,4294967209,120,4136,2,4142,0
162.xxx..5.29,xxxxx,04/28/2017,14:21:28,IAS,COMDOM03,4,162.xxx..5.29,31,111.222.333.98,4108,162.xxx..5.29,4116,0,4128,CGY: DDD DOM03,4154,CGY: CGY DDD CRP,4155,1,4129,COMP\xxxxx,4130,COMP\xxxxx,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 12,4127,1,4136,1,4142,0
162.xxx..5.29,xxxxx,04/28/2017,14:21:28,IAS,COMDOM03,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 12,4127,1,4130,COMP\xxxxx,4108,162.xxx..5.29,4116,0,4128,CGY: DDD DOM03,4154,CGY: CGY DDD CRP,4155,1,4129,COMP\xxxxx,4136,3,4142,16
162.xxx..250.1,xxxxx,04/28/2017,14:21:33,IAS,COMDOM03,5,2101248,6,2,7,1,8,162.xxx..238.19,30,111.222.33312,31,111.222.333.98,40,1,41,0,44,DEE001A5,45,1,61,5,66,111.222.333.98,26,0x00000C04920A4461726B2D44756F,26,0x00000C04960600000002,26,0x00000C04970600000001,26,0x00000C04980600000003,4,162.xxx..250.1,4108,162.xxx..250.1,4116,0,4128,CGY: ASA CGY,4154,CGY: CGY VPN CR,4136,4,4142,0
162.xxx..250.1,xxxxx,04/28/2017,14:45:07,IAS,COMDOM03,5,2101248,6,2,7,1,8,162.xxx..238.19,30,111.222.33312,31,111.222.333.98,40,2,41,0,42,1612796,43,2380880,44,DEE001A5,45,1,46,1414,47,4545,48,4392,49,1,61,5,66,111.222.333.98,26,0x00000C04920A4461726B2D44756F,26,0x00000C04960600000002,26,0x00000C04970600000001,26,0x00000C04980600000003,4,162.xxx..250.1,4108,162.xxx..250.1,4116,0,4128,CGY: ASA CGY,4154,CGY: CGY VPN CR,4136,4,4142,0
162.xxx..5.29,xxxxx,04/28/2017,14:45:37,IAS,COMDOM03,4,162.xxx..5.29,31,111.222.333.98,4108,162.xxx..5.29,4116,0,4128,CGY: DDD DOM03,4154,CGY: CGY DDD CRP,4155,1,4129,COMP\xxxxx,4127,1,4149,CGY: CGY VPN DUO NP,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 13,8136,1,8153,0,8111,0,4130,COMP.local/COM-Users/Employees/Calgary/Greg xxxxx,4136,1,4142,0
162.xxx..5.29,xxxxx,04/28/2017,14:45:37,IAS,COMDOM03,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 13,8153,0,8111,0,4130,COMP.local/COM-Users/Employees/Calgary/Greg xxxxx,4108,162.xxx..5.29,4116,0,4128,CGY: DDD DOM03,4154,CGY: CGY DDD CRP,4155,1,4129,COMP\xxxxx,4127,1,4149,CGY: CGY VPN DUO NP,8136,1,7,1,6,2,4294967210,50,4294967209,120,4136,2,4142,0
162.xxx..250.1,xxxxx,04/28/2017,14:45:40,IAS,COMDOM03,5,2109440,30,111.222.33312,31,111.222.333.98,61,5,66,111.222.333.98,4,162.xxx..250.1,26,0x00000C04920A4461726B2D44756F,26,0x00000C04960600000002,4108,162.xxx..250.1,4116,0,4128,CGY: ASA CGY,5000,ip:source-ip=111.222.333.98,4154,CGY: CGY VPN CR,4155,1,4129,COMP\xxxxx,4130,COMP\xxxxx,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 14,4127,1,4136,1,4142,0
162.xxx..250.1,xxxxx,04/28/2017,14:45:40,IAS,COMDOM03,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 14,4127,1,4130,COMP\xxxxx,4129,COMP\xxxxx,4155,1,4154,CGY: CGY VPN CR,4108,162.xxx..250.1,4116,0,4128,CGY: ASA CGY,4136,3,4142,16
162.xxx..250.1,xxxxx,04/28/2017,14:45:44,IAS,COMDOM03,5,2109440,6,2,7,1,8,162.xxx..238.21,30,111.222.33312,31,111.222.333.98,40,1,41,0,44,DEE001A6,45,1,61,5,66,111.222.333.98,26,0x00000C04920A4461726B2D44756F,26,0x00000C04960600000002,26,0x00000C04970600000001,26,0x00000C04980600000003,4,162.xxx..250.1,4108,162.xxx..250.1,4116,0,4128,CGY: ASA CGY,4154,CGY: CGY VPN CR,4136,4,4142,0

also the index is present in the indexer,
[root@splunk01 /opt/splunk/bin]# cat /opt/splunk/etc/apps/search//local/indexes.conf

[sec-radius]
coldPath = $SPLUNK_DB/sec-radius/colddb
enableDataIntegrityControl = 0
enableTsidxReduction = 0
homePath = $SPLUNK_DB/sec-radius/db
maxTotalDataSizeMB = 512000
thawedPath = $SPLUNK_DB/sec-radius/thaweddb

Preview [Hide]
Hello, I have a win2008 fwd not sending data to a custom index

we have several indexers + 1 searchhead (all centos) and we were getting the data from this fwd up until a week ago, around the time the winbox was rebooted for updates. After that data stopped coming in.

I checked:

network - all ports are open and I can confirm data from this fwd is coming in to other indexes, just not this index

watched file is updated every few min, so new data is coming in

ran btool for syntax check, looks ok

C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf [monitor://C:\Windows\System32\LogFiles]
C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf _rcvbuf = 1572864
C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf alwaysOpenFile = 1
C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf baseline = 0
C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf crcSalt = C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf disabled = false C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf evt_dc_name = C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf evt_dns_name = C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf evt_resolve_ad_obj = 0 C:\Program Files\SplunkUniversalForwarder\etc\system\local\inputs.conf host = BICS03 C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf ignoreOlderThan = 1h C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf index = sec-radius C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf interval = 60 C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf recursive = false C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf sourcetype = RADIUS C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf whitelist = IN.+.log

the watched file is here, just a log looks like this

162.xxx..5.29,xxxxx,04/28/2017,14:20:09,IAS,COMDOM03,4,162.xxx..5.29,31,111.222.333.98,4108,162.xxx..5.29,4116,0,4128,CGY: DDD DOM03,4154,CGY: CGY DDD CRP,4155,1,4129,COMP\xxxxx,4130,COMP\xxxxx,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 8,4127,1,4136,1,4142,0
162.xxx..5.29,xxxxx,04/28/2017,14:20:09,IAS,COMDOM03,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 8,4127,1,4130,COMP\xxxxx,4108,162.xxx..5.29,4116,0,4128,CGY: DDD DOM03,4154,CGY: CGY DDD CRP,4155,1,4129,COMP\xxxxx,4136,3,4142,36
162.xxx..5.29,xxxxx,04/28/2017,14:20:31,IAS,COMDOM03,4,162.xxx..5.29,31,111.222.333.98,4108,162.xxx..5.29,4116,0,4128,CGY: DDD DOM03,4154,CGY: CGY DDD CRP,4155,1,4129,COMP\xxxxx,4130,COMP\xxxxx,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 9,4127,1,4136,1,4142,0
162.xxx..5.29,xxxxx,04/28/2017,14:20:31,IAS,COMDOM03,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 9,4127,1,4130,COMP\xxxxx,4108,162.xxx..5.29,4116,0,4128,CGY: DDD DOM03,4154,CGY: CGY DDD CRP,4155,1,4129,COMP\xxxxx,4136,3,4142,36
162.xxx..5.29,xxxxx,04/28/2017,14:20:36,IAS,COMDOM03,4,162.xxx..5.29,31,111.222.333.98,4108,162.xxx..5.29,4116,0,4128,CGY: DDD DOM03,4154,CGY: CGY DDD CRP,4155,1,4129,COMP\xxxxx,4130,COMP\xxxxx,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 10,4127,1,4136,1,4142,0
162.xxx..5.29,xxxxx,04/28/2017,14:20:36,IAS,COMDOM03,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 10,4127,1,4130,COMP\xxxxx,4108,162.xxx..5.29,4116,0,4128,CGY: DDD DOM03,4154,CGY: CGY DDD CRP,4155,1,4129,COMP\xxxxx,4136,3,4142,36
162.xxx..5.29,xxxxx,04/28/2017,14:21:24,IAS,COMDOM03,4,162.xxx..5.29,31,111.222.333.98,4108,162.xxx..5.29,4116,0,4128,CGY: DDD DOM03,4154,CGY: CGY DDD CRP,4155,1,4129,COMP\xxxxx,4127,1,4149,CGY: CGY VPN DUO NP,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 11,8136,1,8153,0,8111,0,4130,COMP.local/COM-Users/Employees/Calgary/Greg xxxxx,4136,1,4142,0
162.xxx..5.29,xxxxx,04/28/2017,14:21:24,IAS,COMDOM03,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 11,8153,0,8111,0,4130,COMP.local/COM-Users/Employees/Calgary/Greg xxxxx,4108,162.xxx..5.29,4116,0,4128,CGY: DDD DOM03,4154,CGY: CGY DDD CRP,4155,1,4129,COMP\xxxxx,4127,1,4149,CGY: CGY VPN DUO NP,8136,1,7,1,6,2,4294967210,50,4294967209,120,4136,2,4142,0
162.xxx..5.29,xxxxx,04/28/2017,14:21:28,IAS,COMDOM03,4,162.xxx..5.29,31,111.222.333.98,4108,162.xxx..5.29,4116,0,4128,CGY: DDD DOM03,4154,CGY: CGY DDD CRP,4155,1,4129,COMP\xxxxx,4130,COMP\xxxxx,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 12,4127,1,4136,1,4142,0
162.xxx..5.29,xxxxx,04/28/2017,14:21:28,IAS,COMDOM03,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 12,4127,1,4130,COMP\xxxxx,4108,162.xxx..5.29,4116,0,4128,CGY: DDD DOM03,4154,CGY: CGY DDD CRP,4155,1,4129,COMP\xxxxx,4136,3,4142,16
162.xxx..250.1,xxxxx,04/28/2017,14:21:33,IAS,COMDOM03,5,2101248,6,2,7,1,8,162.xxx..238.19,30,111.222.33312,31,111.222.333.98,40,1,41,0,44,DEE001A5,45,1,61,5,66,111.222.333.98,26,0x00000C04920A4461726B2D44756F,26,0x00000C04960600000002,26,0x00000C04970600000001,26,0x00000C04980600000003,4,162.xxx..250.1,4108,162.xxx..250.1,4116,0,4128,CGY: ASA CGY,4154,CGY: CGY VPN CR,4136,4,4142,0
162.xxx..250.1,xxxxx,04/28/2017,14:45:07,IAS,COMDOM03,5,2101248,6,2,7,1,8,162.xxx..238.19,30,111.222.33312,31,111.222.333.98,40,2,41,0,42,1612796,43,2380880,44,DEE001A5,45,1,46,1414,47,4545,48,4392,49,1,61,5,66,111.222.333.98,26,0x00000C04920A4461726B2D44756F,26,0x00000C04960600000002,26,0x00000C04970600000001,26,0x00000C04980600000003,4,162.xxx..250.1,4108,162.xxx..250.1,4116,0,4128,CGY: ASA CGY,4154,CGY: CGY VPN CR,4136,4,4142,0
162.xxx..5.29,xxxxx,04/28/2017,14:45:37,IAS,COMDOM03,4,162.xxx..5.29,31,111.222.333.98,4108,162.xxx..5.29,4116,0,4128,CGY: DDD DOM03,4154,CGY: CGY DDD CRP,4155,1,4129,COMP\xxxxx,4127,1,4149,CGY: CGY VPN DUO NP,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 13,8136,1,8153,0,8111,0,4130,COMP.local/COM-Users/Employees/Calgary/Greg xxxxx,4136,1,4142,0
162.xxx..5.29,xxxxx,04/28/2017,14:45:37,IAS,COMDOM03,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 13,8153,0,8111,0,4130,COMP.local/COM-Users/Employees/Calgary/Greg xxxxx,4108,162.xxx..5.29,4116,0,4128,CGY: DDD DOM03,4154,CGY: CGY DDD CRP,4155,1,4129,COMP\xxxxx,4127,1,4149,CGY: CGY VPN DUO NP,8136,1,7,1,6,2,4294967210,50,4294967209,120,4136,2,4142,0
162.xxx..250.1,xxxxx,04/28/2017,14:45:40,IAS,COMDOM03,5,2109440,30,111.222.33312,31,111.222.333.98,61,5,66,111.222.333.98,4,162.xxx..250.1,26,0x00000C04920A4461726B2D44756F,26,0x00000C04960600000002,4108,162.xxx..250.1,4116,0,4128,CGY: ASA CGY,5000,ip:source-ip=111.222.333.98,4154,CGY: CGY VPN CR,4155,1,4129,COMP\xxxxx,4130,COMP\xxxxx,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 14,4127,1,4136,1,4142,0
162.xxx..250.1,xxxxx,04/28/2017,14:45:40,IAS,COMDOM03,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 14,4127,1,4130,COMP\xxxxx,4129,COMP\xxxxx,4155,1,4154,CGY: CGY VPN CR,4108,162.xxx..250.1,4116,0,4128,CGY: ASA CGY,4136,3,4142,16
162.xxx..250.1,xxxxx,04/28/2017,14:45:44,IAS,COMDOM03,5,2109440,6,2,7,1,8,162.xxx..238.21,30,111.222.33312,31,111.222.333.98,40,1,41,0,44,DEE001A6,45,1,61,5,66,111.222.333.98,26,0x00000C04920A4461726B2D44756F,26,0x00000C04960600000002,26,0x00000C04970600000001,26,0x00000C04980600000003,4,162.xxx..250.1,4108,162.xxx..250.1,4116,0,4128,CGY: ASA CGY,4154,CGY: CGY VPN CR,4136,4,4142,0

also the index is present in the indexer,
[root@splunk01 /opt/splunk/bin]# cat /opt/splunk/etc/apps/search//local/indexes.conf

[sec-radius]
coldPath = $SPLUNK_DB/sec-radius/colddb
enableDataIntegrityControl = 0
enableTsidxReduction = 0
homePath = $SPLUNK_DB/sec-radius/db
maxTotalDataSizeMB = 512000
thawedPath = $SPLUNK_DB/sec-radius/thaweddb

[perfecto25 · Accepted solution]

turns out it was our global conf setting in inputs.conf on forwarder
we had

ignoreOlderThan = 1h

once commented out, i started seeing events come into the indexer.

[aaraneta_splunk]

@perfecto25 - Glad you were able to find the solution to your question. Please don't forget to resolve your post by clicking "Accept" below your answer. Thanks!

[rickrowe]

What version of the forwarder?
If no errors in win event log, reload or update with the latest forwarder on the server again and verify the service is starting automatic.

[ddrillic]

Does it phone home?

[adonio]

hello perfecto,
do you see any errors or warnings about this forwarder in splunkd?
search; index =_intrenal host=YourForwarder log_level=WARN OR log_level = ERROR
also, in the path on your cat command it seems like there are two (2) // after "search"
[root@splunk01 /opt/splunk/bin]# cat /opt/splunk/etc/apps/search//local/indexes.conf
if that is the case, splunk cant recognize that local folder. Can you validate this is exact path?