All Apps and Add-ons

Why am I getting error "bad lexical cast: source type value could not be interpreted as target" when the Splunk App for Stream is starting?

johnsjm
Explorer

I'm troubleshooting why Splunk Stream isn't forwarding events anymore after I updated to 6.5.1 in a simple distributed config. Search head, indexer, universal forwarder. When Stream starts I get these messages and this fatal error. (Turned on TRACE logging level)

2016-06-28 15:50:53 DEBUG [140295494879040] (StreamSender.cpp:397) stream.SplunkSenderModularInput - Updating streams
2016-06-28 15:50:53 DEBUG [140295494879040] (StreamSender.cpp:630) stream.SplunkSenderModularInput - Sending request for /en-us/custom/splunk_app_stream/captureipaddresses/
2016-06-28 15:50:53 DEBUG [140295494879040] (StreamSender.cpp:302) stream.SplunkSenderModularInput - Starting stream sender
2016-06-28 15:50:53 FATAL [140295494879040] (CaptureServer.cpp:1721) stream.CaptureServer - bad lexical cast: source type value could not be interpreted as target (C++ error as likely a variable is null)

Curling the url from the log:

curl -k https://SEARCH_HEAD:8000/en-us/custom/splunk_app_stream/captureipaddresses/

[{"id": "whitelist", "ipAddresses": []}, {"id": "blacklist", "ipAddresses": []}]

I was troubleshooting all morning and ended up blowing everything away and starting over and now have the same result. Are the captureipaddresses needed?

Here's the inputs.conf on our universal forwarder below. I have tried many different configurations in streamfwd.conf, but now are at the most basic.

[streamfwd://streamfwd]
splunk_stream_app_location=https://SEARCH_HEAD:8000/en-us/custom/splunk_app_stream/
stream_forwarder_id=streambox1
disabled=0
index=stream
sourcetype=stream

[streamfwd]
port = 8889
ipAddr = 127.0.0.1
processingThreads = 2
#maxTcpSessionCount = 100000
#maxTcpReassemblyPacketCount = 600000
streamfwdcapture.0.interface = eth2
streamfwdcapture.0.filter = not port 443 and not port 8443
# usePacketMemoryPool = true

I'm seeing streambox1 show up on the Stream app under "distributed forwarder management".

The indexer has the Splunk_TA_stream app and has a wire input that seems to be changing state from enabled to disabled. Events started coming in when I enabled the input, but then later stopped after the next restart while attempting to increase tcp sessions and max tcp reassembly count. Now I can't get events to come in again.

Did I forget something? It's been quite a while since I first configured it.

Any help is appreciated.

Thanks,
John

0 Karma

vshcherbakov_sp
Splunk Employee
Splunk Employee

Hi John,

I believe your .conf files are OK. Do you have any non-trivial stream configuration, like custom streams/filters, etc? It seems like there may be a Stream filter that's configured incorrectly (unfortunately the app backend doesn't do a good job validating filter argument types)

0 Karma

johnsjm
Explorer

Right now it's a fresh Stream install and it's all "out-of-the-box" configurations aside from the configurations shown above. It's not a fresh Splunk install, but it's Splunk Enterprise 6.4.0. I did an ssldump and discovered that the json response from the "captureipaddresses" may have issues, here is the payload from the response, there are 256 bytes of leading spaces and a newline before the json response. I noticed this as well for the /en-us/custom/splunk_app_stream/streams/?streamForwarderId=streambox1 request, is that normal?

0000   48 54 54 50 2f 31 2e 31 20 32 30 30 20 4f 4b 0d  HTTP/1.1 200 OK.
0010   0a 44 61 74 65 3a 20 57 65 64 2c 20 32 39 20 4a  .Date: Wed, 29 J
0020   75 6e 20 32 30 31 36 20 31 39 3a 33 30 3a 31 36  un 2016 19:30:16
0030   20 47 4d 54 0d 0a 43 6f 6e 74 65 6e 74 2d 54 79   GMT..Content-Ty
0040   70 65 3a 20 74 65 78 74 2f 6a 73 6f 6e 3b 63 68  pe: text/json;ch
0050   61 72 73 65 74 3d 75 74 66 2d 38 0d 0a 58 2d 43  arset=utf-8..X-C
0060   6f 6e 74 65 6e 74 2d 54 79 70 65 2d 4f 70 74 69  ontent-Type-Opti
0070   6f 6e 73 3a 20 6e 6f 73 6e 69 66 66 0d 0a 43 6f  ons: nosniff..Co
0080   6e 74 65 6e 74 2d 4c 65 6e 67 74 68 3a 20 33 33  ntent-Length: 33
0090   37 0d 0a 56 61 72 79 3a 20 43 6f 6f 6b 69 65 0d  7..Vary: Cookie.
00a0   0a 43 6f 6e 6e 65 63 74 69 6f 6e 3a 20 43 6c 6f  .Connection: Clo
00b0   73 65 0d 0a 58 2d 46 72 61 6d 65 2d 4f 70 74 69  se..X-Frame-Opti
00c0   6f 6e 73 3a 20 53 41 4d 45 4f 52 49 47 49 4e 0d  ons: SAMEORIGIN.
00d0   0a 53 65 74 2d 43 6f 6f 6b 69 65 3a 20 73 65 73  .Set-Cookie: ses
00e0   73 69 6f 6e 5f 69 64 5f 38 30 30 30 3d 66 33 31  sion_id_8000=f31
00f0   64 64 30 37 39 33 30 30 36 33 66 38 37 34 62 66  dd07930063f874bf
0100   62 37 36 39 66 33 36 36 35 62 66 31 34 62 34 38  b769f3665bf14b48
0110   31 30 39 39 62 3b 20 65 78 70 69 72 65 73 3d 54  1099b; expires=T
0120   68 75 2c 20 33 30 20 4a 75 6e 20 32 30 31 36 20  hu, 30 Jun 2016 
0130   31 39 3a 33 30 3a 31 36 20 47 4d 54 3b 20 68 74  19:30:16 GMT; ht
0140   74 70 6f 6e 6c 79 3b 20 50 61 74 68 3d 2f 3b 20  tponly; Path=/; 
0150   73 65 63 75 72 65 0d 0a 53 65 72 76 65 72 3a 20  secure..Server: 
0160   53 70 6c 75 6e 6b 64 0d 0a 0d 0a 20 20 20 20 20  Splunkd....     
0170   20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20                  
0180   20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20                  
0190   20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20                  
01a0   20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20                  
01b0   20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20                  
01c0   20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20                  
01d0   20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20                  
01e0   20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20                  
01f0   20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20                  
0200   20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20                  
0210   20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20                  
0220   20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20                  
0230   20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20                  
0240   20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20                  
0250   20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20                  
0260   20 20 20 20 20 20 20 20 20 20 20 0a 5b 7b 22 69             .[{"i
0270   70 41 64 64 72 65 73 73 65 73 22 3a 20 5b 5d 2c  pAddresses": [],
0280   20 22 69 64 22 3a 20 22 77 68 69 74 65 6c 69 73   "id": "whitelis
0290   74 22 7d 2c 20 7b 22 69 70 41 64 64 72 65 73 73  t"}, {"ipAddress
02a0   65 73 22 3a 20 5b 5d 2c 20 22 69 64 22 3a 20 22  es": [], "id": "
02b0   62 6c 61 63 6b 6c 69 73 74 22 7d 5d              blacklist"}]                                  ]
0 Karma

vshcherbakov_sp
Splunk Employee
Splunk Employee

I believe the excessive spaces (0x20) are added to prevent some Internet Explorer vulnerability; it should be harmless

0 Karma

johnsjm
Explorer

I commented out processingThreads so now it's back to the default of 1 and it's working. My server has 4 cores. I'm getting a lot of "TCP reassembly queue overflow" messages. Another thing I'm not sure I should be seeing, I have the BPF set to exclude 443 and 8443 but I'm still getting SSL messages, should that be happening?

(SnifferReactor/PacketProcessor.cpp:732) stream.SnifferReactor - SSL decryption error: unexpected client handshake message before ServerHello (ssl) [c=192.168.X.X:XXXXX, s=192.168.Y.Y:443]

0 Karma

johnsjm
Explorer

Today during peak time we started hitting "max packet queue size exceeded" messages, I saw recommendations to increase the processingThreads which wasn't working for me earlier this week. For fun I tried "processingThreads = 4" and encountered the same lexical error, then I tried "ProcessingThreads = 4" (capital P) and it worked. Is there some config case issues persisting from the switch from XML to .conf?

Thanks,
John

0 Karma

vshcherbakov_sp
Splunk Employee
Splunk Employee

re: processingThreads: I just tried processingThreads = 2 and it worked ok.. any chance you might have had some non-printable characters trailing 2 in the config file?

re: BPF: what does curl -k http://localhost:8889/config/sniffer return on your streambox1 machine?

0 Karma

johnsjm
Explorer
<?xml version="1.0" encoding="UTF-8"?>
<CmConfig xmlns="http://purl.org/cloudmeter/config" version="6.5.1">
<Reactor><Workspace>Capture</Workspace><Plugin>SnifferReactor</Plugin><QueueEventDelivery>false</QueueEventDelivery><HideCreditCardNumbers>true</HideCreditCardNumbers><GenerateFlowEvents>true</GenerateFlowEvents><MaxPacketQueueSize>500000</MaxPacketQueueSize><MaxTcpSessionCount>100000</MaxTcpSessionCount><MaxTcpReassemblyPacketCount>2000000</MaxTcpReassemblyPacketCount><AnalyzeRawSSL>true</AnalyzeRawSSL><Capture><Filter>(not port 443 and not port 8443) and not (host 10.X.X.X and port 8000)</Filter><Interface>eth2</Interface></Capture></Reactor></CmConfig>
0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...