Why is dest_content in http stream data getting truncated after updating Splunk App for Stream from 6.1.0 to 6.2.2?

[heath]

We updated stream from 6.1.0 to 6.2.2. Since then the http response data in dest_content has been getting truncated at random locations. The maximum size of dest_content went from about 95K before the update to about 14K now. Is there some kind of new limit or some setting I need to configure?

| eval dest_content_size=len(dest_content) | stats avg(dest_content_size) as avg_size max(dest_content_size) as max_size

stream 6.1.0:

avg_size        max_size
13762.567416    95007

stream 6.2.2:

avg_size        max_size
4971.677987  14235

[csharp_splunk · Accepted solution]

In 6.2 we added a cap for field size to limit the risk of unbounded memory growth. This field is MaxFieldSize, but unfortunately it didn't make it the docs.

http://docs.splunk.com/Documentation/StreamApp/6.2.2/DeployStreamApp/ConfigureStreamForwarder#Config...

That docs page covers how to configure streamfwd.xml, and if you add a MaxFieldSize parameter you can up the limit to something larger if you're concerned about RAM growth or if not you can set the value to 0 for unlimited.