src_content/dest_content fields are available only for HTTP and TCP/UDP protocols and not enabled by default - you'll need to go to the Streams Config page and enable them. Also, there's a default field size limit of 10K that you may want to change by setting the MaxFieldSize parameter (see http://docs.splunk.com/Documentation/StreamApp/6.3.2/DeployStreamApp/ConfigureStreamForwarder#Advanced_configuration for more details) ... View more

Hello, The current version of Stream doesn't reassemble the files sent via FTP/SMTP/etc. Stream does support extracting data from HTTP payload via regular expressions, so it may be feasible to create (clone) a HTTP stream that filters events based on http_content_type field (to, say, start with application/) and cuts off response headers from the dest_content field using content extraction regex (see http://docs.splunk.com/Documentation/StreamApp/6.3.2/DeployStreamApp/ConfigureStreams#Use_Content_Extraction) ... View more

Hello, Stream supports the generic src_content/dest_content fields that represent the "payload" data for certain protocols such as HTTP or TCP. You can also extract specific parts of these fields (or any other textual fields for that matter) with a regular expression using so called "content extraction" feature of Stream. Here's the documentation link for more details: http://docs.splunk.com/Documentation/StreamApp/6.3.2/DeployStreamApp/ConfigureStreams#Use_Content_Extraction ... View more

Processing threads count should be set based on traffic load; there's no benefit of having 32 threads (not uncommon for a number of CPU cores these days) for processing ~1Gbps; in fact, there's some (not significant) memory overhead caused by running multiple processing threads. I believe for most use cases there's no need to set more than 8 processing threads. ... View more

Hi Mathias, Which flow fields do we really need to have the "query" and "answer" field Not sure what you mean by flow fields? The only SPL magic that you may need to do with regards to "stream:dns" events is to normalize the name, host_addr, host_type (and maybe other) fields since they may be single value as well as MV field. This is a known pain with Stream and we're working on addressing it. What do you see if you run a SPL query such as this: sourcetype="stream:dns" dest_port=53 query_type="A" | table query, reply_code, name{}, host_addr{}, host_type{} sourcetype="stream:dns" dest_port=53 query_type="A" | table query, reply_code, name, host_addr, host_type Re: estimating the data volume needs. You can turn the dns stream (or streams) into "stats-only" mode to make streamfwd do pretty much everything it would do when that stream (dns, or any other one for that matter) is enabled - capture and process the data and build an event in memory - except actually sending the data to Splunk indexer. Instead, it sends the aggregated statistics about the size of the events you can see on the App for Stream's "Stream Stats" dashboard. So it's completely free and doesn't affect your license data volume at all. You'll need the setup the _internal index forwarding on your UF layer running Splunk_TA_stream though as the stats events are sent to _internal. ... View more

Hi Mathias, Thanks for the clarification. I agree that Stream implements most of the requirements you outlined, the only missing part is that the multi-value data is "jammed"/poorly structured (and we're working on it..) thanks again, Vladimir ... View more

hi mathiask, Great to hear you find Stream useful, and thanks for your feedback! Could you please provide more details on what you're trying to accomplish so that we can try* devising a workaround? thanks vladimir *backstory: Stream currently has a somewhat limited event model (which for the most part is basically a flat list of key:value pairs). I am guessing this may be the root cause of the issues you're experiencing since the current model works reasonably well for many protocols, but has problems representing more complex structures like the DNS responses you've mentioned. I can tell as much as "we're aware of this limitation and working on addressing it" ... View more

Stream TA need a inputs.conf stanza in order to work; more details can be found in Stream docs at http://docs.splunk.com/Documentation/StreamApp/6.3.2/DeployStreamApp/ConfigureStreamForwarder#Verify_streamfwd_can_communicate_with_splunk_app_stream ... View more

Hello, Have you checked the content of your Splunk_TA_stream/local/inputs.conf file? ... View more

Hi David, No, Splunk_TA_stream requires a live connection to the App For Stream. App for Stream doesn't have to reside on the SH though - you can install it on any splunkweb instance - standalone, etc. HTH, Vladimir ... View more

bwheelock, what do you mean by "events" here? TCP connections? What streams are enabled? By default, Stream captures only TCP and UDP flow events, i.e. it generates a single event for a TCP connection or UDP flow. If you want to capture specific protocol (HTTP, DNS, etc.) events, you will need to enable/configure the corresponding streams in the App for Stream config UI. ... View more

Yes, I'd start with checking packets_in and packets_out fields. There are also data_packets_in and data_packets_out fields indicating the number of TCP payload packets. I'd also suggest upgrading App for Stream to v 6.3 as it contains improvements in the flow classification logic. ... View more

Do you mean the src_content field is not present for flows that could not be classified (app is "unknown")? If so, it's probably because Stream didn't capture any payload packets since the src_content data is captured independently from flow classification. I'd suggest checking the packet count fields to see if these flows have anything substantial. Enabling the dest_content field may also be of value. ... View more

Hi satorn_ja, Did you check if the Wire Data Input is enabled/working on your server? When you install splunk_app_stream, it drops Splunk_TA_stream on your server, but doesn't enable it by default. You can check status/enable it if you to Settings->Data Inputs -> Wire Data. ... View more

... see http://docs.splunk.com/Documentation/StreamApp/6.2.2/DeployStreamApp/Deploymentrequirements#Splunk_component_requirements for more details ... View more

Hello, The data used to populate Stream dashboards is sent to the _internal index that is not being forwarded by default. To enable forwarding the _internal index, please modify your universal forwarder's $SPLUNK_HOME/etc/system/local/outputs.conf file to include the following: [tcpout] forwardedindex.2.whitelist = (_audit|_introspection|_internal) ... View more

Hi, At this moment, Splunk App for Stream capture layer (Splunk_TA_Stream) is only supported on x86/x86_64 architecture. ... View more

BTW, were you able to resolve the FATAL error problem in streamfwd.log? Not sure what the actual issue is since the error message you posted seems to be truncated.. If it's still an issue, please post the full error message to give us a better idea of what's going on. ... View more

You can add index = your_index to the inputs.conf stanza to overwrite the default index for Stream Forwarder events. The index field in the Stream configuration UI (on the search head) overrides the default or inputs.conf value on a per-stream base. ... View more