All Apps and Add-ons

Why is the Splunk App for Stream putting all data into _internal, and can I change the target index?

New Member

It looks like the Splunk App for Stream puts everything into _internal. In general, we don't like to do that. In this case, do we really need to? It seems like too much volume to put in _internal.

My plan is to capture summary data on most network connections for hundreds of product servers. I can't find anything about index choices in the docs. Does anyone have any recommendations on which index(es) to use? This data will be part of our security app we're building, and access will be restricted as well.

I have some data on a test server, just with the tcp and udp streams enabled as they come by default. The only messages I see in the events are: "decodePacket: skip bogus packet with length less than IP header length". Any idea what that means?

thank you!


Tags (2)
0 Karma


Hello yes you can.

I use this technique to put my streams in different indexes and manage the permissions.

You need to declare the index on your Splunk stream app indexes.conf. to do that follow this steps:

1) On your stream server : SPLUNK_HOME/etc/apps/splunk_app_stream/local
2) edit or create the indexes.conf file
3) declare your indexes (already created on your indexers) like this
coldpath = $SPLUNK_DB/yourindex/colddb
enableDataIntegrityControl = 0
enableTsdixReduction = 0
homePath = $SPLUNK_DB/yourindex/db
MaxTotalDataSizeMB = 512000 #(in my case)
thawedPath = $SPLUNK_DB/yourindex/thaweddb

4) restart your splunkd service
5) now when you configure your streams in the stream app you can see your indexes in the dropdown menu

Hope it helps

0 Karma

Splunk Employee
Splunk Employee

This is due to tcp segmentation offload which generates a 0 length IP Header so the stream forwarder skips the data.

A Fix is due out soon, try disabling tcp offload


Splunk Employee
Splunk Employee

Stream only puts internal logs and statistics into the "_internal" index (this doesn't count towards your license volume). By default, it will put all events derived from network traffic into the "main" index. You can change the index used for each stream within the UI (at the top of the page, after you click on a particular stream in the list), or the default index for all streams by setting the following parameter in the streamfwd section of your inputs.conf file:


Note that the priority for index selection is:

  • The config for a specific stream
  • index defined in inputs.conf
  • main
0 Karma

New Member

Thank you both, that helps explain it. Does anyone know what to check why we can't see any data? All we see are the error messages to _internal about "decodePacket: skip bogus packet with length less than IP header length".

Clearly something is not working. This is win2012r2 hosted in Azure, so has a hyperv network card. splunkd is running as localsystem.

Thank you for any insight or steps to try.

0 Karma

Splunk Employee
Splunk Employee

By default only Stream internal information goes to _internal.

The real event data (by default) goes to the main/default index. You can also configure on a per Stream basis and destination index if you like. And finally, if you want to set a global destination, you can modify Splunk_TA_stream/local/inputs.conf and add an "index = foo" attribute.


Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...