Getting Data In

Problems with a dynamic input header in a scripted input

Lowell
Super Champion

I have a number of custom scripted inputs that use the dynamic input header (***SPLUNK*** key=val) to establish various host/source/sourcetype values appropriate for the content I'm loading into splunk. I'm having problems with one of my scripts that uses the dynamic input header repeatedly (since it loads a number of different types of content during each execution of the scripted input). Normally everything works fine, but every once it a while (maybe a couple times a month) the input stream seems to get garbled and the metadata doesn't get assigned properly. Sometimes I can even see ***SPLUNK*** in the middle of indexed events.

I thought that maybe this had to do with flushing the stdout stream, so I've added a number of flush() calls, but it just occurred to me that the events that get corrupted are actually coming from two different input scripts (well, technically, I have just one input script setup twice with different arguments.)

So, now I'm wondering is my issue is could be related to the question: Why do my scripted inputs data get intermingled?, but that question doesn't really talk about the impact on dynamic input headers. Also, sometimes I get partial event that are assigned "sourcetype=exec", and "source=script" (Are those the default values?) The workaround given on the above answer is to simply reassign the source of the scripted input, but how does that work when you are reassigning the source dynamically using the ***SPLUNK*** source=blah. I don't fully get how the dynamic input header works (even though I use it all over the place).


Example event

Here is one of my events that contains the output from two different script runs. The larger event is a stats capture from a MS SQL database, and the event in the middle is just a work queue length that we want to capture periodically.

2010-09-17 14:40:04.895156
type=SQLServer:AccessMethods
server=mssql2000.example.com
FullScansPerSec=294345071
RangeScansPerSec=73737045
ProbeScansPerSec=66468066
ScanPointRevalidationsPerSec=270592972
WorkfilesCreatedPerSec=47953285
WorktablesCreatedPerSec=5196816
WorktablesFromCacheRatio=594427
WorktablesFromCacheBase=1048346
ForwardedRecordsPerSec=-873022806
SkippedGhostedRecordsPerSec=822412
IndexSearchesPerSec=-1772663315
FreeSpaceScansPerSec=-1852602000
FreeSpacePageFetchesPerSec=-1848258364
PagesAllocatedPerSec=32791380
ExtentsAllocatedPerSec=18185647
M***SPLUNK*** host=mssql2000.example.com source=provider_settlementqueue sourcetype=provider_stats
2010-09-17 14:40:04.603000 mssql2000.example.com ProviderSettlementQueue:  count=1 retries=1
ixedpageallocationsPerSec=1793820
ExtentDeallocationsPerSec=27726820
PageDeallocationsPerSec=0
PageSplitsPerSec=3979156
TableLockEscalationsPerSec=286514

Notice that the ***SPLUNK*** message ("provider_stats") occurs in the middle of larger event. Had there not been an "M" right before the header, I suppose the last 5 lines of the message could have been assigned to the "provider_stats" sourcetype as well, instead of the correct "mssql_perfstats" sourcetype.

My inputs.conf looks like this:

[script://.//bin/query_performance.sh --settlement-queue --block-info]
disabled = false
index = osperf
interval = 30

[script://./bin/query_performance.sh --cache-info --perf-info]
disabled = false
index = osperf
interval = 300
0 Karma
1 Solution

Lowell
Super Champion

Yes, the issue was caused by multiple inputs all using the same "source" name, and simply overriding the "source" was all it takes to prevent these inputs from getting intermingled:

My working inputs.conf file:

[script://.//bin/query_performance.sh --settlement-queue --block-info]
disabled = false
index = osperf
interval = 30
source = query_performance-1

[script://./bin/query_performance.sh --cache-info --perf-info]
disabled = false
index = osperf
interval = 300
source = query_performance-2

View solution in original post

0 Karma

Lowell
Super Champion

Yes, the issue was caused by multiple inputs all using the same "source" name, and simply overriding the "source" was all it takes to prevent these inputs from getting intermingled:

My working inputs.conf file:

[script://.//bin/query_performance.sh --settlement-queue --block-info]
disabled = false
index = osperf
interval = 30
source = query_performance-1

[script://./bin/query_performance.sh --cache-info --perf-info]
disabled = false
index = osperf
interval = 300
source = query_performance-2
0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...