Why does Stats command ocassionally produces job search components in results?

[splunkkitty]

I have the following stats search:

index=servers1 OR index=servers2 DBNAME=DATABASENAME source="/my/log/source/*" | stats list(Tablespace), list("Total MB"), list("Used MB"), list("Free MB"), list("Percent Free") by DBNAME

Which is supposed to produce the following results:

DBNAME	list(Tablespace)	list(Total MB)	list(Used MB)	list(Free MB)	list(Percent Free)
DATABASENAME	RED_DATA	165890	46350	119540	72
	BLUE_DATA	2116	1016	1100	52
	PINK_DATA	10	0	10	100
	PURPLE_DATA	34	17	17	50
	GREEN_DATA	6717	0	6717	100
	ORANGE_DATA	51323	295	51028	99
…. Cont'd for 20+ rows

About 25-30% of the time (as both a live search and scheduled report) I get the following results instead. The list(Used MB) column will have extra garbage data which appears to be job search components:

DBNAME	list(Tablespace)	list(Total MB)	list(Used MB)	list(Free MB)	list(Percent Free)
DATABASENAME	RED_DATA	165890	46350	119540	72
	BLUE_DATA	2116	1016	1100	52
	PINK_DATA	10	0	10	100
	PURPLE_DATA	34	17	17	50
	GREEN_DATA	6717	0	6717	100
	ORANGE_DATA	51323	295	51028	99
			ion.command.search
			4
			duration.command.search.calcfields
			0
			duration.command.search.fieldalias
			0
			duration.command.search.filter
			0
			duration.command.search.index
			2
			duration.command.search.index.usec_1_8
			0
			duration.command.search.index.usec_8_64
			0
			duration.command.search.kv
			0
			duration.command.search.lookups
…. Cont'd for 20+ rows

I've adjusted the following limits.conf Search Head settings with no luck:

[stats]
maxresultrows = 50000
maxvalues = 10000
maxvaluesize = 10000
list_maxsize = 10000

The search inspector also does not produce any notable error messages.

Any ideas as to what is happening here and how I can solve it?

[richgalloway]

The erroneous results imply an error extracting the "Used MB" field.  To help fix it, we'll need to see sample data and the props.conf settings for the sourcetype.

[splunkkitty]

The pretrained CSV sourcetype is used. No customized props is referenced in the input.

Inputs.conf:

[monitor:///my/log/source/]
index = my_index
disabled = 0
crcSalt = <SOURCE>

CSV source example:

"Tablespace","Used MB","Free MB","Total MB","Percent Free","DBNAME"
"RED_DATA",1974,6,1980,0,"DATABASENAME"
"BLUE_DATA",217532,605514,823046,74,"DATABASENAME"
"PINK_DATA",0,10,10,100,"DATABASENAME"
"PURPLE_DATA",0,100,100,100,"DATABASENAME"
"GREEN_DATA",23682,1258,24940,5,"DATABASENAME"
"ORANGE_DATA",241,996,1237,81,"DATABASENAME"

[richgalloway]

You say the csv sourcetype is used, but the inputs.conf does not specify any sourcetype.  Let's start by adding sourcetype = csv to the inputs.conf stanza.

[splunkkitty]

Updated and deployed inputs.conf with csv sourcetype (and additionally allow list for CSVs only):

[monitor:///my/log/source/]
index = my_index
whitelist = (\.csv$)
sourcetype = csv
recursive = false
disabled = 0
crcSalt = <SOURCE>

As of this time nothing has changed with the execution of the search (job search components still appear under the "Used MB" field with a small percentage of searches).

Thank you for your input!

[richgalloway]

Thanks for trying that.  Have you looked at the raw data?  It sure seems like there are some non-numerical values in the "Used MB" field that are throwing off the results.

[splunkkitty]

I had a chance to sit down with the owner of the hosts. Nothing looks amiss with any of the CSV files (we have this same file ingesting from multiple hosts). Formatting is good.

When we were originally set things up the "Percentage Free" column was named "% Free" which was causing a problem. We had to update it so the files would ingest.

[brijebi]

I am running into the same problem.   I am also using stats list() and getting job inspector data in the list column.  My source type is syslog.

It only happens when I use a search head to search another indexer.  When I search on the indexer itself, the results are fine.

.

[splunkkitty]

I ended up reaching out to Splunk support on the issue. Per their reply there were bugs in 7.3.8 and some versions of 8.2.* that were similar. Our deployment is running on 8.2.2.1.

Unfortunately after some lab testing and back and forth we were unable to isolate and resolve it. In the meantime I've been using the table command with this particular dataset to get around the garbage output.

They recommended upgrading our Splunk deployment to the most recent release and then checking to see if it is still happening. We haven't gotten this far yet.