Stash Files contain binary data

[sdwilkerson]

Folks,

I'm at a site with 3 search heads and 6 indexers. One of the SH is ES-2.0.2.

All SH and Indexers were upgraded to Splunk-5.0.1 last week.

All SH are configured to spray/output summaries back to Indexers.

All three SH have unprocessed stash files in $SPLUNK_HOME/var/spool/splunk. These unprocessed files started the day the customer upgraded from 4.3.3 to 4.3.4 back on Oct. 24th and there are some-to-many a day since then.

index=_internal shows logs like this:

12-17-2012 15:57:37.416 -0500 INFO  TailingProcessor - Ignoring file /opt/splunk/var/spool/splunk/Network
- Proxy Events by Content Type - Summary Gen_829222797.stash_new' due to: binary 
12-17-2012 15:57:37.416 -0500 WARN  FileClassifierManager - The file /opt/splunk/var/spool/splunk/Network - Proxy Events by Content Type - Summary Gen_829222797.stash_new' is invalid. Reason: binary

When inspecting these unprocessed stash files, they all seem to contain blocks of binary data. Unprocessed stash files representing ES data are the biggest culprit but I see some from other apps too including DeploymentMonitor.

Using vi/more/shell I can locate a single line that is binary data. Using sed to extract that line and hexdump, I see this:

00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00009540  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 61  |...............a|
00009550  6d 65 3d 22 4e 65 74 77  6f 72 6b 20 2d 20 50 72  |me="Network - Pr|
00009560  6f 78 79 20 45 76 65 6e  74 73 20 62 79 20 55 73  |oxy Events by Us|
00009570  65 72 20 41 67 65 6e 74  20 2d 20 53 75 6d 6d 61  |er Agent - Summa|
00009580  72 79 20 47 65 6e 22 2c  20 73 65 61 72 63 68 5f  |ry Gen", search_|
00009590  6e 6f 77 3d 31 33 35 34  32 38 39 31 39 38 2e 30  |now=1354289198.0|
000095a0  30 30 2c 20 69 6e 66 6f  5f 73 65 61 72 63 68 5f  |00, info_search_|
000095b0  74 69 6d 65 3d 31 33 35  34 32 38 39 31 39 38 2e  |time=1354289198.|
000095c0  37 30 32 2c 20 63 6f 75  6e 74 3d 36 38 37 36 2c  |702, count=6876,|
000095d0  20 68 74 74 70 5f 75 73  65 72 5f 61 67 65 6e 74  | http_user_agent|
000095e0  3d 22 4d 6f 7a 69 6c 6c  61 2f 35 2e 30 20 28 63  |="Mozilla/5.0 (c|
000095f0  6f 6d 70 61 74 69 62 6c  65 3b 20 4d 53 49 45 20  |ompatible; MSIE |
00009600  39 2e 30 3b 20 57 69 6e  64 6f 77 73 20 4e 54 20  |9.0; Windows NT |
00009610  36 2e 31 3b 20 54 72 69  64 65 6e 74 2f 35 2e 30  |6.1; Trident/5.0|
00009620  3b 20 42 4f 49 45 39 3b  45 4e 55 53 29 22 0a     |; BOIE9;ENUS)".|
0000962f

So, what causes binary data to be in a stash file?

Is there something about splunk-4.3.4 that would have started this?

Any other thoughts besides masking the problem with ignoring binary in props.conf?

Thanks in advance,

Sean

[sdwilkerson · Accepted solution]

Update:

Answering my own question here after talking to Support, PM and devs.

• What we are experiencing is part of a known bug: SPL-59578 that will be fixed in 4.3.6 and 5.0.3
• This bug comes up when large amounts of summary data is returned.
• Support says that in the known cases, the data was not invalid, so excluding the binary section within the input would be sufficient to mitigating the problem

• You can tell Splunk to read this data and exclude the binary data by using the NO_BINARY_CHECK in your props.conf with something like this:

  [stash_new]
  
  NO_BINARY_CHECK = true

One note, if this is particularly a problem with ES (Enterprise Security) then don't spend too much time with this issue and instead upgrade to ES-2.2.x or greater since from this version on, summary data is hardly used at all and won't be driving the majority of the dashboards like it does in previous versions.

When I get onsite today, I will finish the upgrade to ES-2.2, then evaluate if any of the other apps/SH are still experiencing this issue. I might put the NO_BINARY_CHECK work-around in place since other apps were having this problem previously.

[yannK]

similar thread here
http://splunk-base.splunk.com/answers/70072/summary-indexing-blocked-and-binary-file-warning
fix will be in 5.0.3