topic Re: Stash Files contain binary data in Splunk Enterprise Security

Stash Files contain binary data

sdwilkerson — Mon, 17 Dec 2012 22:56:56 GMT

Folks,

I'm at a site with 3 search heads and 6 indexers. One of the SH is ES-2.0.2.

All SH and Indexers were upgraded to Splunk-5.0.1 last week.

All SH are configured to spray/output summaries back to Indexers.

All three SH have unprocessed stash files in $SPLUNK_HOME/var/spool/splunk. These unprocessed files started the day the customer upgraded from 4.3.3 to 4.3.4 back on Oct. 24th and there are some-to-many a day since then.

index=_internal shows logs like this:

12-17-2012 15:57:37.416 -0500 INFO  TailingProcessor - Ignoring file /opt/splunk/var/spool/splunk/Network
- Proxy Events by Content Type - Summary Gen_829222797.stash_new' due to: binary 
12-17-2012 15:57:37.416 -0500 WARN  FileClassifierManager - The file /opt/splunk/var/spool/splunk/Network - Proxy Events by Content Type - Summary Gen_829222797.stash_new' is invalid. Reason: binary

When inspecting these unprocessed stash files, they all seem to contain blocks of binary data. Unprocessed stash files representing ES data are the biggest culprit but I see some from other apps too including DeploymentMonitor.

Using vi/more/shell I can locate a single line that is binary data. Using sed to extract that line and hexdump, I see this:

00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00009540  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 61  |...............a|
00009550  6d 65 3d 22 4e 65 74 77  6f 72 6b 20 2d 20 50 72  |me="Network - Pr|
00009560  6f 78 79 20 45 76 65 6e  74 73 20 62 79 20 55 73  |oxy Events by Us|
00009570  65 72 20 41 67 65 6e 74  20 2d 20 53 75 6d 6d 61  |er Agent - Summa|
00009580  72 79 20 47 65 6e 22 2c  20 73 65 61 72 63 68 5f  |ry Gen", search_|
00009590  6e 6f 77 3d 31 33 35 34  32 38 39 31 39 38 2e 30  |now=1354289198.0|
000095a0  30 30 2c 20 69 6e 66 6f  5f 73 65 61 72 63 68 5f  |00, info_search_|
000095b0  74 69 6d 65 3d 31 33 35  34 32 38 39 31 39 38 2e  |time=1354289198.|
000095c0  37 30 32 2c 20 63 6f 75  6e 74 3d 36 38 37 36 2c  |702, count=6876,|
000095d0  20 68 74 74 70 5f 75 73  65 72 5f 61 67 65 6e 74  | http_user_agent|
000095e0  3d 22 4d 6f 7a 69 6c 6c  61 2f 35 2e 30 20 28 63  |="Mozilla/5.0 (c|
000095f0  6f 6d 70 61 74 69 62 6c  65 3b 20 4d 53 49 45 20  |ompatible; MSIE |
00009600  39 2e 30 3b 20 57 69 6e  64 6f 77 73 20 4e 54 20  |9.0; Windows NT |
00009610  36 2e 31 3b 20 54 72 69  64 65 6e 74 2f 35 2e 30  |6.1; Trident/5.0|
00009620  3b 20 42 4f 49 45 39 3b  45 4e 55 53 29 22 0a     |; BOIE9;ENUS)".|
0000962f

So, what causes binary data to be in a stash file?

Is there something about splunk-4.3.4 that would have started this?

Any other thoughts besides masking the problem with ignoring binary in props.conf?

Thanks in advance,

Sean

Re: Stash Files contain binary data

sdwilkerson — Mon, 28 Sep 2020 12:59:12 GMT

Update:

Answering my own question here after talking to Support, PM and devs.

What we are experiencing is part of a known bug: SPL-59578 that will be fixed in 4.3.6 and 5.0.3
This bug comes up when large amounts of summary data is returned.
Support says that in the known cases, the data was not invalid, so excluding the binary section within the input would be sufficient to mitigating the problem
You can tell Splunk to read this data and exclude the binary data by using the NO_BINARY_CHECK in your props.conf with something like this:

[stash_new]

NO_BINARY_CHECK = true

One note, if this is particularly a problem with ES (Enterprise Security) then don't spend too much time with this issue and instead upgrade to ES-2.2.x or greater since from this version on, summary data is hardly used at all and won't be driving the majority of the dashboards like it does in previous versions.

When I get onsite today, I will finish the upgrade to ES-2.2, then evaluate if any of the other apps/SH are still experiencing this issue. I might put the NO_BINARY_CHECK work-around in place since other apps were having this problem previously.

Re: Stash Files contain binary data

yannK — Fri, 08 Feb 2013 23:07:54 GMT

similar thread here
http://splunk-base.splunk.com/answers/70072/summary-indexing-blocked-and-binary-file-warning
fix will be in 5.0.3