topic Re: Splunk is not indexing log events that may have binary content in Getting Data In

Splunk is not indexing log events that may have binary content

lpolo — Mon, 28 Sep 2020 12:56:48 GMT

I have a source log that sometimes contains binary characters. Splunk is not indexing any events for this source type. The source type in question was configured as follow at the universal forwarder:
This configuration should work but it is not... Any idea...

I tried with NO_BINARY_CHECK = true and NO_BINARY_CHECK = false.

Thanks,
Lp

inputs.conf:

[monitor:///usr/local/azkaban/logs/azkaban.log]
sourcetype = azkaban
_blacklist=\.(gz|log.*|out.*|run.properties)$
disabled = false

props.conf:

[azkaban]
NO_BINARY_CHECK = true

The problem is that this source type is not being indexed and events are being appended. Splunkd.log does not complain about it.

12-11-2012 14:24:37.318 +0000 INFO  TailingProcessor - Parsing configuration stanza: monitor:///usr/local/azkaban/logs/azkaban.log.

Btool reports:
inputs

[monitor:///usr/local/azkaban/logs/azkaban.log]
_blacklist = \.(gz|log.*|out.*|run.properties)$
_rcvbuf = 1572864
blacklist = \.(gz|log.*|out.*)$
disabled = false
host = abc.com
index = default
sourcetype = azkaban

props

[azkaban]
ANNOTATE_PUNCT = True
BREAK_ONLY_BEFORE =
BREAK_ONLY_BEFORE_DATE = True
CHARSET = UTF-8
DATETIME_CONFIG = /etc/datetime.xml
HEADER_MODE =
LEARN_SOURCETYPE = true
LINE_BREAKER_LOOKBEHIND = 100
MAX_DAYS_AGO = 2000
MAX_DAYS_HENCE = 2
MAX_DIFF_SECS_AGO = 3600
MAX_DIFF_SECS_HENCE = 604800
MAX_EVENTS = 256
MAX_TIMESTAMP_LOOKAHEAD = 128
MUST_BREAK_AFTER =
MUST_NOT_BREAK_AFTER =
MUST_NOT_BREAK_BEFORE =
NO_BINARY_CHECK = true
SEGMENTATION = indexing
SEGMENTATION-all = full
SEGMENTATION-inner = inner
SEGMENTATION-outer = outer
SEGMENTATION-raw = none
SEGMENTATION-standard = standard
SHOULD_LINEMERGE = True
TRANSFORMS =
TRUNCATE = 10000
TZ = UTC

Re: Splunk is not indexing log events that may have binary content

Drainy — Tue, 11 Dec 2012 14:53:13 GMT

Your config looks wrong to me, from the docs;

NO_BINARY_CHECK = [true|false]
* When set to true, Splunk processes binary files.
* Can only be used on the basis of [<sourcetype>], or [source::<source>], not [host::<host>].
* Defaults to false (binary files are ignored).

At the moment you have it configured to ignore binary files

Re: Splunk is not indexing log events that may have binary content

lpolo — Mon, 28 Sep 2020 12:56:53 GMT

I tried both ways:

NO_BINARY_CHECK = true and NO_BINARY_CHECK = false

Still does not work. Any ideas...

Re: Splunk is not indexing log events that may have binary content

RicoSuave — Wed, 12 Dec 2012 01:07:36 GMT

[monitor:///usr/local/azkaban/logs/azkaban.log]

_blacklist = .(gz|log.|out.|run.properties)$
_rcvbuf = 1572864
blacklist = .(gz|log.|out.)$
disabled = false
host = abc.com
index = default
sourcetype = azkaban

You are blacklisting .log

Re: Splunk is not indexing log events that may have binary content

lpolo — Wed, 12 Dec 2012 02:20:37 GMT

I disagree. I am blacklisting ".log.". See the initial question.

_blacklist=.(gz|log.|out.|run.properties)$

Re: Splunk is not indexing log events that may have binary content

lpolo — Wed, 12 Dec 2012 15:59:44 GMT

For the benefit of doubt I tried your recommended blacklist. It worked. I do not see why the _blacklist presented in original question should not work.

Thanks,
LP

Re: Splunk is not indexing log events that may have binary content

lpolo — Wed, 12 Dec 2012 16:16:11 GMT

My original config was this one:

[monitor:///usr/local/azkaban/logs/azkaban.log]
sourcetype = azkaban
_blacklist=\.(gz|log.*|out.*|run.properties)$
disabled = false

I changed it to:

[monitor:///usr/local/azkaban/logs/azkaban.log]
sourcetype = azkaban
blacklist=.(gz|log.*|out.*|run.properties)$
disabled = false

And the source type is back to work.

Thanks,
Lp

Re: Splunk is not indexing log events that may have binary content

lpolo — Wed, 12 Dec 2012 16:21:50 GMT

My original config was this one:

[monitor:///usr/local/azkaban/logs/azkaban.log]
sourcetype = azkaban
_blacklist=\.(gz|log.*|out.*|run.properties)$
disabled = false

I changed it to:

[monitor:///usr/local/azkaban/logs/azkaban.log]
sourcetype = azkaban
blacklist=.(gz|log.*|out.*|run.properties)$
disabled = false

And the source type is back to work.

Thanks,
Lp