Getting Data In

Splunk is not indexing log events that may have binary content

lpolo
Motivator

I have a source log that sometimes contains binary characters. Splunk is not indexing any events for this source type. The source type in question was configured as follow at the universal forwarder:
This configuration should work but it is not... Any idea...

I tried with NO_BINARY_CHECK = true and NO_BINARY_CHECK = false.

Thanks,
Lp

inputs.conf:

[monitor:///usr/local/azkaban/logs/azkaban.log]
sourcetype = azkaban
_blacklist=\.(gz|log.*|out.*|run.properties)$
disabled = false

props.conf:

[azkaban]
NO_BINARY_CHECK = true

The problem is that this source type is not being indexed and events are being appended. Splunkd.log does not complain about it.

12-11-2012 14:24:37.318 +0000 INFO  TailingProcessor - Parsing configuration stanza: monitor:///usr/local/azkaban/logs/azkaban.log.

Btool reports:
inputs

[monitor:///usr/local/azkaban/logs/azkaban.log]
_blacklist = \.(gz|log.*|out.*|run.properties)$
_rcvbuf = 1572864
blacklist = \.(gz|log.*|out.*)$
disabled = false
host = abc.com
index = default
sourcetype = azkaban

props

[azkaban]
ANNOTATE_PUNCT = True
BREAK_ONLY_BEFORE =
BREAK_ONLY_BEFORE_DATE = True
CHARSET = UTF-8
DATETIME_CONFIG = /etc/datetime.xml
HEADER_MODE =
LEARN_SOURCETYPE = true
LINE_BREAKER_LOOKBEHIND = 100
MAX_DAYS_AGO = 2000
MAX_DAYS_HENCE = 2
MAX_DIFF_SECS_AGO = 3600
MAX_DIFF_SECS_HENCE = 604800
MAX_EVENTS = 256
MAX_TIMESTAMP_LOOKAHEAD = 128
MUST_BREAK_AFTER =
MUST_NOT_BREAK_AFTER =
MUST_NOT_BREAK_BEFORE =
NO_BINARY_CHECK = true
SEGMENTATION = indexing
SEGMENTATION-all = full
SEGMENTATION-inner = inner
SEGMENTATION-outer = outer
SEGMENTATION-raw = none
SEGMENTATION-standard = standard
SHOULD_LINEMERGE = True
TRANSFORMS =
TRUNCATE = 10000
TZ = UTC
Tags (1)
0 Karma

lpolo
Motivator

My original config was this one:

[monitor:///usr/local/azkaban/logs/azkaban.log]
sourcetype = azkaban
_blacklist=\.(gz|log.*|out.*|run.properties)$
disabled = false

I changed it to:

[monitor:///usr/local/azkaban/logs/azkaban.log]
sourcetype = azkaban
blacklist=.(gz|log.*|out.*|run.properties)$
disabled = false

And the source type is back to work.

Thanks,
Lp

0 Karma

lpolo
Motivator

My original config was this one:

[monitor:///usr/local/azkaban/logs/azkaban.log]
sourcetype = azkaban
_blacklist=\.(gz|log.*|out.*|run.properties)$
disabled = false

I changed it to:

[monitor:///usr/local/azkaban/logs/azkaban.log]
sourcetype = azkaban
blacklist=.(gz|log.*|out.*|run.properties)$
disabled = false

And the source type is back to work.

Thanks,
Lp

0 Karma

RicoSuave
Builder
[monitor:///usr/local/azkaban/logs/azkaban.log]

_blacklist = .(gz|log.|out.|run.properties)$
_rcvbuf = 1572864
blacklist = .(gz|log.|out.)$
disabled = false
host = abc.com
index = default
sourcetype = azkaban

You are blacklisting .log

lpolo
Motivator

For the benefit of doubt I tried your recommended blacklist. It worked. I do not see why the _blacklist presented in original question should not work.

Thanks,
LP

0 Karma

lpolo
Motivator

I disagree. I am blacklisting ".log.". See the initial question.

_blacklist=.(gz|log.|out.|run.properties)$

0 Karma

Drainy
Champion

Your config looks wrong to me, from the docs;

NO_BINARY_CHECK = [true|false]
* When set to true, Splunk processes binary files.
* Can only be used on the basis of [<sourcetype>], or [source::<source>], not [host::<host>].
* Defaults to false (binary files are ignored).

At the moment you have it configured to ignore binary files

0 Karma

lpolo
Motivator

I tried both ways:

NO_BINARY_CHECK = true and NO_BINARY_CHECK = false

Still does not work. Any ideas...

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...