topic Re: Sending HEC data to Nullqueue in Getting Data In

Sending HEC data to Nullqueue

sun1000 — Wed, 30 Sep 2020 05:15:43 GMT

We are using HEC collector endpoint to consume logs from FluentD, we recently identified filtering opportunity and trying to apply props/transforms to send data to null queue which is not working.

Source field is sent by Fluentd, so we are using that field to create sourcetype as below

props.conf
[source::*.journald]
TRANSFORMS-override = override_st_journald,override_host_journald
SHOULD_LINEMERGE = false
TIME_PREFIX = SOURCE_REALTIME_TIMESTAMP\":\"
TIME_FORMAT = %s%6Q

transforms.conf
[override_st_journald]
SOURCE_KEY = _raw
REGEX = SYSTEMD_UNIT\":\"([^.\s\"0-9]+)
FORMAT = sourcetype::$1
DEST_KEY = MetaData:Sourcetype

[override_host_journald]
SOURCE_KEY = _raw
REGEX = instance_id\":\"([^\"]+)
FORMAT = host::$1
DEST_KEY = MetaData:Host

Now I want to send partial of data for this source to null queue which is not working

my configuration in props.conf
[source::.journald]
**TRANSFORMS-null= setnullsourcetype*
TRANSFORMS-override = override_st_journald,override_host_journald
SHOULD_LINEMERGE = false
TIME_PREFIX = SOURCE_REALTIME_TIMESTAMP\":\"
TIME_FORMAT = %s%6Q

transforms.conf
[setnullsourcetype]
SOURCE_KEY = _raw
REGEX = \"SYSTEMD_UNIT\":\"rsyslog.service\"
DEST_KEY = queue
FORMAT = nullQueue

Can you please help me understand why it is not working. Please help me to identify how can I fix this

Re: Sending HEC data to Nullqueue

to4kawa — Thu, 30 Apr 2020 05:22:40 GMT

your props.conf is no problem.

check REGEX

\"SYSTEMD_UNIT\":\"rsyslog.service\"

and SOURCE_KEY = _raw is no need.

Don't forget reboot Splunk

Re: Sending HEC data to Nullqueue

sun1000 — Wed, 30 Sep 2020 05:16:18 GMT

Regex is correct, I validated with regex101 and also ran search | regex _raw to validate it is correct
Even without source_key it is still a problem
Not sure why it is not working - Tried plenty of options
Please provide more details

Re: Sending HEC data to Nullqueue

to4kawa — Thu, 30 Apr 2020 20:28:04 GMT

how about rsyslog\.service ?
your log is JSON and auto extracted.

for me, I confirm REGEX by rex (regex101 is good, but default option is different from Splunk)

Re: Sending HEC data to Nullqueue

sun1000 — Thu, 30 Apr 2020 20:30:11 GMT

I tried that one as well - Is there some limitation with HEC input ?

Re: Sending HEC data to Nullqueue

to4kawa — Thu, 30 Apr 2020 20:39:41 GMT

The input of HEC is stdout, not file.
maybe, there is extra spaces.

Re: Sending HEC data to Nullqueue

sun1000 — Tue, 05 May 2020 03:57:24 GMT

no extra spaces

Re: Sending HEC data to Nullqueue

to4kawa — Tue, 05 May 2020 05:50:45 GMT

How can you be sure?

"SYSTEMD_UNIT": "rsyslog.service"

It was like this, wasn't it?

Re: Sending HEC data to Nullqueue

esix_splunk — Tue, 05 May 2020 08:23:29 GMT

Are you using the raw or events endpoint with HEC? The way these process events is different.

Out of curiosity, you have :

[source::.journald]
TRANSFORMS-null= setnullsourcetype*

You can remove that *****from the end of the transforms. This should work as long and you're not using the /events endpoint on HEC.

Re: Sending HEC data to Nullqueue

to4kawa — Tue, 05 May 2020 08:32:55 GMT

this is markdown typo.

Re: Sending HEC data to Nullqueue

robert_miller — Mon, 24 May 2021 21:22:38 GMT

@sun1000 Did you ever figure this out? I am running into the same issue with nullqueue not working with HEC.

Re: Sending HEC data to Nullqueue

sun1000 — Tue, 25 May 2021 04:43:21 GMT

@robert_miller Yes, i was able to get around this

In props.conf., I added setnull3 at the end

[props.conf]
TRANSFORMS-override = override_st_journald,override_host_journald,setnull3

And in transforms.conf, I add the below for setnull3

[setnull3]
SOURCE_KEY = _raw
REGEX = \"SYSTEMD_UNIT\":(\"elasticsearch.service\"|\"rsyslog.service\")
DEST_KEY = queue
FORMAT = nullQueue

Please accept my answer if this solved your problem