Getting Data In
Highlighted

Sending HEC data to Nullqueue

Engager

We are using HEC collector endpoint to consume logs from FluentD, we recently identified filtering opportunity and trying to apply props/transforms to send data to null queue which is not working.

Source field is sent by Fluentd, so we are using that field to create sourcetype as below

props.conf
[source::*.journald]
TRANSFORMS-override = overridestjournald,overridehostjournald
SHOULDLINEMERGE = false
TIME
PREFIX = SOURCEREALTIMETIMESTAMP\":\"
TIME_FORMAT = %s%6Q

transforms.conf
[overridestjournald]
SOURCEKEY = _raw
REGEX = SYSTEMD
UNIT\":\"([^.\s\"0-9]+)
FORMAT = sourcetype::$1
DEST_KEY = MetaData:Sourcetype

[overridehostjournald]
SOURCEKEY = _raw
REGEX = instance
id\":\"([^\"]+)
FORMAT = host::$1
DEST_KEY = MetaData:Host

Now I want to send partial of data for this source to null queue which is not working

my configuration in props.conf
[source::.journald]
*
TRANSFORMS-null= setnullsourcetype**
TRANSFORMS-override = overridestjournald,overridehostjournald
SHOULDLINEMERGE = false
TIME
PREFIX = SOURCEREALTIMETIMESTAMP\":\"
TIME_FORMAT = %s%6Q

transforms.conf
[setnullsourcetype]
SOURCEKEY = _raw
REGEX = \"SYSTEMD
UNIT\":\"rsyslog.service\"
DEST_KEY = queue
FORMAT = nullQueue

Can you please help me understand why it is not working. Please help me to identify how can I fix this

0 Karma
Highlighted

Re: Sending HEC data to Nullqueue

Ultra Champion

your props.conf is no problem.

check REGEX

\"SYSTEMD_UNIT\":\"rsyslog.service\"

and SOURCE_KEY = _raw is no need.

Don't forget reboot Splunk

0 Karma
Highlighted

Re: Sending HEC data to Nullqueue

Engager

Regex is correct, I validated with regex101 and also ran search | regex raw to validate it is correct
Even without source
key it is still a problem
Not sure why it is not working - Tried plenty of options
Please provide more details

0 Karma
Highlighted

Re: Sending HEC data to Nullqueue

Ultra Champion

how about rsyslog\.service ?
your log is JSON and auto extracted.

for me, I confirm REGEX by rex (regex101 is good, but default option is different from Splunk)

0 Karma
Highlighted

Re: Sending HEC data to Nullqueue

Engager

I tried that one as well - Is there some limitation with HEC input ?

0 Karma
Highlighted

Re: Sending HEC data to Nullqueue

Ultra Champion

The input of HEC is stdout, not file.
maybe, there is extra spaces.

0 Karma
Highlighted

Re: Sending HEC data to Nullqueue

Engager

no extra spaces

0 Karma
Highlighted

Re: Sending HEC data to Nullqueue

Ultra Champion

How can you be sure?

"SYSTEMD_UNIT": "rsyslog.service"

It was like this, wasn't it?

0 Karma
Highlighted

Re: Sending HEC data to Nullqueue

Super Champion

Are you using the raw or events endpoint with HEC? The way these process events is different.

Out of curiosity, you have :

[source::.journald]
TRANSFORMS-null= setnullsourcetype*

You can remove that *****from the end of the transforms. This should work as long and you're not using the /events endpoint on HEC.

0 Karma
Highlighted

Re: Sending HEC data to Nullqueue

Ultra Champion

this is markdown typo.

0 Karma
Speak Up for Splunk Careers!

We want to better understand the impact Splunk experience and expertise has has on individuals' careers, and help highlight the growing demand for Splunk skills.