Getting Data In

Sending HEC data to Nullqueue

sun1000
Path Finder

We are using HEC collector endpoint to consume logs from FluentD, we recently identified filtering opportunity and trying to apply props/transforms to send data to null queue which is not working.

Source field is sent by Fluentd, so we are using that field to create sourcetype as below

props.conf
[source::*.journald]
TRANSFORMS-override = override_st_journald,override_host_journald
SHOULD_LINEMERGE = false
TIME_PREFIX = SOURCE_REALTIME_TIMESTAMP\":\"
TIME_FORMAT = %s%6Q

transforms.conf
[override_st_journald]
SOURCE_KEY = _raw
REGEX = SYSTEMD_UNIT\":\"([^.\s\"0-9]+)
FORMAT = sourcetype::$1
DEST_KEY = MetaData:Sourcetype

[override_host_journald]
SOURCE_KEY = _raw
REGEX = instance_id\":\"([^\"]+)
FORMAT = host::$1
DEST_KEY = MetaData:Host

Now I want to send partial of data for this source to null queue which is not working

my configuration in props.conf
[source::.journald]
**TRANSFORMS-null= setnullsourcetype
*
TRANSFORMS-override = override_st_journald,override_host_journald
SHOULD_LINEMERGE = false
TIME_PREFIX = SOURCE_REALTIME_TIMESTAMP\":\"
TIME_FORMAT = %s%6Q

transforms.conf
[setnullsourcetype]
SOURCE_KEY = _raw
REGEX = \"SYSTEMD_UNIT\":\"rsyslog.service\"
DEST_KEY = queue
FORMAT = nullQueue

Can you please help me understand why it is not working. Please help me to identify how can I fix this

0 Karma
1 Solution

sun1000
Path Finder

@robert_miller  Yes, i was able to get around this

In props.conf., I added setnull3 at the end

[props.conf]
TRANSFORMS-override = override_st_journald,override_host_journald,setnull3

And in transforms.conf, I add the below for setnull3

[setnull3]
SOURCE_KEY = _raw
REGEX = \"SYSTEMD_UNIT\":(\"elasticsearch.service\"|\"rsyslog.service\")
DEST_KEY = queue
FORMAT = nullQueue

 

Please accept my answer if this solved your problem

View solution in original post

0 Karma

robert_miller
Path Finder

@sun1000 Did you ever figure this out? I am running into the same issue with nullqueue not working with HEC.

0 Karma

sun1000
Path Finder

@robert_miller  Yes, i was able to get around this

In props.conf., I added setnull3 at the end

[props.conf]
TRANSFORMS-override = override_st_journald,override_host_journald,setnull3

And in transforms.conf, I add the below for setnull3

[setnull3]
SOURCE_KEY = _raw
REGEX = \"SYSTEMD_UNIT\":(\"elasticsearch.service\"|\"rsyslog.service\")
DEST_KEY = queue
FORMAT = nullQueue

 

Please accept my answer if this solved your problem

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

Are you using the raw or events endpoint with HEC? The way these process events is different.

Out of curiosity, you have :

[source::.journald]
TRANSFORMS-null= setnullsourcetype*

You can remove that *****from the end of the transforms. This should work as long and you're not using the /events endpoint on HEC.

0 Karma

to4kawa
Ultra Champion

this is markdown typo.

0 Karma

to4kawa
Ultra Champion

your props.conf is no problem.

check REGEX

\"SYSTEMD_UNIT\":\"rsyslog.service\"

and SOURCE_KEY = _raw is no need.

Don't forget reboot Splunk

0 Karma

sun1000
Path Finder

Regex is correct, I validated with regex101 and also ran search | regex _raw to validate it is correct
Even without source_key it is still a problem
Not sure why it is not working - Tried plenty of options
Please provide more details

0 Karma

to4kawa
Ultra Champion

how about rsyslog\.service ?
your log is JSON and auto extracted.

for me, I confirm REGEX by rex (regex101 is good, but default option is different from Splunk)

0 Karma

sun1000
Path Finder

I tried that one as well - Is there some limitation with HEC input ?

0 Karma

to4kawa
Ultra Champion

The input of HEC is stdout, not file.
maybe, there is extra spaces.

0 Karma

sun1000
Path Finder

no extra spaces

0 Karma

to4kawa
Ultra Champion

How can you be sure?

"SYSTEMD_UNIT": "rsyslog.service"

It was like this, wasn't it?

0 Karma
Get Updates on the Splunk Community!

What’s new on Splunk Lantern in August

This month’s Splunk Lantern update gives you the low-down on all of the articles we’ve published over the past ...

Welcome to the Future of Data Search & Exploration

You have more data coming at you than ever before. Over the next five years, the total amount of digital data ...

This Week's Community Digest - Splunk Community Happenings [8.3.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...