Getting Data In

Sending HEC data to Nullqueue

sun1000
Path Finder

We are using HEC collector endpoint to consume logs from FluentD, we recently identified filtering opportunity and trying to apply props/transforms to send data to null queue which is not working.

Source field is sent by Fluentd, so we are using that field to create sourcetype as below

props.conf
[source::*.journald]
TRANSFORMS-override = override_st_journald,override_host_journald
SHOULD_LINEMERGE = false
TIME_PREFIX = SOURCE_REALTIME_TIMESTAMP\":\"
TIME_FORMAT = %s%6Q

transforms.conf
[override_st_journald]
SOURCE_KEY = _raw
REGEX = SYSTEMD_UNIT\":\"([^.\s\"0-9]+)
FORMAT = sourcetype::$1
DEST_KEY = MetaData:Sourcetype

[override_host_journald]
SOURCE_KEY = _raw
REGEX = instance_id\":\"([^\"]+)
FORMAT = host::$1
DEST_KEY = MetaData:Host

Now I want to send partial of data for this source to null queue which is not working

my configuration in props.conf
[source::.journald]
**TRANSFORMS-null= setnullsourcetype
*
TRANSFORMS-override = override_st_journald,override_host_journald
SHOULD_LINEMERGE = false
TIME_PREFIX = SOURCE_REALTIME_TIMESTAMP\":\"
TIME_FORMAT = %s%6Q

transforms.conf
[setnullsourcetype]
SOURCE_KEY = _raw
REGEX = \"SYSTEMD_UNIT\":\"rsyslog.service\"
DEST_KEY = queue
FORMAT = nullQueue

Can you please help me understand why it is not working. Please help me to identify how can I fix this

0 Karma
1 Solution

sun1000
Path Finder

@robert_miller  Yes, i was able to get around this

In props.conf., I added setnull3 at the end

[props.conf]
TRANSFORMS-override = override_st_journald,override_host_journald,setnull3

And in transforms.conf, I add the below for setnull3

[setnull3]
SOURCE_KEY = _raw
REGEX = \"SYSTEMD_UNIT\":(\"elasticsearch.service\"|\"rsyslog.service\")
DEST_KEY = queue
FORMAT = nullQueue

 

Please accept my answer if this solved your problem

View solution in original post

0 Karma

robert_miller
Path Finder

@sun1000 Did you ever figure this out? I am running into the same issue with nullqueue not working with HEC.

0 Karma

sun1000
Path Finder

@robert_miller  Yes, i was able to get around this

In props.conf., I added setnull3 at the end

[props.conf]
TRANSFORMS-override = override_st_journald,override_host_journald,setnull3

And in transforms.conf, I add the below for setnull3

[setnull3]
SOURCE_KEY = _raw
REGEX = \"SYSTEMD_UNIT\":(\"elasticsearch.service\"|\"rsyslog.service\")
DEST_KEY = queue
FORMAT = nullQueue

 

Please accept my answer if this solved your problem

View solution in original post

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

Are you using the raw or events endpoint with HEC? The way these process events is different.

Out of curiosity, you have :

[source::.journald]
TRANSFORMS-null= setnullsourcetype*

You can remove that *****from the end of the transforms. This should work as long and you're not using the /events endpoint on HEC.

0 Karma

to4kawa
SplunkTrust
SplunkTrust

this is markdown typo.

0 Karma

to4kawa
SplunkTrust
SplunkTrust

your props.conf is no problem.

check REGEX

\"SYSTEMD_UNIT\":\"rsyslog.service\"

and SOURCE_KEY = _raw is no need.

Don't forget reboot Splunk

0 Karma

sun1000
Path Finder

Regex is correct, I validated with regex101 and also ran search | regex _raw to validate it is correct
Even without source_key it is still a problem
Not sure why it is not working - Tried plenty of options
Please provide more details

0 Karma

to4kawa
SplunkTrust
SplunkTrust

how about rsyslog\.service ?
your log is JSON and auto extracted.

for me, I confirm REGEX by rex (regex101 is good, but default option is different from Splunk)

0 Karma

sun1000
Path Finder

I tried that one as well - Is there some limitation with HEC input ?

0 Karma

to4kawa
SplunkTrust
SplunkTrust

The input of HEC is stdout, not file.
maybe, there is extra spaces.

0 Karma

sun1000
Path Finder

no extra spaces

0 Karma

to4kawa
SplunkTrust
SplunkTrust

How can you be sure?

"SYSTEMD_UNIT": "rsyslog.service"

It was like this, wasn't it?

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.