We are using HEC collector endpoint to consume logs from FluentD, we recently identified filtering opportunity and trying to apply props/transforms to send data to null queue which is not working.
Source field is sent by Fluentd, so we are using that field to create sourcetype as below
props.conf
[source::*.journald]
TRANSFORMS-override = override_st_journald,override_host_journald
SHOULD_LINEMERGE = false
TIME_PREFIX = SOURCE_REALTIME_TIMESTAMP\":\"
TIME_FORMAT = %s%6Q
transforms.conf
[override_st_journald]
SOURCE_KEY = _raw
REGEX = SYSTEMD_UNIT\":\"([^.\s\"0-9]+)
FORMAT = sourcetype::$1
DEST_KEY = MetaData:Sourcetype
[override_host_journald]
SOURCE_KEY = _raw
REGEX = instance_id\":\"([^\"]+)
FORMAT = host::$1
DEST_KEY = MetaData:Host
Now I want to send partial of data for this source to null queue which is not working
my configuration in props.conf
[source::.journald]
**TRANSFORMS-null= setnullsourcetype*
TRANSFORMS-override = override_st_journald,override_host_journald
SHOULD_LINEMERGE = false
TIME_PREFIX = SOURCE_REALTIME_TIMESTAMP\":\"
TIME_FORMAT = %s%6Q
transforms.conf
[setnullsourcetype]
SOURCE_KEY = _raw
REGEX = \"SYSTEMD_UNIT\":\"rsyslog.service\"
DEST_KEY = queue
FORMAT = nullQueue
Can you please help me understand why it is not working. Please help me to identify how can I fix this
@robert_miller Yes, i was able to get around this
In props.conf., I added setnull3 at the end
[props.conf]
TRANSFORMS-override = override_st_journald,override_host_journald,setnull3
And in transforms.conf, I add the below for setnull3
[setnull3]
SOURCE_KEY = _raw
REGEX = \"SYSTEMD_UNIT\":(\"elasticsearch.service\"|\"rsyslog.service\")
DEST_KEY = queue
FORMAT = nullQueue
Please accept my answer if this solved your problem
@sun1000 Did you ever figure this out? I am running into the same issue with nullqueue not working with HEC.
@robert_miller Yes, i was able to get around this
In props.conf., I added setnull3 at the end
[props.conf]
TRANSFORMS-override = override_st_journald,override_host_journald,setnull3
And in transforms.conf, I add the below for setnull3
[setnull3]
SOURCE_KEY = _raw
REGEX = \"SYSTEMD_UNIT\":(\"elasticsearch.service\"|\"rsyslog.service\")
DEST_KEY = queue
FORMAT = nullQueue
Please accept my answer if this solved your problem
Are you using the raw or events endpoint with HEC? The way these process events is different.
Out of curiosity, you have :
[source::.journald]
TRANSFORMS-null= setnullsourcetype*
You can remove that *****from the end of the transforms. This should work as long and you're not using the /events endpoint on HEC.
this is markdown typo.
your props.conf is no problem.
check REGEX
\"SYSTEMD_UNIT\":\"rsyslog.service\"
and SOURCE_KEY = _raw
is no need.
Don't forget reboot Splunk
Regex is correct, I validated with regex101 and also ran search | regex _raw to validate it is correct
Even without source_key it is still a problem
Not sure why it is not working - Tried plenty of options
Please provide more details
how about rsyslog\.service
?
your log is JSON and auto extracted.
for me, I confirm REGEX by rex
(regex101 is good, but default option is different from Splunk)
I tried that one as well - Is there some limitation with HEC input ?
The input of HEC is stdout, not file.
maybe, there is extra spaces.
no extra spaces
How can you be sure?
"SYSTEMD_UNIT": "rsyslog.service"
It was like this, wasn't it?