Sending multi-line Events as a single line to archive and unchanged to indexer

[dkrey]

Hi there,
I'm struggling with the following:
On a heavy forwarder I get two types of data: windows events and firewall logs and I need to forward the data to an archive, as well as to the indexer tier. But I also need the get rid of the line breaks of the windows events, before I send them to the archive (which is syslog-ng btw)

So when I clone the data with defaultGroups like this in the outputs.conf

[tcpout]
defaultGroup = indexers, syslog-ng-tls
indexAndForward = false

The data gets forwarded to both destinations, but I have no chance to remove the linebreaks since tcpout:syslog-ng-tls has no options for altering the data.

But when I do it via props.conf and transforms.conf like this:

props.conf:

[host::*]
SEDCMD-rmlines=s/[\n\r\t]/ /g
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE_DATE = True
MAX_EVENTS = 256
TRANSFORMS-syslog = syslog_routing

transforms.conf

[syslog_routing]
REGEX = .
DEST_KEY = _TCP_ROUTING
FORMAT = syslog-ng-tls

outputs.conf:

[tcpout]
defaultGroup = indexers
indexAndForward = false

[tcpout:indexers]
server = *.*.*.*:9997,*.*.*.*:9997
sslVersions = tls1.2

[tcpout:syslog-ng-tls]
server = *.*.*.*:1516
sendCookedData = false
useSSL = true
sslVerifyServerCert = false

Then the default group is being ignored and the events are being sent only to the syslog-server.

The latter way works on an indexer, but apperently not on a heavy forwarder.
So I wonder: is there a way to send the data unaltered to the indexers and transformed to the archive on a heavy forwarder?

Thanks for your help!

[DavidHourani · Accepted solution]

Hi @dkrey,

In your second approach you're only sending data to syslog, it will never go to the default tcp output. It works on an indexer because indexAndForward=truewhich means it's still not forwarding to two destinations, only to one but indexing locally as well.

To get it to forward to two places without the line breaks, try this config :
props.conf:

 [host::*]
 SEDCMD-rmlines=s/[\n\r\t]/ /g
 SHOULD_LINEMERGE = True
 BREAK_ONLY_BEFORE_DATE = True
 MAX_EVENTS = 256

And outputs.conf:

 [tcpout]
 defaultGroup = indexers, syslog-ng-tls
 indexAndForward = false

 [tcpout:indexers]
 server = *.*.*.*:9997,*.*.*.*:9997
 sslVersions = tls1.2

 [tcpout:syslog-ng-tls]
 server = *.*.*.*:1516
 sendCookedData = false
 useSSL = true
 sslVerifyServerCert = false

Let me know how that works out for you.

Cheers,
David

[dkrey]

Alright, so now I could either: forward to both places without line breaks or forward to both places with the data untouched.

But could I also forward the data a) untouched to the indexers and b) with line-breaks removed to syslog-ng?

[DavidHourani]

Hi @dkrey,

Once you set _TCP_ROUTINGdata will just go to the destinations specified by that option, this means you can either send the filtered data to two destinations or the non-filtered data to two destinations.

You have two solutions to handle what you're asking for :

1- Send the data twice to the HF from your UF. Handle one of the pipelines as non-filtered and send to your indexers and then filter the other pipeline and send it to your syslog.

2- (not recommended but possible) On you HF, you can loop your data, during the first loop you forward it to the indexers, during the second loop (entering on a new port on your same server) you filter and forward it to the syslog server. Not exactly sure if that's any good for production maybe @woodcock can help.

EDIT: You could also stick to letting the indexers forward to syslog if that's working for you, but you'll have to be sure it doesn't impact performance.

Cheers,
David

[dkrey]

Hi @DavidHourani,
thanks for all your help.
If I implement forwarding on both indexers, that would mean that I put twice the data in the archive, right?

[DavidHourani]

No, not at all, because the forwarding will only happen on the indexer receiving the data not during replication. So both indexers will be sending data, but it will be 50-50. Test it out before applying into production to confirm and avoid any issues.

[dkrey]

That sounds like the best solution! Thanks a lot!
Cheers
Dirk