Forward data input copy to syslog without apply SEDCMD

fabiocaldas — Mon, 28 Sep 2020 15:09:27 GMT

I collect my data using UniveralForwarder, them send it to HeavyForwarder.

I would like to send a copy of data that enter into HF to a syslog server. On my HF, I have the following configs:

inputs.conf

[splunktcp://9997]

disabled = 0

_SYSLOG_ROUTING = syslogserver

outputs.conf

[syslog:syslogserver]

server = MY-SYSLOG-IP:514

sendCookedData = false

[tcpout:LB_indexers]

server=MY-SPLUNK-IDX-SERVER:9997

[tcpout]

defaultGroup=LB_indexers

disabled=false

props.conf

[WinEventLog:Application]

SEDCMD-remove_eol = s/[\n\r]/ /g

SEDCMD-remove_message_flag = s/(\d{2}\/\d{2}\/\d{4}\s\d{2}:\d{2}:\d{2}\s\w{2})(.)(VtexLog|VtexMetric|VtexMachineMetric)(.)(Message=# )(.*)/\1 \3 \6/g

The point is that, I would like to apply those SEDCMD only to data that is sent to Splunk Indexer. I would like to send to syslog the data without any cut.

I need suggestions, thanks folks !!

Re: Forward data input copy to syslog without apply SEDCMD

fabiocaldas — Mon, 28 Sep 2020 15:12:48 GMT

So, in order to solve my problem I had to re-think about it and thanks God it´s worked.

I will put config first, and them I will explain the idea behind it.

outputs.conf

[tcpout:LB_indexers]

server=SPLUNK-IDX-IP:9997

[tcpout]

defaultGroup=LB_indexers

disabled=false

[syslog:my_syslog]

server = SYSLOG-IP:514

props.conf

[WinEventLog:Application]

SEDCMD-remove_eol = s/[\n\r]/ /g

SEDCMD-remove_message_flag = s/(\d{2}\/\d{2}\/\d{4}\s\d{2}:\d{2}:\d{2}\s\w{2})(.)(VtexMetric|VtexMachineMetric)(.)(Message=# )(.*)/\1 \3 \6/g

TRANSFORMS-sendData = send2Syslog,sendNull,setLogger

TRANSFORMS-setDataSource = setMetricSource,setMachineMetricSource

TRANSFORMS-setDataIndex = setLoggerLogsIndex,setRequestCaptureIndex,setLogisticsIndex

transforms.conf

[send2Syslog]

REGEX = VtexLog

DEST_KEY = _SYSLOG_ROUTING

FORMAT = sumologic_syslog

[sendNull]

REGEX = ^(?!.VtexLog|VtexMetric|VtexMachineMetric).$

DEST_KEY = queue

FORMAT = nullQueue

[setLogger]

REGEX = VtexLog|VtexMetric|VtexMachineMetric

DEST_KEY = queue

FORMAT = indexQueue

[setMetricSource]

REGEX = VtexMetric

FORMAT = sourcetype::VtexMetric

DEST_KEY = MetaData:Sourcetype

[setMachineMetricSource]

REGEX = VtexMachineMetric

FORMAT = sourcetype::VtexMachineMetric

DEST_KEY = MetaData:Sourcetype

[setLoggerLogsIndex]

REGEX = VtexLog

FORMAT = logger_logs

DEST_KEY = _MetaData:Index

[setRequestCaptureIndex]

REGEX = VtexMetric|VtexMachineMetric.*AppName=vtex.requestcapture

FORMAT = rc

DEST_KEY = _MetaData:Index

[setLogisticsIndex]

REGEX = VtexMetric|VtexMachineMetric.*AppName=vtex.commerce.logistics

FORMAT = logistics

DEST_KEY = _MetaData:Index

The objetive was to send to Splunk index only data generated from EventView with source VtexMetric or VtexMachineMetric. At same time I would like to forward data with source VtexLog to a third system syslog.

On Props file first the line breaks are removed, then for VtexMetric and VtexMachineMetric data is cleaned it has the EventViewer header removed and only Message data is keeped.

The tricks are on transforms, first data that has VtexLog is forwarder to _SYSLOG_ROUTING at same time there is a selective filtering running on sendNull and setLogger stanza. On sendNull all data that aren´t VtexLog or VtexMetric or VtexMachineMetric is discarded, that way I avoided windows logs. The oposite is made on setLogger where I set to indexQueue data that has desired sources.

On setMetricSource and setMachineMetricSource transforms the sourcetype is changed based on windows event viewer source.

Things stats to be cool on setLoggerLogsIndex transformation, since I allow logs to be indexed on setLogger transformation now I redirect all logs to this specific indexs. Remember that I don´t want to have those data indexed and I only keep it in order to continue to send it to third party syslog system. So the trick is create the index on Splunk Indexer and then disable it. If you desire you can send your not wanted data to a non-created index but it will put a warning on your SearchHead.

On setRequestCaptureIndex and setLogisticsIndex I finish my needs sending rest of data to correct index.

And that how I made to send data to a third party syslog server without index what I don´t want.

topic Re: Forward data input copy to syslog without apply SEDCMD in Getting Data In

Forward data input copy to syslog without apply SEDCMD

Re: Forward data input copy to syslog without apply SEDCMD