Forward logs to syslog with metadata

[prioska]

Hello everyone,

I have a splunk server installed locally and there are logs being ingested already.

I'd like to forward the logs to a syslog server while adding the host, index, source and sourcetype metadata to the data being forwarded.

I have used the following configuration:

outputs.confg:

```

[tcpout]
defaultGroup = logstash
indexAndForward = true

[tcpout:logstash]
server = 192.168.100.173:514
sendCookedData = false
type = tcp

```

props.conf

```

[default]
TRANSFORMS-GC_throughput = GC_add_sourcetype, GC_add_source, GC_add_index, GC_add_host

```

transforms.conf

```

[GC_add_host]
SOURCE_KEY = MetaData:Host
REGEX = ^host::(.*)$
FORMAT = host=$1 $0
DEST_KEY = _raw

[GC_add_index]
SOURCE_KEY = MetaData:Index
REGEX = ^index::(.*)$
FORMAT = index=$1 $0
DEST_KEY = _raw

[GC_add_source]
SOURCE_KEY = MetaData:Source
REGEX = ^source::(.*)$
FORMAT = source=$1 $0
DEST_KEY = _raw

[GC_add_sourcetype]
SOURCE_KEY = MetaData:Sourcetype
REGEX = ^sourcetype::(.*)$
FORMAT = sourcetype=$1 $0
DEST_KEY = _raw

```

And I was able to make the logs having a prepended host, index, source and sourcetype. But the issue is that I'd like to make Splunk keep logging the cooked data while it transforms the uncooked data by adding the mentioned metadata without indexing it.

With that actual configuration Splunk will index the transformed data and it messes with everything (the parsed fields). I don't want to touch the original parsed data that used to be ingested.

If I set `indexAndForward = false`, the logs stop being ingested in Splunk.

The reason why I'm adding the prepended metadata is because at the Syslog server level I have logstash and I'm planning to parse the received data to identify from where it came and based on that I'll delete the prepended metadata and I'll parse the remaining original logs (they could be in text format or in json format or in an xml format).

I'm not the real owner of Splunk server so I'd like to avoid changing how the logs used to be indexed there. That's why I wished to avoid changing it during the indexing.

Is my configuration correct and I misunderstood something? or is my configuration wrong?

Thank you

[PickleRick]

Well, it just works this way. See the invaluable Masa diagrams - https://community.splunk.com/t5/Getting-Data-In/Diagrams-of-how-indexing-works-in-the-Splunk-platfor...

An event passes through the indexing pipeline and at the end it's being written to a local index or (here the alternative is non-exclusive) getting sent out to output(s).

The important thing is that it's still the same event. So you cannot modify an event only for a single output.

The only way to make it work with transforms would be to clone the event using CLONE_SOURCETYPE into a separate sourcetype and only this instance subject to modifications and only those events send to your syslog output.

Since you want to retain your sourcetype you'd have to save it into a field (which you'd have to remove in the original pipeline after cloning).

It is an ugly solution however, not intuitive and might be confusing to maintain.

One caveat - a transform-based solution will not work on events which had already been parsed.

With a sufficiently modern Splunk (it's introduced in 10.0) you could also try using Edge Processor.