Splunk logs dedup/consolidation

[astatrial]

Hi Splunkers!

Do any of you know if there is a built-in feature or mechanism in Splunk that aggregates similar logs into one log?

In other words, i want to know if Splunk knows not to index identical or similar logs, and by that, to reduce the license's usage.

I know that this can be achieved by manually editing the props and transforms files, but if i'm not wrong, this process will need to be done specifically to every log structure.

I'm looking for something more generic and on the back-end

Thanks !

[richgalloway]

Splunk forwarders will not ingest the same file twice (there are ways around that), but the same data from multiple sources are happily onboarded.

[astatrial]

Hi, thanks for the response,
I will try to explain myself better.
I want to know if there is any process that Splunk does referring to the content of the log, and not on the file level.
For example, if there is an event with the same parameters at the same timestamp/very close timestamps, it will indexed only one time.
Like coalescing in QRadar.

[richgalloway]

Splunk ingests everything. There is no automatic deduplication.

[astatrial]

Thanks, I already managed to put together an answer for that.
There is a raw data compression, which is approximately 40%-50% than the original raw data size, but it doesn't affect the license.

[richgalloway]

Data compression is not the same as deduplicating or coalescing.

[astatrial]

Yes, i am aware of that, but as well as i could find, this is the only built-in thing that decrease the size of the logs in Splunk (as you mentioned, there is no deduplicating or coalescing).
In any case, thanks for everything.

[adonio]

what is the problem you are trying to solve?
where is the data coming from? can you elaborate on the meaning of "... similar logs into one log ..."?