topic Re: Are logs still kept locally? in Getting Data In

Are logs still kept locally?

mellqui — Fri, 03 Sep 2021 14:08:39 GMT

Brand new to using the Universal Forwarder, and Splunk in general.

Question:

When using the forwarder/monitor, the logs on the forwarding server are still kept locally, correct? They aren't removed/modified in any way?

Re: Are logs still kept locally?

gcusello — Fri, 03 Sep 2021 14:32:01 GMT

Hi @mellqui,

If your trouble if for eventual modifications of the read log files, I can confirm that there isn't any modification of the local log files.

If instead your trouble is that there could be a modification of the logs before arriving to Splunk, it's possible as I describe below, it isn't possible to modify data after Splunk Indexing, or better ,if someone modify indexed data this is highlighted by an Integrity Check fail.

By default logs on forwarder, can be filtered (only Windows eventlog) on Forwarders but not modified.

It's possible to modify data on Indexers before indexing, e.g. to mask credit card numbers.

Then, after indexing, isn't possible to modify data.

In other words, analyzing the indexing process:

Obviously, you can modify a log before ingestion by the Forwarder, in this case, Splunk isn't able to check the data modification,
when data is ingested by the Forwarder, isn't possible to modify them until they arrive to indexer, eventually using SSL for securing the transmission,
When data arrive to the indexer, it's possing to modify them, but only using special configurations of the Indexers, but not manually,
after data are indexed they aren't modificable more.
The modification by configurations (not manual) is also possible on intermediate Heavy Forwarders that works as an Indexer.

In few words, if you control the process and the configurations, it's possible to modify Splunk data only if you want.

Ciao.

Giuseppe

Re: Are logs still kept locally?

mellqui — Fri, 03 Sep 2021 15:43:48 GMT

Hey @gcusello,

I'm more less looking to see if the local log files are removed/transferred from the server when they are forwarded. We have other monitoring tools that use these logs.

Thanks for your reply.

Re: Are logs still kept locally?

yannK — Fri, 03 Sep 2021 16:17:52 GMT

Hi.

If you have a splunk forwarder using a monitor input or a modular input to read local log files, it will just read them and forward them to splunk indexers. It does not delete the local logs.

The only exception is if you are using a "batch" inputs stanza with the move_policy = sinkhole, then splunk will read the file and delete it once finished (so you do not want to use that for dynamic files, only static files). This is the mechanism used in the splunk "spooler" inputs. (if you drop a file in the $SPLUNK/var/spool/splunk folder)

see https://docs.splunk.com/Documentation/Splunk/8.2.2/Data/Monitorfilesanddirectorieswithinputs.conf#Batch_syntax

Remarks :

- keep in mind that your OS or application generating your log may have their own rotation/archive/deletion rules.
- Splunk internal logs (like $SPLUNK_HOME/var/log/splunk/*.log) do have their own rotation mechanism. So you may still find the recent ones locally, while a copy was ingested in splunk index=_internal.

Re: Are logs still kept locally?

mellqui — Fri, 03 Sep 2021 16:55:02 GMT

Hey @yannK,

This answers my question -- much appreciated!

Re: Are logs still kept locally?

gcusello — Sat, 04 Sep 2021 09:05:00 GMT

Hi @mellqui,

good for you, see next time!

Ciao and happy splunking.

Giuseppe

P.S.: Karma points are appreciated by all the Contributors 😉