topic Re: What are the configurations required to forward specific log messages to Splunk? in Getting Data In

What are the configurations required to forward specific log messages to Splunk?

ssoftility — Fri, 29 Oct 2021 12:54:29 GMT

What are the configurations required to forward specific log messages to Splunk.

Every log message that contains "ScanStatistics" this phrase needs to get forwarded to Splunk.
Let us know what are the configurations to be done.

Re: What are the configurations required to forward specific log messages to Splunk?

gcusello — Fri, 29 Oct 2021 13:07:52 GMT

Hi @ssoftility,

some additional information, please:

what architecture do you have: stand-alone or distributed?
which kind of logs are you speaking of (wineventlog, syslog, linux, firewall, etc...)?
do you want to find the word "ScanStatistics" searching in your logs or do you want to index only the logs containing this word?
how do you ingest your logs, or you're not able to ingest logs and this is your main question?
could you share some example of your logs?

In few words, I need to understand:

the kind of logs you're speaking;
if you need an help in searching the word in your logs or in ingesting and indexing logs.

because:

knowing the kind of your logs I can suggest the best way to ingest and parse your logs;
if you're speaking of a search, I can hint the search to find the events that contain the above word;
if you're speaking of ingesting logs, I can hint how to ingest, parse and eventually filter your logs.

Ciao.

Giuseppe

Re: What are the configurations required to forward specific log messages to Splunk?

ssoftility — Mon, 01 Nov 2021 15:10:43 GMT

Hi ,Please find the answers below.

what architecture do you have: stand-alone or distributed? Stand-alone
which kind of logs are you speaking of (wineventlog, syslog, linux, firewall, etc...)? Application logs
do you want to find the word "ScanStatistics" searching in your logs or do you want to index only the logs containing this word? Yes, we want to index/forward logs which contains word "ScanStatistics".
how do you ingest your logs, or you're not able to ingest logs and this is your main question?No, we use splunk forwarder to ingest logs. But here I need specific configuartions required to forward only "Scanstatistics" logs to splunk terminating all other log
could you share some example of your logs? Yes
```
2021-10-28 01:31:47,535 - cm_scan_statistics.cm_scan_statistics - INFO - c0:a0:0d:06:5b:c7 - CmScanStatistics. CM(c0:a0:0d:06:5b:c7) fbc.status=SUCCESS
2021-10-28 01:31:47,535 - cm_scan_statistics.cm_scan_statistics - INFO - c0:a0:0d:06:5b:c7 - CmScanStatistics: updating cm(c0:a0:0d:06:5b:c7) for ['metadata.scans.success', 'metadata.scans.lastupdate', 'metadata.scans.lastevent']
```
In few words, I need to understand:
- the kind of logs you're speaking; Application logs
- if you need an help in searching the word in your logs or in ingesting and indexing logs.
  because:
  - knowing the kind of your logs I can suggest the best way to ingest and parse your logs;
  - if you're speaking of a search, I can hint the search to find the events that contain the above word;
  - if you're speaking of ingesting logs, I can hint how to ingest, parse and eventually filter your logs.

@gcusello

Re: What are the configurations required to forward specific log messages to Splunk?

gcusello — Mon, 01 Nov 2021 17:37:58 GMT

Hi @ssoftility,

if you want to filter your logs before indexing, you surely use less license for these logs but you have less logs for your searches: you cannot use the discarded logs.

Anyway, with the only exception of WindEventLogs, logs can be filtered only on indexers or (when present) on heavy Forwarders.

To do this, you have to follow the instructions at https://docs.splunk.com/Documentation/Splunk/8.2.3/Forwarding/Routeandfilterdatad#Keep_specific_events_and_discard_the_rest

in few word, on you Indexers you have to create a props.conf file containing:

[your_sourcetype] TRANSFORMS-set= setnull,setparsing

and a trandforms.conf like this:

[setnull] REGEX = . DEST_KEY = queue FORMAT = nullQueue [setparsing] REGEX = CmScanStatistics DEST_KEY = queue FORMAT = indexQueue

In this way you discard all logs except the ones containing "CmScanStatistics".

Remember to restart Splunk if you manually modify files.

Ciao.

Giuseppe