What are the configurations required to forward specific log messages to Splunk?

[ssoftility]

What are the configurations required to forward specific log messages to Splunk.

Every  log message that contains "ScanStatistics" this phrase needs to get forwarded to Splunk.
Let us know what are the configurations to be done.

[gcusello]

Hi @ssoftility,

some additional information, please:

• what architecture do you have: stand-alone or distributed?
• which kind of logs are you speaking of (wineventlog, syslog, linux, firewall, etc...)?
• do you want to find the word "ScanStatistics" searching in your logs or do you want to index only the logs containing this word?
• how do you ingest your logs, or you're not able to ingest logs and this is your main question?
• could you share some example of your logs?

In few words, I need to understand:

• the kind of logs you're speaking;
• if you need an help in searching the word in your logs or in ingesting and indexing logs.

because:

• knowing the kind of your logs I can suggest the best way to ingest and parse your logs;
• if you're speaking of a search, I can hint the search to find the events that contain the above word;
• if you're speaking of ingesting logs, I can hint how to ingest, parse and eventually filter your logs.

Ciao.

Giuseppe

[ssoftility]

Hi  ,Please find the answers below.

• what architecture do you have: stand-alone or distributed? Stand-alone
• which kind of logs are you speaking of (wineventlog, syslog, linux, firewall, etc...)? Application logs
• do you want to find the word "ScanStatistics" searching in your logs or do you want to index only the logs containing this word? Yes, we want to index/forward logs which contains word "ScanStatistics".
• how do you ingest your logs, or you're not able to ingest logs and this is your main question?No, we use splunk forwarder to ingest logs. But here I need specific configuartions required to forward only "Scanstatistics" logs to splunk terminating all other log
• could you share some example of your logs? Yes

  2021-10-28 01:31:47,535 - cm_scan_statistics.cm_scan_statistics - INFO - c0:a0:0d:06:5b:c7 - CmScanStatistics. CM(c0:a0:0d:06:5b:c7) fbc.status=SUCCESS
  2021-10-28 01:31:47,535 - cm_scan_statistics.cm_scan_statistics - INFO - c0:a0:0d:06:5b:c7 - CmScanStatistics: updating cm(c0:a0:0d:06:5b:c7) for ['metadata.scans.success', 'metadata.scans.lastupdate', 'metadata.scans.lastevent']

  In few words, I need to understand:

  • the kind of logs you're speaking; Application logs
  • if you need an help in searching the word in your logs or in ingesting and indexing logs.

    because:

    • knowing the kind of your logs I can suggest the best way to ingest and parse your logs;
    • if you're speaking of a search, I can hint the search to find the events that contain the above word;
    • if you're speaking of ingesting logs, I can hint how to ingest, parse and eventually filter your logs.

@gcusello

[gcusello]

Hi @ssoftility,

if you want to filter your logs before indexing, you surely use less license for these logs but you have less logs for your searches: you cannot use the discarded logs.

Anyway, with the only exception of WindEventLogs, logs can be filtered only on indexers or (when present) on heavy Forwarders.

To do this, you have to follow the instructions at https://docs.splunk.com/Documentation/Splunk/8.2.3/Forwarding/Routeandfilterdatad#Keep_specific_even...

in few word, on you Indexers you have to create a props.conf file containing:

[your_sourcetype]
TRANSFORMS-set= setnull,setparsing

and a trandforms.conf like this:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = CmScanStatistics
DEST_KEY = queue
FORMAT = indexQueue

In this way you discard all logs except the ones containing "CmScanStatistics".

Remember to restart Splunk if you manually modify files.

Ciao.

Giuseppe