Getting Data In

What are the configurations required to forward specific log messages to Splunk?

ssoftility
Loves-to-Learn

What are the configurations required to forward specific log messages to Splunk.

Every  log message that contains "ScanStatistics" this phrase needs to get forwarded to Splunk.
Let us know what are the configurations to be done.

0 Karma

gcusello
Legend

Hi @ssoftility,

some additional information, please:

  • what architecture do you have: stand-alone or distributed?
  • which kind of logs are you speaking of (wineventlog, syslog, linux, firewall, etc...)?
  • do you want to find the word "ScanStatistics" searching in your logs or do you want to index only the logs containing this word?
  • how do you ingest your logs, or you're not able to ingest logs and this is your main question?
  • could you share some example of your logs?

In few words, I need to understand:

  • the kind of logs you're speaking;
  • if you need an help in searching the word in your logs or in ingesting and indexing logs.

because:

  • knowing the kind of your logs I can suggest the best way to ingest and parse your logs;
  • if you're speaking of a search, I can hint the search to find the events that contain the above word;
  • if you're speaking of ingesting logs, I can hint how to ingest, parse and eventually filter your logs.

Ciao.

Giuseppe

 

ssoftility
Loves-to-Learn

Hi  ,Please find the answers below.

  • what architecture do you have: stand-alone or distributed? Stand-alone
  • which kind of logs are you speaking of (wineventlog, syslog, linux, firewall, etc...)? Application logs
  • do you want to find the word "ScanStatistics" searching in your logs or do you want to index only the logs containing this word? Yes, we want to index/forward logs which contains word "ScanStatistics".
  • how do you ingest your logs, or you're not able to ingest logs and this is your main question?No, we use splunk forwarder to ingest logs. But here I need specific configuartions required to forward only "Scanstatistics" logs to splunk terminating all other log
  • could you share some example of your logs? Yes
    2021-10-28 01:31:47,535 - cm_scan_statistics.cm_scan_statistics - INFO - c0:a0:0d:06:5b:c7 - CmScanStatistics. CM(c0:a0:0d:06:5b:c7) fbc.status=SUCCESS
    2021-10-28 01:31:47,535 - cm_scan_statistics.cm_scan_statistics - INFO - c0:a0:0d:06:5b:c7 - CmScanStatistics: updating cm(c0:a0:0d:06:5b:c7) for ['metadata.scans.success', 'metadata.scans.lastupdate', 'metadata.scans.lastevent']

    In few words, I need to understand:

    • the kind of logs you're speaking; Application logs
    • if you need an help in searching the word in your logs or in ingesting and indexing logs.

      because:

      • knowing the kind of your logs I can suggest the best way to ingest and parse your logs;
      • if you're speaking of a search, I can hint the search to find the events that contain the above word;
      • if you're speaking of ingesting logs, I can hint how to ingest, parse and eventually filter your logs.

@gcusello

0 Karma

gcusello
Legend

Hi @ssoftility,

if you want to filter your logs before indexing, you surely use less license for these logs but you have less logs for your searches: you cannot use the discarded logs.

Anyway, with the only exception of WindEventLogs, logs can be filtered only on indexers or (when present) on heavy Forwarders.

To do this, you have to follow the instructions at https://docs.splunk.com/Documentation/Splunk/8.2.3/Forwarding/Routeandfilterdatad#Keep_specific_even...

in few word, on you Indexers you have to create a props.conf file containing:

[your_sourcetype]
TRANSFORMS-set= setnull,setparsing

and a trandforms.conf like this:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = CmScanStatistics
DEST_KEY = queue
FORMAT = indexQueue

In this way you discard all logs except the ones containing "CmScanStatistics".

Remember to restart Splunk if you manually modify files.

Ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...