Knowledge Management

Is it possible to only record or see critical logs and not every single log reported?

jboike
Explorer

Is it possible to only record or see critical logs and not every single log reported?

0 Karma
1 Solution

rphillips_splk
Splunk Employee
Splunk Employee

Yes. You have several options:

  • if using syslog you can set your logging level appropriately on the sending device
  • if monitoring a log file you can use props & transforms to filter events: (Note: you will need to run a full instance of Splunk Enterprise on your host you are monitoring log files on, not a Universal forwarder)

Filter event data and send to queues[edit]
Although similar to forwarder-based routing, queue routing can be performed by an indexer, as well as a heavy forwarder. It does not use the outputs.conf file, just props.conf and transforms.conf.

You can eliminate unwanted data by routing it to nullQueue, the Splunk Enterprise equivalent of the /dev/null device. When you filter out data in this way, the filtered data is not forwarded or added to the Splunk Enterprise index at all, and doesn't count toward your indexing volume.

Note: See "Caveats for routing and filtering structured data" later in this topic.

Discard specific events and keep the rest[edit]
This example discards all sshd events in /var/log/messages by sending them to nullQueue:

  1. In props.conf, set the TRANSFORMS-null attribute:

[source::/var/log/messages]
TRANSFORMS-null= setnull
2. Create a corresponding stanza in transforms.conf. Set DEST_KEY to "queue" and FORMAT to "nullQueue":

[setnull]
REGEX = [sshd]
DEST_KEY = queue
FORMAT = nullQueue
That does it.

Keep specific events and discard the rest[edit]
Here's the opposite scenario. In this example, you use two transforms to keep only the sshd events. One transform routes sshd events to indexQueue, while another routes all other events to nullQueue.

Note: In this example, the order of the transforms in props.conf matters. The null queue transform must come first; if it comes later, it will invalidate the previous transform and route all events to the null queue.

  1. In props.conf:

[source::/var/log/messages]
TRANSFORMS-set= setnull,setparsing
2. In transforms.conf:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = [sshd]
DEST_KEY = queue
FORMAT = indexQueue

http://docs.splunk.com/Documentation/Splunk/6.2.5/Forwarding/Routeandfilterdatad#Filter_event_data_a...

View solution in original post

bmacias84
Champion

No sure what you asking. Logging is up to your developer or software vendor. You can use Transforms to filter which data you want to index, but you have know exactly what events you want to send to null.

0 Karma

rphillips_splk
Splunk Employee
Splunk Employee

Yes. You have several options:

  • if using syslog you can set your logging level appropriately on the sending device
  • if monitoring a log file you can use props & transforms to filter events: (Note: you will need to run a full instance of Splunk Enterprise on your host you are monitoring log files on, not a Universal forwarder)

Filter event data and send to queues[edit]
Although similar to forwarder-based routing, queue routing can be performed by an indexer, as well as a heavy forwarder. It does not use the outputs.conf file, just props.conf and transforms.conf.

You can eliminate unwanted data by routing it to nullQueue, the Splunk Enterprise equivalent of the /dev/null device. When you filter out data in this way, the filtered data is not forwarded or added to the Splunk Enterprise index at all, and doesn't count toward your indexing volume.

Note: See "Caveats for routing and filtering structured data" later in this topic.

Discard specific events and keep the rest[edit]
This example discards all sshd events in /var/log/messages by sending them to nullQueue:

  1. In props.conf, set the TRANSFORMS-null attribute:

[source::/var/log/messages]
TRANSFORMS-null= setnull
2. Create a corresponding stanza in transforms.conf. Set DEST_KEY to "queue" and FORMAT to "nullQueue":

[setnull]
REGEX = [sshd]
DEST_KEY = queue
FORMAT = nullQueue
That does it.

Keep specific events and discard the rest[edit]
Here's the opposite scenario. In this example, you use two transforms to keep only the sshd events. One transform routes sshd events to indexQueue, while another routes all other events to nullQueue.

Note: In this example, the order of the transforms in props.conf matters. The null queue transform must come first; if it comes later, it will invalidate the previous transform and route all events to the null queue.

  1. In props.conf:

[source::/var/log/messages]
TRANSFORMS-set= setnull,setparsing
2. In transforms.conf:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = [sshd]
DEST_KEY = queue
FORMAT = indexQueue

http://docs.splunk.com/Documentation/Splunk/6.2.5/Forwarding/Routeandfilterdatad#Filter_event_data_a...

rphillips_splk
Splunk Employee
Splunk Employee

Here is an example of filtering out events that include the string 'Teardown'

12:26:35.000 PM 2015-08-10T12:26:35-04:00 host-706-zz-ASA5520-A : %ASA-6-305012: Teardown dynamic UDP translation from inside:10.1.1.1/51971 to outside:x.x.x.x/51971 duration 0:00:31

on your heavy forwarder:
$SPLUNK_HOME/etc/system/local

props.conf
[host::xyz]
TRANSFORMS-null = nullteardowns

transforms.conf
[nullteardowns]
REGEX = Teardown
DEST_KEY = queue
FORMAT = nullQueue

restart heavy forwarder after the change.

Note: if this is syslog data , and you are filtering on host , use the host name of the syslog server itself as this will be processed before the actual host name of the sending device is assigned.

Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...