Is it possible to only record or see critical logs and not every single log reported?
Yes. You have several options:
Filter event data and send to queues[edit]
Although similar to forwarder-based routing, queue routing can be performed by an indexer, as well as a heavy forwarder. It does not use the outputs.conf file, just props.conf and transforms.conf.
You can eliminate unwanted data by routing it to nullQueue, the Splunk Enterprise equivalent of the /dev/null device. When you filter out data in this way, the filtered data is not forwarded or added to the Splunk Enterprise index at all, and doesn't count toward your indexing volume.
Note: See "Caveats for routing and filtering structured data" later in this topic.
Discard specific events and keep the rest[edit]
This example discards all sshd events in /var/log/messages by sending them to nullQueue:
[source::/var/log/messages]
TRANSFORMS-null= setnull
2. Create a corresponding stanza in transforms.conf. Set DEST_KEY to "queue" and FORMAT to "nullQueue":
[setnull]
REGEX = [sshd]
DEST_KEY = queue
FORMAT = nullQueue
That does it.
Keep specific events and discard the rest[edit]
Here's the opposite scenario. In this example, you use two transforms to keep only the sshd events. One transform routes sshd events to indexQueue, while another routes all other events to nullQueue.
Note: In this example, the order of the transforms in props.conf matters. The null queue transform must come first; if it comes later, it will invalidate the previous transform and route all events to the null queue.
[source::/var/log/messages]
TRANSFORMS-set= setnull,setparsing
2. In transforms.conf:
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[setparsing]
REGEX = [sshd]
DEST_KEY = queue
FORMAT = indexQueue
No sure what you asking. Logging is up to your developer or software vendor. You can use Transforms to filter which data you want to index, but you have know exactly what events you want to send to null.
Yes. You have several options:
Filter event data and send to queues[edit]
Although similar to forwarder-based routing, queue routing can be performed by an indexer, as well as a heavy forwarder. It does not use the outputs.conf file, just props.conf and transforms.conf.
You can eliminate unwanted data by routing it to nullQueue, the Splunk Enterprise equivalent of the /dev/null device. When you filter out data in this way, the filtered data is not forwarded or added to the Splunk Enterprise index at all, and doesn't count toward your indexing volume.
Note: See "Caveats for routing and filtering structured data" later in this topic.
Discard specific events and keep the rest[edit]
This example discards all sshd events in /var/log/messages by sending them to nullQueue:
[source::/var/log/messages]
TRANSFORMS-null= setnull
2. Create a corresponding stanza in transforms.conf. Set DEST_KEY to "queue" and FORMAT to "nullQueue":
[setnull]
REGEX = [sshd]
DEST_KEY = queue
FORMAT = nullQueue
That does it.
Keep specific events and discard the rest[edit]
Here's the opposite scenario. In this example, you use two transforms to keep only the sshd events. One transform routes sshd events to indexQueue, while another routes all other events to nullQueue.
Note: In this example, the order of the transforms in props.conf matters. The null queue transform must come first; if it comes later, it will invalidate the previous transform and route all events to the null queue.
[source::/var/log/messages]
TRANSFORMS-set= setnull,setparsing
2. In transforms.conf:
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[setparsing]
REGEX = [sshd]
DEST_KEY = queue
FORMAT = indexQueue
Here is an example of filtering out events that include the string 'Teardown'
12:26:35.000 PM 2015-08-10T12:26:35-04:00 host-706-zz-ASA5520-A : %ASA-6-305012: Teardown dynamic UDP translation from inside:10.1.1.1/51971 to outside:x.x.x.x/51971 duration 0:00:31
on your heavy forwarder:
$SPLUNK_HOME/etc/system/local
props.conf
[host::xyz]
TRANSFORMS-null = nullteardowns
transforms.conf
[nullteardowns]
REGEX = Teardown
DEST_KEY = queue
FORMAT = nullQueue
restart heavy forwarder after the change.
Note: if this is syslog data , and you are filtering on host , use the host name of the syslog server itself as this will be processed before the actual host name of the sending device is assigned.