How do I filter (using regex) only particular lines from the log files?

[ApurvaB]

I am using Splunk forwarder to receive log files from multiple monitors. I need to filter events, based on a regex, from one particular monitor. Is there a way I can do that?
I cannot use the RegEx in the Search filed of Splunk Web as it will apply it to the events coming from all the monitors.
To be specific, my inputs.conf file has following monitors:

[monitor:////Monitor1]
disabled = false
followTail = true
index = myIndex
whitelist = .log$

[monitor:////Monitor2]
disabled = false
followTail = true
index = myIndex
whitelist = .log$

Files from Monitor1 contain lines that begin with [NOTE/WARN/ERROR] and other lines that don't. I am only interested in the lines that being with [NOTE/WARN/ERROR] and want to filter out the other lines so that they don't appear in the search results on Splunk Web.

Is there a way I can achieve this?

[the_wolverine]

Assuming you are using a light (or universal) forwarder, you will need to use props.conf and transforms.conf set on your indexer (Splunk server) to parse the data.

• props.conf

[source::.../Monitor1] 
TRANSFORMS-route = nullqueue,notewarnerror

• transforms.conf

[notewarnerror]
REGEX = ^(NOTE|WARN|ERROR)
DEST_KEY = _MetaData:Index
FORMAT = myIndex

[nullqueue]
REGEX=.
DEST_KEY=queue
FORMAT=nullQueue

If you are using a heavy forwarder, these would need to be set on the heavy forwarder.

[ApurvaB]

Thanks for your response. Each if my event contains a line count of about 40-50. If I use ^(NOTE|ERROR|WARN) it gives me only the first line of every log file.
Do I need to use a multi-line character in my regex? Something like (?m)^(NOTE|WARN|ERROR)
But this doesn't seem to filter correctly for me. Do you know any reason why this might happen?

[tom_frotscher]

Hi!

You can filter out some events by using regular expressions. This is possible in the transforms.conf. For example you can send all events that match a particular regular expression to the index queue, and all that don't match are sent to the null queue, which means they will be discarded.

For proper configuration information, take a look at link

I am not sure if it is applicable to a specific monitor, but i think you can identify/establish a unique source or sourcetype for all events from this monitor.