topic Re: How do you exclude all lines with INFO or WARN from being indexed? in Getting Data In

How do you exclude all lines with INFO or WARN from being indexed?

nls7010 — Thu, 28 Feb 2019 15:54:50 GMT

I have been reading through a lot of the previous answers to exclusion, but none match what I need. I need to exclude all INFO and WARN lines from one of my indexes, so that they are never processed. Only the ERROR lines should be processed.

I have this so far, but I'm not certain of a couple of things. One what should I have in the props file to complete it and two, do I need the in the transforms.conf file:

PROPS.CONF

TRANSFORMS-set = setnull, setparsing

TRANSFORMS.CONF

<code>[setnull]
REGEX = INFO, WARN
DEST_KEY = queue
FORMAT = nullQueue
</code>

Re: How do you exclude all lines with INFO or WARN from being indexed?

nls7010 — Thu, 28 Feb 2019 15:57:02 GMT

There were some code lines surrounding the transforms lines, those were what I was asking about

Re: How do you exclude all lines with INFO or WARN from being indexed?

lakshman239 — Thu, 28 Feb 2019 16:01:11 GMT

You need configs in both props.conf and transforms.conf. The following sends events with only ERROR to your indexer.

PROPS.CONF
[your_sourcetype]
TRANSFORMS-set = setnull, setparsing

TRANSFORMS.CONF

[setnull] 
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue 

[setparsing]
REGEX = ERROR
DEST_KEY = queue
FORMAT = indexQueue

Re: How do you exclude all lines with INFO or WARN from being indexed?

nls7010 — Thu, 28 Feb 2019 16:08:00 GMT

What if there are multiple source types? I wanted to exclude those lines for the entire index which was created for this customer and they have multiple sourcetypes.

Re: How do you exclude all lines with INFO or WARN from being indexed?

lakshman239 — Thu, 28 Feb 2019 16:21:07 GMT

for each sourcetype, you need to have the same settings

[your_sourcetype1]
TRANSFORMS-set = setnull, setparsing

[your_souretype2]
TRANSFORMS-set = setnull, setparsing

However, you just need one setting for transforms, as provided earlier.

After updating, you need to restart your indexer, so any new data will have these filtering applied.

Re: How do you exclude all lines with INFO or WARN from being indexed?

nls7010 — Thu, 28 Feb 2019 17:16:58 GMT

So, If I'm understanding this correctly, I would do something like the following:
PROPS.CONF

[value1-logs]
TRANSFORMS-set = setnull, setparsing

[value2-logs]
TRANSFORMS-set = setnull, setparsing

TRANSFORMS.CONF

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = ERROR
DEST_KEY = queue
FORMAT = indexQueue

So that only lines with ERROR will show?

Re: How do you exclude all lines with INFO or WARN from being indexed?

lakshman239 — Thu, 28 Feb 2019 17:18:58 GMT

yes correct

Re: How do you exclude all lines with INFO or WARN from being indexed?

nls7010 — Thu, 28 Feb 2019 18:57:54 GMT

quick question. the FORMAT key, should that be the name of the actual index in the transforms.conf file? Or it that literal? Asking because I still see the WARN and INFO logs coming in after pushing out the following:
[my1-logs]
TRANSFORMS-set = setnull, setparsing

[my2-logs]
TRANSFORMS-set = setnull, setparsing

TRANSFORMS.CONF

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = ERROR
DEST_KEY = queue
FORMAT = indexQueue

Re: How do you exclude all lines with INFO or WARN from being indexed?

lakshman239 — Thu, 28 Feb 2019 19:10:56 GMT

its a literal. Did you restart your indexer after the changes?

Re: How do you exclude all lines with INFO or WARN from being indexed?

nls7010 — Thu, 28 Feb 2019 19:22:38 GMT

we have clustered indexers and I pushed it our via the cluster master. I believe it's supposed to restart them all

Re: How do you exclude all lines with INFO or WARN from being indexed?

nls7010 — Thu, 28 Feb 2019 21:43:37 GMT

They have all been restarted, but it's still sending the INFO and WARN messages 😞

Re: How do you exclude all lines with INFO or WARN from being indexed?

nls7010 — Fri, 01 Mar 2019 14:27:23 GMT

Yes, I did a rolling restart, but am still seeing INFO and WARN which I wanted to block out and only get ERROR.

Re: How do you exclude all lines with INFO or WARN from being indexed?

nls7010 — Fri, 01 Mar 2019 14:44:34 GMT

I just realized I asked that comment on the wrong value. It's the REGEX= . Is that supposed to have a value of INFO,WARN?

Re: How do you exclude all lines with INFO or WARN from being indexed?

lakshman239 — Fri, 01 Mar 2019 14:46:46 GMT

can you test the config in your dev with the above config?

Re: How do you exclude all lines with INFO or WARN from being indexed?

nls7010 — Fri, 01 Mar 2019 15:02:54 GMT

I am testing in my development environment and I get the same issue. Nothing gets parched out, it's all still coming into the logs. I will be working with my client in a little while to ensure the props and transforms made it to the servers (they are containers).

Re: How do you exclude all lines with INFO or WARN from being indexed?

lakshman239 — Fri, 01 Mar 2019 15:16:25 GMT

what do you want? do you want INFO and WARN OR ERROR? you can adjust them as per your needs.

# This sends all events to be ignored
[setnull] 
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

# this says ignore all events, except the ones containing ERROR
[setparsing]
REGEX = ERROR
DEST_KEY = queue
FORMAT = indexQueue

Re: How do you exclude all lines with INFO or WARN from being indexed?

nls7010 — Fri, 01 Mar 2019 15:29:14 GMT

I just want the ERROR lines. So I would want to eliminate INFO, WARN, etc that are not ERROR related.

Re: How do you exclude all lines with INFO or WARN from being indexed?

nls7010 — Fri, 01 Mar 2019 15:31:09 GMT

ok, I understand, so now I'll need to work with the client to see if everything is getting to the forwarders as needed. I did verify that the props and transform files did make it to all the indexers and the indexer cluster members were all restarted.