topic Re: Nullqueue filtering in Getting Data In

Nullqueue filtering

timrich66 — Wed, 03 Feb 2021 10:06:02 GMT

Hello All,

Grateful for assistance on this one.

We have several areas where servers are HA pairs and write to a server specific log. However, because they are an HA pair, their own log and the equivalent log on the paired server is visible via a shared drive.

Thus, server 'A' produces 'serverlogA' but can also see 'serverlogB'. Server 'B' produces 'serverlogB' and can also see 'serverlogA'.

Because both servers are in the same Server Class, we end up with duplicated events from both server logs.

We cannot only ingest from one server because they also have unique log files and a failure on the ingesting server would require manual intervention to move to the paired server.

I have tried to nullqueue the events as shown below, but not had any success.

Please let me know your thoughts on how to work around this issue.

Thanks

props.conf

[source::/apps/lvservices/mnt/logs/abc/whyluaap182_ContractEnquiry.log]
TRANSFORMS-nullq_cms_uaap181 = nullq_uaap181
[source::/apps/lvservices/mnt/logs/abc/whyluaap181_ContractEnquiry.log]
TRANSFORMS-nullq_cms_uaap182 = nullq_uaap182

transforms.conf

[nullq_uaap181]
SOURCE_KEY = MetaData:Host
REGEX = whyluaap181
DEST_KEY = queue
FORMAT = nullQueue

[nullq_uaap182]
SOURCE_KEY = MetaData:Host
REGEX = whyluaap182
DEST_KEY = queue
FORMAT = nullQueue

Re: Nullqueue filtering

scelikok — Wed, 03 Feb 2021 19:41:55 GMT

Hi @timrich66,

These conf files should be on Indexers. Could you please confirm?

Re: Nullqueue filtering

timrich66 — Thu, 11 Feb 2021 16:06:38 GMT

Hi @scelikok , yes, the code is deployed from our Cluster Master (/opt/splunk/etc/master-apps/_cluster/local) to indexers (/opt/splunk/etc/slave-apps/_cluster/local). It is also present on HF in /opt/splunk/etc/system/local).

Re: Nullqueue filtering

isoutamo — Thu, 11 Feb 2021 16:22:53 GMT

Is it totally impossible idea to have two almost identical serverclasses one for each?

Re: Nullqueue filtering

timrich66 — Thu, 11 Feb 2021 16:58:08 GMT

Hi, @isoutamo Can you explain your thinking? It is unlikely, but may be possible. I don't understand why the nullqueue option isn't working though.

Re: Nullqueue filtering

isoutamo — Thu, 11 Feb 2021 17:20:29 GMT

Now you have one serverclass where you have defined all those files: common, nodeA and nodeB if I understood right. As your log files contains name of nodeA or nodeB you could remove those from common serverclass and create separate classes for those which are installed only to corresponding node. All common files could be in common serverclass which will installed to both. Then there is no need for that nullQfiltering.