Nullqueue filtering

[timrich66]

Hello All,

Grateful for assistance on this one.

We have several areas where servers are HA pairs and write to a server specific log.  However, because they are an HA pair, their own log and the equivalent log on the paired server is visible via a shared drive.

Thus,  server 'A' produces 'serverlogA' but can also see 'serverlogB'.  Server 'B' produces 'serverlogB' and can also see 'serverlogA'.

Because both servers are in the same Server Class, we end up with duplicated events from both server logs.

We cannot only ingest from one server because they also have unique log files and a failure on the ingesting server would require manual intervention to move to the paired server.

I have tried to nullqueue the events as shown below, but not had any success.  

Please let me know your thoughts on how to work around this issue.

Thanks

props.conf

[source::/apps/lvservices/mnt/logs/abc/whyluaap182_ContractEnquiry.log]
TRANSFORMS-nullq_cms_uaap181 = nullq_uaap181
[source::/apps/lvservices/mnt/logs/abc/whyluaap181_ContractEnquiry.log]
TRANSFORMS-nullq_cms_uaap182 = nullq_uaap182

transforms.conf

[nullq_uaap181]
SOURCE_KEY = MetaData:Host
REGEX = whyluaap181
DEST_KEY = queue
FORMAT = nullQueue

[nullq_uaap182]
SOURCE_KEY = MetaData:Host
REGEX = whyluaap182
DEST_KEY = queue
FORMAT = nullQueue

[isoutamo]

Is it totally impossible idea to have two almost identical serverclasses one for each?

[timrich66]

Hi, @isoutamo  Can you explain your thinking?  It is unlikely, but may be possible.  I don't understand why the nullqueue option isn't working though.

[isoutamo]

Now you have one serverclass where you have defined all those files: common, nodeA and nodeB if I understood right. As your log files contains name of nodeA or nodeB you could remove those from common serverclass and create separate classes for those which are installed only to corresponding node. All common files could be in common serverclass which will installed to both. Then there is no need for that nullQfiltering. 

[scelikok]

Hi @timrich66,

These conf files should be on Indexers. Could you please confirm?

[timrich66]

Hi @scelikok , yes, the code is deployed from our Cluster Master (/opt/splunk/etc/master-apps/_cluster/local) to indexers (/opt/splunk/etc/slave-apps/_cluster/local).  It is also present on HF in /opt/splunk/etc/system/local).