topic Re: How to send events to the nullQueue on indexer? in Getting Data In

How to send events to the nullQueue on indexer?

dingesbr — Fri, 04 Sep 2015 23:44:10 GMT

The strange thing is that I can send events to the nullQueue on my Local installation of Enterprise Splunk (6.2.2.5). Using the same app I created on the indexer for my Local Installation, I tried to use on our Production installation of Enterprise Splunk (6.2.2.2), and it did not work. The source type is a custom one we create on the forwarder, oracle:oag:trc, for the trace files.

Am I allowed to create an app on the Splunk Indexer that will send the events to the nullQueue? If, not, where is the correct place to put the "props.conf" and "transforms.conf" files so that the indexer will pick up the transforms during Indexing/Parsing?

My Local Topology is: Universal Forwarder => Splunk Instance

My Production Topology is: Universal Forwarder => Splunk Indexer/Deployer => Search Head

props.conf

[oracle:oag:trc]
TRANSFORMS-nullqueue-oag=nullqueue-oag-filter

transforms.conf

[nullqueue-oag-filter]
REGEX=^ERROR[\s\S]{0,100}(\bCardinality violation\b|\berror handling connection: peer disconnected unexpectedly\b)
DEST_KEY=queue
FORMAT=nullQueue

Re: How to send events to the nullQueue on indexer?

woodcock — Sat, 05 Sep 2015 03:58:53 GMT

You need to bounce the Splunk instances on all the Indexers and then it should work.

Re: How to send events to the nullQueue on indexer?

dingesbr — Tue, 08 Sep 2015 18:09:29 GMT

+Woodcock

Thanks for the reply. The indexer(s) were restarted but did not resolve the issue.

We ran the following command to verify that the transform was loaded:
splunk cmd btool transforms list --debug

It showed up in the list of transforms.

Any other ideas come to mind? I appreciate your help!

Re: How to send events to the nullQueue on indexer?

somesoni2 — Tue, 08 Sep 2015 18:46:23 GMT

Can you provide some sample events which you want to filter (to ensure the REGEX is correct)?

Re: How to send events to the nullQueue on indexer?

dingesbr — Tue, 08 Sep 2015 19:10:18 GMT

+somesoni2
Sure, thanks.

Event 1:
ERROR 26/Aug/2015:23:59:59.655 [b2db7700] java exception:
com.vordel.es.EntityStoreException: Cardinality violation - field [timezoneAware] should have 1 values
at com.vordel.es.Field.checkCardinality(Field.java:232)
at com.vordel.es.Field.setValues(Field.java:147)
at com.vordel.store.util.env.EnvExpandEntity.expandField(EnvExpandEntity.java:112)
at com.vordel.store.util.env.EnvExpandEntity.(EnvExpandEntity.java:39)
at com.vordel.store.util.env.EnvExpandEntityStore.getEntity(EnvExpandEntityStore.java:119)
at com.vordel.es.util.ShorthandKeyFinder$SingleEntityPF.end(ShorthandKeyFinder.java:155)
at com.vordel.es.util.ShorthandKeyFinder$PartFinder.process(ShorthandKeyFinder.java:91)
at com.vordel.es.util.ShorthandKeyFinder$SingleEntityPF.process(ShorthandKeyFinder.java:119)
at com.vordel.es.util.ShorthandKeyFinder.getEntity(ShorthandKeyFinder.java:262)
at com.vordel.es.util.ShorthandKeyFinder.getEntity(ShorthandKeyFinder.java:248)
at com.vordel.common.db.DbConnectionCache.getConnection(DbConnectionCache.java:49)
at com.vordel.common.db.JDBCDataSource.getConnection(JDBCDataSource.java:50)
at org.apache.openjpa.lib.jdbc.DelegatingDataSource.getConnection(DelegatingDataSource.java:110)
at org.apache.openjpa.lib.jdbc.DecoratingDataSource.getConnection(DecoratingDataSource.java:87)
at org.apache.openjpa.jdbc.kernel.JDBCStoreManager.connectInternal(JDBCStoreManager.java:982)
at org.apache.openjpa.jdbc.kernel.JDBCStoreManager.connect(JDBCStoreManager.java:967)
at org.apache.openjpa.jdbc.kernel.JDBCStoreManager.getConnection(JDBCStoreManager.java:240)
at org.apache.openjpa.jdbc.kernel.PreparedSQLStoreQuery$PreparedSQLExecutor.executeQuery(PreparedSQLStoreQuery.java:104)
at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:1005)
at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:863)
at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:794)
at org.apache.openjpa.kernel.DelegatingQuery.execute(DelegatingQuery.java:542)
at org.apache.openjpa.persistence.QueryImpl.execute(QueryImpl.java:286)
at org.apache.openjpa.persistence.QueryImpl.getResultList(QueryImpl.java:302)
at com.vordel.circuit.oauth.persistence.JpaAccessTokenStore.removeExpiredTokens(JpaAccessTokenStore.java:621)
at com.vordel.circuit.oauth.persistence.SynchronizedTokenStore.removeExpiredTokens(SynchronizedTokenStore.java:173)
at com.vordel.circuit.oauth.persistence.OauthLoadableCache$2.run(OauthLoadableCache.java:84)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)

Event 2:
ERROR 25/Aug/2015:18:02:33.449 [94754700] error handling connection: peer disconnected unexpectedly

Re: How to send events to the nullQueue on indexer?

somesoni2 — Tue, 08 Sep 2015 19:50:12 GMT

The REGEX is definitely correct. And since you're using Universal forwarder, the correct location for props/transforms is on Indexer. Now, let's check the event breaking is correct OR not as the event filter works at invidual event level. Do you have other attribute defined on the props.conf for the sourcetype?

Re: How to send events to the nullQueue on indexer?

dingesbr — Tue, 29 Sep 2020 07:14:14 GMT

This what was done on Production. On my Local Installation, previously the forwarder had all the details except the transform, but I changed it to the below but that still worked Locally.

The input in coming from TCP (from the forwarder).

For Production, the below is applied to both the indexer and the forwarder:

Re: How to send events to the nullQueue on indexer?

somesoni2 — Tue, 08 Sep 2015 21:33:20 GMT

We if you're using Universal forwarders, they can't do event parsing so props/transform their is dead code. They have to be there on Indexer OR heavy forwarders. Which location you're putting your props/transforms etc/system OR etc/apps/AppName?

Re: How to send events to the nullQueue on indexer?

dingesbr — Tue, 08 Sep 2015 21:43:30 GMT

"/etc/apps/AppName" is where the props.conf and transforms.conf is being placed on the indexer.

Re: How to send events to the nullQueue on indexer?

dingesbr — Thu, 10 Sep 2015 17:54:32 GMT

somesoni, I appreciate the time you spent on this. Can you confirm that "/etc/apps/AppName" is a correct location for a transform.conf? Some of the documentation doesn't mention "/etc/app/AppName" for transforms during index time? I am not real clear on what gets dedicated to search time versus index time. Again, I appreciate your assistance!

Re: How to send events to the nullQueue on indexer?

dingesbr — Sat, 19 Sep 2015 18:47:04 GMT

Turns out a buggy version of the REGEX was deployey for the transform. We're noobs! Thanks everyone for their support!

We corrected our transforms in "/etc/apps/myapp/transform.conf"

We ran "https://myindexer:myport/en-US/debug/refresh" to get it loaded and verify via command line "splunk cmd btool transforms list --debug".

That fixed the issue for us.

Re: How to send events to the nullQueue on indexer?

neelamssantosh — Sat, 10 Oct 2015 12:49:16 GMT

Same issue.

Kindly provide the steps, if it has been resolved