How to send events to the nullQueue on indexer?

[dingesbr]

The strange thing is that I can send events to the nullQueue on my Local installation of Enterprise Splunk (6.2.2.5). Using the same app I created on the indexer for my Local Installation, I tried to use on our Production installation of Enterprise Splunk (6.2.2.2), and it did not work. The source type is a custom one we create on the forwarder, oracle:oag:trc, for the trace files.

Am I allowed to create an app on the Splunk Indexer that will send the events to the nullQueue? If, not, where is the correct place to put the "props.conf" and "transforms.conf" files so that the indexer will pick up the transforms during Indexing/Parsing?

My Local Topology is: Universal Forwarder => Splunk Instance

My Production Topology is: Universal Forwarder => Splunk Indexer/Deployer => Search Head

props.conf

[oracle:oag:trc]
TRANSFORMS-nullqueue-oag=nullqueue-oag-filter

transforms.conf

[nullqueue-oag-filter]
REGEX=^ERROR[\s\S]{0,100}(\bCardinality violation\b|\berror handling connection: peer disconnected unexpectedly\b)
DEST_KEY=queue
FORMAT=nullQueue

[dingesbr · Accepted solution]

Turns out a buggy version of the REGEX was deployey for the transform. We're noobs! Thanks everyone for their support!

We corrected our transforms in "/etc/apps/myapp/transform.conf"

We ran "https://myindexer:myport/en-US/debug/refresh" to get it loaded and verify via command line "splunk cmd btool transforms list --debug".

That fixed the issue for us.

[somesoni2]

The REGEX is definitely correct. And since you're using Universal forwarder, the correct location for props/transforms is on Indexer. Now, let's check the event breaking is correct OR not as the event filter works at invidual event level. Do you have other attribute defined on the props.conf for the sourcetype?

[dingesbr]

This what was done on Production. On my Local Installation, previously the forwarder had all the details except the transform, but I changed it to the below but that still worked Locally.

The input in coming from TCP (from the forwarder).

For Production, the below is applied to both the indexer and the forwarder:

[oracle:oag:trc]
TIME_PREFIX=^(ERROR|DEBUG|INFO|DATA|FATAL)\s+
TIME_FORMAT=%d/%b/%Y:%H:%M:%S.%3N
MAX_TIMESTAMP_LOOKAHEAD=35
SHOULD_LINEMERGE=True
BREAK_ONLY_BEFORE=^(ERROR|DEBUG|INFO|DATA|FATAL)
TRUNCATE=999999
TRANSFORMS-nullqueue-oag=nullqueue-oag-filter

[somesoni2]

We if you're using Universal forwarders, they can't do event parsing so props/transform their is dead code. They have to be there on Indexer OR heavy forwarders. Which location you're putting your props/transforms etc/system OR etc/apps/AppName?

[dingesbr]

somesoni, I appreciate the time you spent on this. Can you confirm that "/etc/apps/AppName" is a correct location for a transform.conf? Some of the documentation doesn't mention "/etc/app/AppName" for transforms during index time? I am not real clear on what gets dedicated to search time versus index time. Again, I appreciate your assistance!

[dingesbr]

+1

"/etc/apps/AppName" is where the props.conf and transforms.conf is being placed on the indexer.

[somesoni2]

Can you provide some sample events which you want to filter (to ensure the REGEX is correct)?

[dingesbr]

+somesoni2
Sure, thanks.

Event 1:
ERROR 26/Aug/2015:23:59:59.655 [b2db7700] java exception:
com.vordel.es.EntityStoreException: Cardinality violation - field [timezoneAware] should have 1 values
at com.vordel.es.Field.checkCardinality(Field.java:232)
at com.vordel.es.Field.setValues(Field.java:147)
at com.vordel.store.util.env.EnvExpandEntity.expandField(EnvExpandEntity.java:112)
at com.vordel.store.util.env.EnvExpandEntity.(EnvExpandEntity.java:39)
at com.vordel.store.util.env.EnvExpandEntityStore.getEntity(EnvExpandEntityStore.java:119)
at com.vordel.es.util.ShorthandKeyFinder$SingleEntityPF.end(ShorthandKeyFinder.java:155)
at com.vordel.es.util.ShorthandKeyFinder$PartFinder.process(ShorthandKeyFinder.java:91)
at com.vordel.es.util.ShorthandKeyFinder$SingleEntityPF.process(ShorthandKeyFinder.java:119)
at com.vordel.es.util.ShorthandKeyFinder.getEntity(ShorthandKeyFinder.java:262)
at com.vordel.es.util.ShorthandKeyFinder.getEntity(ShorthandKeyFinder.java:248)
at com.vordel.common.db.DbConnectionCache.getConnection(DbConnectionCache.java:49)
at com.vordel.common.db.JDBCDataSource.getConnection(JDBCDataSource.java:50)
at org.apache.openjpa.lib.jdbc.DelegatingDataSource.getConnection(DelegatingDataSource.java:110)
at org.apache.openjpa.lib.jdbc.DecoratingDataSource.getConnection(DecoratingDataSource.java:87)
at org.apache.openjpa.jdbc.kernel.JDBCStoreManager.connectInternal(JDBCStoreManager.java:982)
at org.apache.openjpa.jdbc.kernel.JDBCStoreManager.connect(JDBCStoreManager.java:967)
at org.apache.openjpa.jdbc.kernel.JDBCStoreManager.getConnection(JDBCStoreManager.java:240)
at org.apache.openjpa.jdbc.kernel.PreparedSQLStoreQuery$PreparedSQLExecutor.executeQuery(PreparedSQLStoreQuery.java:104)
at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:1005)
at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:863)
at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:794)
at org.apache.openjpa.kernel.DelegatingQuery.execute(DelegatingQuery.java:542)
at org.apache.openjpa.persistence.QueryImpl.execute(QueryImpl.java:286)
at org.apache.openjpa.persistence.QueryImpl.getResultList(QueryImpl.java:302)
at com.vordel.circuit.oauth.persistence.JpaAccessTokenStore.removeExpiredTokens(JpaAccessTokenStore.java:621)
at com.vordel.circuit.oauth.persistence.SynchronizedTokenStore.removeExpiredTokens(SynchronizedTokenStore.java:173)
at com.vordel.circuit.oauth.persistence.OauthLoadableCache$2.run(OauthLoadableCache.java:84)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)

Event 2:
ERROR 25/Aug/2015:18:02:33.449 [94754700] error handling connection: peer disconnected unexpectedly

[woodcock]

You need to bounce the Splunk instances on all the Indexers and then it should work.

[dingesbr]

+Woodcock

Thanks for the reply. The indexer(s) were restarted but did not resolve the issue.

We ran the following command to verify that the transform was loaded:
splunk cmd btool transforms list --debug

It showed up in the list of transforms.

Any other ideas come to mind? I appreciate your help!

[neelamssantosh]

Same issue.

Kindly provide the steps, if it has been resolved