topic Re: How do I edit my props.conf and transforms.conf to filter select events from one source file from getting indexed? in Getting Data In

How do I edit my props.conf and transforms.conf to filter select events from one source file from getting indexed?

bfnpmsz — Tue, 27 Oct 2015 15:57:34 GMT

We have a vanilla install, just one stand alone Splunk Server. I am wanting to filter select events from one source file. Not sure how to do it.

I have attempted to research the solution, but nothing so far has worked as expected. Maybe my expectations are not what they should be.

Here is my props.conf:

[source::"\\\\Alvionix03\\d\\InCharge\\SAM\\smarts\\local\\logs\\TRAPS-Proview.log"]
TRANSFORMS-null= discards

Here is my transforms.conf

[discards]
REGEX = Discard:\s+'YES'
DEST_KEY = queue
FORMAT = nullQueue

My expectations are that all the records which are marked as discarded in our log will not be indexed.
Example of one record of my data:

======================== Trap attributes =========================
Timestamp:           'October 27, 2015 10:54:16 AM CDT'
Agent:               '10.10.54.82'
Enterprise OID:      '.1.3.6.1.4.1.14760'
Generic Type:        '6'
Specific Type:       '1'
Varbinds:            [oid]->[varbind]
                     '.1.3.6.1.4.1.14760.2.1.2.1' --> 'A362-2250'
                     '.1.3.6.1.4.1.14760.2.1.2.2' --> '20151027095414'
                     '.1.3.6.1.4.1.14760.2.1.2.11' --> 'WFS_SYSE_DEVICE_STATUS: PhysicalName=CIM_CCDMWorkstationName=A362-2250 State=WFS_STAT_DEVONLINE (CIM_CCDM)'
                     '.1.3.6.1.4.1.14760.2.1.2.12' --> 'Device Cashin CCDM Module online'
                     '.1.3.6.1.4.1.14760.2.1.2.15' --> 'Cash/Cheque In'
=================== ICS_Notification attributes ==================
ClassName:           'Proview'
InstanceName:        'A362-2250'
EventName:           'ATM - A362-2250 - Device Cashin CCDM Module online'
Severity:            '5'
EventText:           'Proview/ATM Event: A362-2250 20151027095414 WFS_SYSE_DEVICE_STATUS: PhysicalName=CIM_CCDMWorkstationName=A362-2250 State=WFS_STAT_DEVONLINE (CIM_CCDM) Device Cashin CCDM Module online'
Category:            'SNMPTrap'
**Discard:             'YES'**
ForceOcc:            'A362-2250'
SuppressAgentOcc:    ''
UpdateUD:            ''
Expiration:          '600'
State:               'NOTIFY'
InMaintenance:       'FALSE'
ClearOnAcknowledge:  'TRUE'
TrapSource:          'Trap Processor'
EventType:           'MOMENTARY'
ASL:                 'proview.asl'
ElementClassName:    'Host'
ElementInstanceName: '10.10.54.82'
SysNameOrAddr:       'A362-2250'
UnknownAgent:        'CREATE'
LogFile:             'TRAPS-Proview.log'
UserDefined1:        '10.10.54.82'
UserDefined2:        ''
UserDefined3:        ''
UserDefined4:        ''
UserDefined5:        ''
UserDefined6:        ''
UserDefined7:        'Device Cashin CCDM Module online'
UserDefined8:        'Proview ATM Trap 1 from 10.10.54.82/10.10.54.82
MIB Module:     
wnProviewDeviceId:  A362-2250
wnProviewOriginalTime:  20151027095414
wnProviewServerTimed:   WFS_SYSE_DEVICE_STATUS: PhysicalName=CIM_CCDMWorkstationName=A362-2250 State=WFS_STAT_DEVONLINE (CIM_CCDM)
wnProviewEventType: Device Cashin CCDM Module online
wnProviewEventNumber:   Cash/Cheque In
wnProviewOriginalEventNumber:   
wnProviewDeviceState:   
wnProviewSetStateChange:    
wnProviewUnsetStateChange:  
wnProviewEventMask: 
wnProviewOriginalEventText: 
wnProviewEventText: 
wnProviewSetBitMask:    
wnProviewUnsetBitMask:  
wnProviewComponentName: 
wnProviewComponentState:    
wnProviewTransportAddress:  '
UserDefined9:        ''
UserDefined10:       ''
UserDefined11:        ''
UserDefined12:        ''
UserDefined13:        ''
UserDefined14:        ''
UserDefined15:        ''
UserDefined16:        ''
UserDefined17:        ''
UserDefined18:        ''
UserDefined19:        ''
UserDefined20:       ''
==================================================================

It is just not working at this time. I am still seeing the Discarded records indexed in Splunk.

Any assistance you can provide will be appreciated.

bfnpmsz

Re: How do I edit my props.conf and transforms.conf to filter select events from one source file from getting indexed?

somesoni2 — Tue, 27 Oct 2015 16:51:55 GMT

Try something like this for your props.conf entry

[source::...\\d\\InCharge\\SAM\\smarts\\local\\logs\\TRAPS-Proview.log]
 TRANSFORMS-null= discards

Re: How do I edit my props.conf and transforms.conf to filter select events from one source file from getting indexed?

bfnpmsz — Tue, 27 Oct 2015 17:37:44 GMT

I removed the quotes as you suggested, seemed logical, no luck though.

[source::\\\\Alvionix03\\d\\InCharge\\SAM\\smarts\\local\\logs\\TRAPS-Proview.log]
TRANSFORMS-null= discards

Still the events are getting indexed.

Any other ideas?

bfnpmsz

Re: How do I edit my props.conf and transforms.conf to filter select events from one source file from getting indexed?

somesoni2 — Tue, 27 Oct 2015 20:25:22 GMT

Can you try with exact stanza as mine (you seem to shared directory and I would suggest to try option without that)?

Re: How do I edit my props.conf and transforms.conf to filter select events from one source file from getting indexed?

bfnpmsz — Tue, 27 Oct 2015 20:44:36 GMT

[source::...\\d\\InCharge\\SAM\\smarts\\local\\logs\\TRAPS-Proview.log]
TRANSFORMS-null= discards

Still no love.... All records are getting indexed. The discards are still there.

bfnpmsz

Re: How do I edit my props.conf and transforms.conf to filter select events from one source file from getting indexed?

somesoni2 — Tue, 27 Oct 2015 20:52:08 GMT

Just to confirm, you're restarting Splunk after the change?

Re: How do I edit my props.conf and transforms.conf to filter select events from one source file from getting indexed?

woodcock — Wed, 28 Oct 2015 03:46:41 GMT

You need to deploy these files all of your Indexers (or if using them, Heavy Forwarders) and then restart all splunk instances there. When verifying function, only check NEW events, events indexed previous to the restart will not be effected.

Re: How do I edit my props.conf and transforms.conf to filter select events from one source file from getting indexed?

bfnpmsz — Wed, 28 Oct 2015 20:58:52 GMT

Woodcock,

Yeah, I wish I have not tried that already. I only have the one server with Splunk installed and so therefore one indexer. Each time I have restarted Splunk and the Discarded records are still indexed. The old records are not affected because they have been indexed before this change.

I am not sure where I am going wrong, but something is amiss.

Thanks for your comment and help.
bfnpmsz

Re: How do I edit my props.conf and transforms.conf to filter select events from one source file from getting indexed?

woodcock — Thu, 29 Oct 2015 17:07:52 GMT

The OS is windows, right?

Re: How do I edit my props.conf and transforms.conf to filter select events from one source file from getting indexed?

bfnpmsz — Thu, 29 Oct 2015 17:30:09 GMT

Yes, its Windows Server 2012.

bfnpmsz

Re: How do I edit my props.conf and transforms.conf to filter select events from one source file from getting indexed?

bfnpmsz — Thu, 29 Oct 2015 21:37:49 GMT

Yes, a restart after each config change.