Getting Data In

What am I missing in this props/transforms pair to filter "noise" from my log?

rgcurry
Contributor

I am trying to filter log "noise" before the data gets indexed but the filtering is not working. I have tested the REGEX in the transforms.conf in a search, it works perfectly. But the noise is coming thru and I am not seeing what I have missed. I've dug thru probably 2 dozen other questions/inquiries here with similar situations but none of those answers has solved my situation. Perhaps your review will pick up what I am not.

My props.conf file looks like this:

[log_data]
TRANSFORMS-set = dumpNoise,keepInfo 
EXTRACT-filename = loaded\"\s(?<filename>\S+) 
EXTRACT-course_filedate = coursefiledate\w(?<course_filedate>\w+) 
EXTRACT-transcript_filedate = transcriptfiledate\w(?<transcript_filedate>\w+)

"log_date" is the correct sourcetype for this data coming into the Indexer.

The transforms.conf file looks like this:

[dumpKeepAlives]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[keepFileXferInfo]
REGEX = action executed|action failed|file upload|file download|trigger error
DEST_KEY = queue
FORMAT = indexQueue

Here is a sample of the data in the log that I am tring to filter:

2012-07-16 17:08:42 192.168.71.22 24504 192.168.64.188 22 - - - "session started" - - -^M
2012-07-16 17:08:42 192.168.71.22 24504 192.168.64.188 22 - - - SSH-2.0-JSCAPE - - -^M
2012-07-16 17:08:42 192.168.71.22 24504 192.168.64.188 22 - - - "logged out" - - -^M
2012-07-16 17:08:42 192.168.71.22 24504 192.168.64.188 22 - - - "session closed; " - - -^M
2012-07-16 17:08:43 - - sftp-stg.bazaarvoice.com 22 - - - "external file uploaded" /jscape/JSCAPE_MFT_Server/users/ExternalUsers/BazaarSFTP/outbound/bv_catal
og.xml 4857550 -^M
2012-07-16 17:08:43 - - - - - - - "action executed" "trigger=BazaarVoiceOutbound_LENA288_sftp-stg.bazaarvoice.com_ENTMFILE_3_4_DigitalPlatform; class=class c
om.jscape.inet.mft.workflow.actions.SftpRegexFileUploadAction; message=files which matches with .* has been uploaded" - -^M
2012-07-16 17:08:44 - - - - - - - "action executed" "trigger=BazaarVoiceOutbound_LENA288_sftp-stg.bazaarvoice.com_ENTMFILE_3_4_DigitalPlatform; class=class c
om.jscape.inet.mft.workflow.actions.MoveRegexFileAction; message=the file(s) /jscape/JSCAPE_MFT_Server/users/ExternalUsers/BazaarSFTP/outbound/bv_catalog.xml
 has been moved" - -^M
2012-07-16 17:08:44 - - - - - - - "action executed" "trigger=BazaarVoiceOutbound_LENA288_sftp-stg.bazaarvoice.com_ENTMFILE_3_4_DigitalPlatform; class=class c
om.jscape.inet.mft.workflow.actions.DeleteFileAction; message=file /jscape/JSCAPE_MFT_Server/users/ExternalUsers/BazaarSFTP/trigger/Done deleted" - -^M
2012-07-16 17:08:47 192.168.71.22 36487 192.168.64.188 22 - - - "session started" - - -^M
2012-07-16 17:08:47 192.168.71.22 36487 192.168.64.188 22 - - - SSH-2.0-JSCAPE - - -^M
2012-07-16 17:08:47 192.168.71.22 36487 192.168.64.188 22 - - - "logged out" - - -^M
2012-07-16 17:08:47 192.168.71.22 36487 192.168.64.188 22 - - - "session closed; " - - -^M
2012-07-16 17:08:52 192.168.71.22 27319 192.168.64.188 22 - - - "session started" - - -^M
2012-07-16 17:08:52 192.168.71.22 27319 192.168.64.188 22 - - - SSH-2.0-JSCAPE - - -^M
2012-07-16 17:08:52 192.168.71.22 27319 192.168.64.188 22 - - - "logged out" - - -^M
2012-07-16 17:08:52 192.168.71.22 27319 192.168.64.188 22 - - - "session closed; " - - -^M

Everything is coming thru, the noise and the wanted data. What do you see that I am missing?

I appreciate any suggestions on this.

0 Karma

Ayn
Legend

In props.conf, you refer to the transforms dumpNoise and keepInfo, but in transforms.conf the names are dumpKeepAlives and keepFileXferInfo, so obviously that won't match. Typo in your post here or typo in your conf files as well?

rgcurry
Contributor

OOPS! Typo in my post. The values in the transforms.conf is what I am really using in both on the system. I plan to change the names as they reflect more what I am trying to do now. I was filtering a smaller set of data before, which was working (and still working) fine.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...