I am trying to filter log "noise" before the data gets indexed but the filtering is not working. I have tested the REGEX in the transforms.conf in a search, it works perfectly. But the noise is coming thru and I am not seeing what I have missed. I've dug thru probably 2 dozen other questions/inquiries here with similar situations but none of those answers has solved my situation. Perhaps your review will pick up what I am not.
My props.conf file looks like this:
[log_data]
TRANSFORMS-set = dumpNoise,keepInfo
EXTRACT-filename = loaded\"\s(?<filename>\S+)
EXTRACT-course_filedate = coursefiledate\w(?<course_filedate>\w+)
EXTRACT-transcript_filedate = transcriptfiledate\w(?<transcript_filedate>\w+)
"log_date" is the correct sourcetype for this data coming into the Indexer.
The transforms.conf file looks like this:
[dumpKeepAlives]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[keepFileXferInfo]
REGEX = action executed|action failed|file upload|file download|trigger error
DEST_KEY = queue
FORMAT = indexQueue
Here is a sample of the data in the log that I am tring to filter:
2012-07-16 17:08:42 192.168.71.22 24504 192.168.64.188 22 - - - "session started" - - -^M
2012-07-16 17:08:42 192.168.71.22 24504 192.168.64.188 22 - - - SSH-2.0-JSCAPE - - -^M
2012-07-16 17:08:42 192.168.71.22 24504 192.168.64.188 22 - - - "logged out" - - -^M
2012-07-16 17:08:42 192.168.71.22 24504 192.168.64.188 22 - - - "session closed; " - - -^M
2012-07-16 17:08:43 - - sftp-stg.bazaarvoice.com 22 - - - "external file uploaded" /jscape/JSCAPE_MFT_Server/users/ExternalUsers/BazaarSFTP/outbound/bv_catal
og.xml 4857550 -^M
2012-07-16 17:08:43 - - - - - - - "action executed" "trigger=BazaarVoiceOutbound_LENA288_sftp-stg.bazaarvoice.com_ENTMFILE_3_4_DigitalPlatform; class=class c
om.jscape.inet.mft.workflow.actions.SftpRegexFileUploadAction; message=files which matches with .* has been uploaded" - -^M
2012-07-16 17:08:44 - - - - - - - "action executed" "trigger=BazaarVoiceOutbound_LENA288_sftp-stg.bazaarvoice.com_ENTMFILE_3_4_DigitalPlatform; class=class c
om.jscape.inet.mft.workflow.actions.MoveRegexFileAction; message=the file(s) /jscape/JSCAPE_MFT_Server/users/ExternalUsers/BazaarSFTP/outbound/bv_catalog.xml
has been moved" - -^M
2012-07-16 17:08:44 - - - - - - - "action executed" "trigger=BazaarVoiceOutbound_LENA288_sftp-stg.bazaarvoice.com_ENTMFILE_3_4_DigitalPlatform; class=class c
om.jscape.inet.mft.workflow.actions.DeleteFileAction; message=file /jscape/JSCAPE_MFT_Server/users/ExternalUsers/BazaarSFTP/trigger/Done deleted" - -^M
2012-07-16 17:08:47 192.168.71.22 36487 192.168.64.188 22 - - - "session started" - - -^M
2012-07-16 17:08:47 192.168.71.22 36487 192.168.64.188 22 - - - SSH-2.0-JSCAPE - - -^M
2012-07-16 17:08:47 192.168.71.22 36487 192.168.64.188 22 - - - "logged out" - - -^M
2012-07-16 17:08:47 192.168.71.22 36487 192.168.64.188 22 - - - "session closed; " - - -^M
2012-07-16 17:08:52 192.168.71.22 27319 192.168.64.188 22 - - - "session started" - - -^M
2012-07-16 17:08:52 192.168.71.22 27319 192.168.64.188 22 - - - SSH-2.0-JSCAPE - - -^M
2012-07-16 17:08:52 192.168.71.22 27319 192.168.64.188 22 - - - "logged out" - - -^M
2012-07-16 17:08:52 192.168.71.22 27319 192.168.64.188 22 - - - "session closed; " - - -^M
Everything is coming thru, the noise and the wanted data. What do you see that I am missing?
I appreciate any suggestions on this.
In props.conf
, you refer to the transforms dumpNoise
and keepInfo
, but in transforms.conf
the names are dumpKeepAlives
and keepFileXferInfo
, so obviously that won't match. Typo in your post here or typo in your conf files as well?
OOPS! Typo in my post. The values in the transforms.conf is what I am really using in both on the system. I plan to change the names as they reflect more what I am trying to do now. I was filtering a smaller set of data before, which was working (and still working) fine.