Getting Data In

What am I missing in this props/transforms pair to filter "noise" from my log?

rgcurry
Contributor

I am trying to filter log "noise" before the data gets indexed but the filtering is not working. I have tested the REGEX in the transforms.conf in a search, it works perfectly. But the noise is coming thru and I am not seeing what I have missed. I've dug thru probably 2 dozen other questions/inquiries here with similar situations but none of those answers has solved my situation. Perhaps your review will pick up what I am not.

My props.conf file looks like this:

[log_data]
TRANSFORMS-set = dumpNoise,keepInfo 
EXTRACT-filename = loaded\"\s(?<filename>\S+) 
EXTRACT-course_filedate = coursefiledate\w(?<course_filedate>\w+) 
EXTRACT-transcript_filedate = transcriptfiledate\w(?<transcript_filedate>\w+)

"log_date" is the correct sourcetype for this data coming into the Indexer.

The transforms.conf file looks like this:

[dumpKeepAlives]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[keepFileXferInfo]
REGEX = action executed|action failed|file upload|file download|trigger error
DEST_KEY = queue
FORMAT = indexQueue

Here is a sample of the data in the log that I am tring to filter:

2012-07-16 17:08:42 192.168.71.22 24504 192.168.64.188 22 - - - "session started" - - -^M
2012-07-16 17:08:42 192.168.71.22 24504 192.168.64.188 22 - - - SSH-2.0-JSCAPE - - -^M
2012-07-16 17:08:42 192.168.71.22 24504 192.168.64.188 22 - - - "logged out" - - -^M
2012-07-16 17:08:42 192.168.71.22 24504 192.168.64.188 22 - - - "session closed; " - - -^M
2012-07-16 17:08:43 - - sftp-stg.bazaarvoice.com 22 - - - "external file uploaded" /jscape/JSCAPE_MFT_Server/users/ExternalUsers/BazaarSFTP/outbound/bv_catal
og.xml 4857550 -^M
2012-07-16 17:08:43 - - - - - - - "action executed" "trigger=BazaarVoiceOutbound_LENA288_sftp-stg.bazaarvoice.com_ENTMFILE_3_4_DigitalPlatform; class=class c
om.jscape.inet.mft.workflow.actions.SftpRegexFileUploadAction; message=files which matches with .* has been uploaded" - -^M
2012-07-16 17:08:44 - - - - - - - "action executed" "trigger=BazaarVoiceOutbound_LENA288_sftp-stg.bazaarvoice.com_ENTMFILE_3_4_DigitalPlatform; class=class c
om.jscape.inet.mft.workflow.actions.MoveRegexFileAction; message=the file(s) /jscape/JSCAPE_MFT_Server/users/ExternalUsers/BazaarSFTP/outbound/bv_catalog.xml
 has been moved" - -^M
2012-07-16 17:08:44 - - - - - - - "action executed" "trigger=BazaarVoiceOutbound_LENA288_sftp-stg.bazaarvoice.com_ENTMFILE_3_4_DigitalPlatform; class=class c
om.jscape.inet.mft.workflow.actions.DeleteFileAction; message=file /jscape/JSCAPE_MFT_Server/users/ExternalUsers/BazaarSFTP/trigger/Done deleted" - -^M
2012-07-16 17:08:47 192.168.71.22 36487 192.168.64.188 22 - - - "session started" - - -^M
2012-07-16 17:08:47 192.168.71.22 36487 192.168.64.188 22 - - - SSH-2.0-JSCAPE - - -^M
2012-07-16 17:08:47 192.168.71.22 36487 192.168.64.188 22 - - - "logged out" - - -^M
2012-07-16 17:08:47 192.168.71.22 36487 192.168.64.188 22 - - - "session closed; " - - -^M
2012-07-16 17:08:52 192.168.71.22 27319 192.168.64.188 22 - - - "session started" - - -^M
2012-07-16 17:08:52 192.168.71.22 27319 192.168.64.188 22 - - - SSH-2.0-JSCAPE - - -^M
2012-07-16 17:08:52 192.168.71.22 27319 192.168.64.188 22 - - - "logged out" - - -^M
2012-07-16 17:08:52 192.168.71.22 27319 192.168.64.188 22 - - - "session closed; " - - -^M

Everything is coming thru, the noise and the wanted data. What do you see that I am missing?

I appreciate any suggestions on this.

0 Karma

Ayn
Legend

In props.conf, you refer to the transforms dumpNoise and keepInfo, but in transforms.conf the names are dumpKeepAlives and keepFileXferInfo, so obviously that won't match. Typo in your post here or typo in your conf files as well?

rgcurry
Contributor

OOPS! Typo in my post. The values in the transforms.conf is what I am really using in both on the system. I plan to change the names as they reflect more what I am trying to do now. I was filtering a smaller set of data before, which was working (and still working) fine.

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

[Puzzles] Solve, Learn, Repeat: Matching cron expressions

This puzzle (first published here) is based on matching timestamps to cron expressions.All the timestamps ...

Design, Compete, Win: Submit Your Best Splunk Dashboards for a .conf26 Pass

Hello Splunkers,  We’re excited to kick off a Splunk Dashboard contest! We know that dashboards are a primary ...

May 2026 Splunk Expert Sessions: Security & Observability

Level Up Your Operations: May 2026 Splunk Expert Sessions Whether you are refining your security posture or ...