How to whitelist specific keywords and lines from a logfile and ignore rest?

[prateeksawhney]

Hi Team,

I need urgent help on how to whitelist  specific lines from logfile and ignoring rest.

As an example this is a feed in my logfile :-

[2021-08-18 03:32:09.797] 2021-08-18 03:31:59.000, ip: 10.7.128.219, folder: 0, size: <nil>, event: ObjectRemoved:DeleteMarkerCreated, session: 15849,10.7.128.219, type: 2, region: eu-west-2, bucket: proftpd-prod-replicated, topic: arn:aws:sns:eu-west-2:563028249984:proftpd_prod_replicated_event_topic, key: export/sftp/ABE0A4FD16B68ADBC0B28AD415F/Credit_Index_Live_Latest/CREDIT_INDICES_LIVE_PRICING-20210811-0315.csv, sequencer: 00611C7F3529A4C883
deleteObject: Warning: Couldn't remove object '/export/sftp/ABE0A4FD16B68ADBC0B28AD415F/Credit_Index_Live_Latest/CREDIT_INDICES_LIVE_PRICING-20210811-0315.csv' from cache, cache might be stale
Detected cache out of sync, now relisting whole directory [/export/sftp/ABE0A4FD16B68ADBC0B28AD415F/Credit_Index_Live_Latest]
=== Now testing diff of folder and cache... [folder: export/sftp/ABE0A4FD16B68ADBC0B28AD415F/Credit_Index_Live_Latest/] ===============================
DIFF CALCULATION TOOK: 0.015115 [diffs: [/export/sftp/ABE0A4FD16B68ADBC0B28AD415F/Credit_Index_Live_Latest]: additions: 0, removals: 0, updates: 0, timestamp: 1629257529.797306]
Updating timestamp from: 1629253911.017497 to: 1629257529.797306
RESULT [/export/sftp/ABE0A4FD16B68ADBC0B28AD415F/Credit_Index_Live_Latest]: Size: 499, folders: 5, footprint: 30856, cache_: 0x7f6781fd2878
/:D:1 1629257529.812982
/..:D: [VIRTUAL]
- export 0
/export:D:1 1629257529.812975
/export/..:D: [VIRTUAL]
- sftp 0
/export/sftp:D:1 1629257529.812971
/export/sftp/..:D: [VIRTUAL]
- ABE0A4FD16B68ADBC0B28AD415F 0
/export/sftp/ABE0A4FD16B68ADBC0B28AD415F:D:1 1629257529.812988
/export/sftp/ABE0A4FD16B68ADBC0B28AD415F/..:D: [VIRTUAL]
- Credit_Index_Live_Latest 0
/export/sftp/ABE0A4FD16B68ADBC0B28AD415F/Credit_Index_Live_Latest:D:494 1629257529.797306
/export/sftp/ABE0A4FD16B68ADBC0B28AD415F/Credit_Index_Live_Latest/..:D: [VIRTUAL]
- CREDIT_INDICES_LIVE_PRICING-20210818-0330.csv 115660
- CREDIT_INDICES_LIVE_PRICING-20210818-0315.csv 115638
- CREDIT_INDICES_LIVE_PRICING-20210818-0300.csv 115636
- CREDIT_INDICES_LIVE_PRICING-20210818-0245.csv 115636

Out of the above lines I want only to enable feed for the line which is highlighted in red and ignore rest of the lines. 

Please suggest this can be achieved? 

Thanks in advance.

Regards,

Prateek Sawhney 

[venkatasri]

@prateeksawhney  if all the content belongs to same event its not possible to send partial event to nullQueue.

instead you can do SEDCMD- in props conf at index-time, deploy the props.conf to HF/indexers to make it work.

Read about SEDCMD here - https://docs.splunk.com/Documentation/Splunk/latest/admin/propsconf

---

An upvote would be appreciated if this reply helps!

[prateeksawhney]

@venkatasri 

Thanks for your reply here. Actually these are many separate lines which are coming in same logfile.

We want to index only those lines which starts with such date format - [2021-08-18 03:32:09.797]

We want to ignore all the other lines which do not start with such date format.

Example - RESULT [/export/sftp/ABE0A4FD16B68ADBC0B28AD415F

OR

Example - Credit_Index_Live_Latest 0

So just to clarify again we only want to index lines which starts with date format highlighted in green and ignore all the other lines which starts like highlighted in red. Let me know if still any doubt.

Hoping for a quick reply on this.

Thanks a lot again.

[venkatasri]

@prateeksawhney  How is your events looks like in Splunk line_breaker set correctly?

All these lines as single event or multiple events?