Splunk Search

Is it possible before doing indexing I want to ignore some lines from a log file?

Path Finder

I have a Log file. below mentioned lines are available in that Log file. I want to ignore all lines after the entire data of line number 14.

  1. EVENT_SHURU;0;03/01/2018 01:00:04:4000;1;SHOP;;5040;1;0;0;;Imports for the MASTER STORE
  3. EVENT_SHURU;0;03/01/2018 01:00:04:4000;1;SHOP;;1360;1;0;0;;Start WSUPDATE execution
  4. EVENT_SESH;0;03/01/2018 01:00:04:5000;1;SHOP;;1360;1;0;;;End of WSUPDATE execution
  5. EVENT_SHURU;0;03/01/2018 01:00:06:0000;1;SHOP;;1030;1;0;0;;C:\SHOP\COMS\GARIA\IN\TAXES.TXT
  6. EVENT_GOL;0;03/01/2018 01:00:06:0000;1;SHOP;;1030;1;1;;;File not found : C:\SHOP\COMS\GARIA\IN\TAXES.TXT
  7. EVENT_SHURU;0;03/01/2018 01:05:03:0000;1;SHOP;;1940;IMPORT OF THE GIFTLIST;1;0;0;;C:\SHOP\COMS\GARIA\IN\GIFTLIST.TXT
  8. EVENT_GOL;0;03/01/2018 01:05:03:1000;1;SHOP;;1940;1;1;;;File not found : C:\SHOP\COMS\GARIA\IN\GIFTLIST.TXT
  9. EVENT_SESH;0;03/01/2018 01:05:03:1000;1;SHOP;;1940;1;0;;;Import canceled : import file not found
  10. EVENT_GOL;0;03/01/2018 01:05:04:6000;1;SHOP;;1410;1;1;;;Expected File #0001 not found
  11. EVENT_SESH;0;03/01/2018 01:05:04:6000;1;SHOP;;1410;1;0;;;END OF THE AUTOMATE UNPACKING PROCESS FOR THE STORE :
  12. EVENT_SHURU;0;03/01/2018 01:05:04:6000;1;SHOP;;1360;1;0;0;;Start WSUPDATE execution
  13. EVENT_SESH;0;03/01/2018 01:05:04:7000;1;SHOP;;1360;1;0;;;End of WSUPDATE execution
  14. EVENT_SESH;0;03/01/2018 01:05:04:7000;1;SHOP;;5040;1;0;;;END OF IMPORT PROCESS FOR THE MASTER STORE - PENDING_TXT(0)
  15. EVENT_SHURU;0;04/01/2018 06:30:37:1000;1;SHOP;;1970;1;0;0;;C:\SHOP\COMS\NEWTOWN\IN\0331\TILLLIST.TXT
  16. EVENT_GOL;0;04/01/2018 06:30:37:1000;1;SHOP;;1970;1;1;;;File not found : C:\SHOP\COMS\NEWTOWN\IN\0331\TILLLIST.TXT
  17. EVENT_SESH;0;04/01/2018 06:30:37:2000;1;SHOP;;1970;1;0;;;Import canceled : import file not found
  18. EVENT_SHURU;0;04/01/2018 06:30:37:2000;1;SHOP;;1010;1;0;0;;C:\SHOP\COMS\NEWTOWN\IN\0331\PROD.TXT
  19. EVENT_GOL;0;04/01/2018 06:30:37:2000;1;SHOP;;1010;1;1;;;File not found : C:\SHOP\COMS\NEWTOWN\IN\0331\PROD.TXT
  20. EVENT_SESH;0;04/01/2018 06:30:37:3000;1;SHOP;;1010;1;0;;;Import canceled : import file not found
  21. EVENT_SHURU;0;04/01/2018 06:30:37:3000;1;SHOP;;1080;1;0;0;;C:\SHOP\COMS\NEWTOWN\IN\0331\MTPRO.TXT
  22. EVENT_GOL;0;04/01/2018 06:30:37:4000;1;SHOP;;1080;1;1;;;File not found : C:\SHOP\COMS\NEWTOWN\IN\0331\MTPRO.TXT
  23. EVENT_SESH;0;04/01/2018 06:30:37:4000;1;SHOP;;1080;1;0;;;Import canceled : import file not found
  24. EVENT_SHURU;0;04/01/2018 06:30:37:5000;1;SHOP;;1180;1;0;0;;C:\SHOP\COMS\NEWTOWN\IN\0331\STAFF.TXT
  25. EVENT_GOL;0;04/01/2018 06:30:45:5000;1;SHOP;;1180;1;2;0;;0 salemen inserted, 6 salesmen updated.

I want to this entire ignoring process will be done before indexing in any log file in everyday.

Please tell me the whole process with the regular expression.(line numbers are not present in Log files, I have added it here only for help to understand after what line i want to ignore the other lines )

0 Karma

0 Karma


1) The key word you are probably looking for is nullqueue. You can send data to the null queue by setting up a regex to match that data.

2) We have no idea why line 14 is different from any other line. you need to explain exactly what is different about that line or the lines after it.

I notice that those dates are a few weeks off. If you are trying to ignore future records, then we need to know whether this is a one-shot or an ongoing thing that you might be receiving future records you want to drop.

0 Karma
Get Updates on the Splunk Community!

Database Performance Sidebar Panel Now on APM Database Query Performance & Service ...

We’ve streamlined the troubleshooting experience for database-related service issues by adding a database ...

IM Landing Page Filter - Now Available

We’ve added the capability for you to filter across the summary details on the main Infrastructure Monitoring ...

Dynamic Links from Alerts to IM Navigators - New in Observability Cloud

Splunk continues to improve the troubleshooting experience in Observability Cloud with this latest enhancement ...