Security

How to extract all sub strings ends with .csv in a string?

SumitPan
Explorer

Sorry I'm new to regex. I'm trying to get some meaning full data from the log files.

I want all the sub-strings ending with .csv in my log file at any given point of time. Below is the the log file preview. Any leads would be highly appreciated.

alt text

Tags (2)
0 Karma
1 Solution

SumitPan
Explorer

it fixed the problem.......

rex field=_raw ".*\s(?P.*\.csv)$" |search CSVFiles=*

View solution in original post

0 Karma

SumitPan
Explorer

it fixed the problem.......

rex field=_raw ".*\s(?P.*\.csv)$" |search CSVFiles=*
0 Karma

lakromani
Builder

This rex does not work with the above data.
It can not be complete here?

PS you do not need to specify field=_raw, if omitted, _raw is used by default.

0 Karma

SumitPan
Explorer

I have tried both the rex and both seems to be working fine.

0 Karma

lakromani
Builder

(?P..csv) this does not extract anything. If it was more like (?<CSVFiles>\w+\.csv) it will work.

0 Karma

lakromani
Builder

It's better if you past the text in stead of a picture of the text.

0 Karma

SumitPan
Explorer

Below are the logs. Need to list down all files names ending with .csv. e.g.:
1. adn_attribute_set.csv
2. adn_navigation_attributes.csv

host=mdc1vr1002 sourcetype=MCOM_ETL_OUT

2016-11-06 19:42:35,800 | DEBUG | main:ConcatNCopy | Appending smaller file: adn_attribute_set.csv
2016-11-06 19:42:35,801 | DEBUG | main:ConcatNCopy | Copy: adn_attribute_set.csv to /opt/pim/ETL/MCOM/etlc/output/site/adn_attribute_set.csv, size: 0, elapsed ms: 1
2016-11-06 19:42:35,801 | DEBUG | main:ConcatNCopy | Appending smaller file: adn_navigation_attributes.csv
2016-11-06 19:42:35,801 | DEBUG | main:ConcatNCopy | Copy: adn_navigation_attributes.csv to /opt/pim/ETL/MCOM/etlc/output/site/adn_navigation_attributes.csv, size: 0, elapsed ms: 0
2016-11-06 19:42:35,809 | DEBUG | main:ConcatNCopy | Appending smaller file: archived_products.csv
2016-11-06 19:42:35,830 | DEBUG | main:ConcatNCopy | Copy: archived_products.csv to /opt/pim/ETL/MCOM/etlc/output/site/archived_products.csv, size: 2768026, elapsed ms: 21
2016-11-06 19:42:35,853 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_cat.csv
2016-11-06 19:42:36,043 | DEBUG | main:ConcatNCopy | Copy: attr_cat.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_cat.csv, size: 201223799, elapsed ms: 190
2016-11-06 19:42:36,043 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_category_exclusion.csv
2016-11-06 19:42:36,044 | DEBUG | main:ConcatNCopy | Copy: attr_category_exclusion.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_category_exclusion.csv, size: 16705, elapsed ms: 1
2016-11-06 19:42:36,045 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_page_media.csv
2016-11-06 19:42:36,079 | DEBUG | main:ConcatNCopy | Copy: attr_page_media.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_page_media.csv, size: 38563205, elapsed ms: 34
2016-11-06 19:42:36,125 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_prod_brand_ship.csv
2016-11-06 19:42:36,169 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_prod_copy_reg.csv
2016-11-06 19:42:36,359 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_prod_data_source.csv
2016-11-06 19:42:36,366 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_prod_forced_new.csv
2016-11-06 19:42:36,422 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_prod_site_search.csv
2016-11-06 19:42:36,773 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_prod_trigger_data.csv
2016-11-06 19:42:36,773 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_prod_tuple_data.csv
2016-11-06 19:42:36,977 | DEBUG | main:ConcatNCopy | Copy: attr_prod_brand_ship.csv attr_prod_copy_reg.csv attr_prod_data_source.csv attr_prod_forced_new.csv attr_prod_site_search.csv attr_prod_trigger_data.csv attr_prod_tuple_data.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_prod.csv, size: 876915462, elapsed ms: 852
2016-11-06 19:42:36,977 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_prod2.csv
2016-11-06 19:42:36,977 | DEBUG | main:ConcatNCopy | Copy: attr_prod2.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_prod2.csv, size: 0, elapsed ms: 0
2016-11-06 19:42:36,977 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_prod3.csv
2016-11-06 19:42:36,977 | DEBUG | main:ConcatNCopy | Copy: attr_prod3.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_prod3.csv, size: 0, elapsed ms: 0
2016-11-06 19:42:36,977 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_prod4.csv
2016-11-06 19:42:36,978 | DEBUG | main:ConcatNCopy | Copy: attr_prod4.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_prod4.csv, size: 0, elapsed ms: 1
2016-11-06 19:42:36,978 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_prod5.csv
2016-11-06 19:42:36,984 | DEBUG | main:ConcatNCopy | Copy: attr_prod5.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_prod5.csv, size: 6903099, elapsed ms: 6
2016-11-06 19:42:36,985 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_prod7.csv
2016-11-06 19:42:36,985 | DEBUG | main:ConcatNCopy | Copy: attr_prod7.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_prod7.csv, size: 0, elapsed ms: 1
2016-11-06 19:42:36,985 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_prod8.csv
2016-11-06 19:42:36,985 | DEBUG | main:ConcatNCopy | Copy: attr_prod8.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_prod8.csv, size: 0, elapsed ms: 0
2016-11-06 19:42:36,985 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_prod_colorway.csv
2016-11-06 19:42:36,985 | DEBUG | main:ConcatNCopy | Copy: attr_prod_colorway.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_prod_colorway.csv, size: 11624, elapsed ms: 0
2016-11-06 19:42:36,985 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_product_exclusion.csv
2016-11-06 19:42:36,991 | DEBUG | main:ConcatNCopy | Copy: attr_product_exclusion.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_product_exclusion.csv, size: 5340406, elapsed ms: 6
2016-11-06 19:42:36,991 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_promo.csv
2016-11-06 19:42:36,992 | DEBUG | main:ConcatNCopy | Copy: attr_promo.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_promo.csv, size: 577403, elapsed ms: 1
2016-11-06 19:42:36,992 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_req.csv
2016-11-06 19:42:36,992 | DEBUG | main:ConcatNCopy | Copy: attr_req.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_req.csv, size: 52738, elapsed ms: 0
2016-11-06 19:42:37,010 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_upc.csv
2016-11-06 19:42:37,495 | DEBUG | main:ConcatNCopy | Copy: attr_upc.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_upc.csv, size: 478738319, elapsed ms: 485
2016-11-06 19:42:37,495 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_vl.csv
2016-11-06 19:42:37,495 | DEBUG | main:ConcatNCopy | Copy: attr_vl.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_vl.csv, size: 21284, elapsed ms: 0
2016-11-06 19:42:37,496 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_vlitems.csv
2016-11-06 19:42:37,497 | DEBUG | main:ConcatNCopy | Copy: attr_vlitems.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_vlitems.csv, size: 1181621, elapsed ms: 1
2016-11-06 19:42:37,497 | DEBUG | main:ConcatNCopy | Appending smaller file: attribute.csv
2016-11-06 19:42:37,497 | DEBUG | main:ConcatNCopy | Copy: attribute.csv to /opt/pim/ETL/MCOM/etlc/output/site/attribute.csv, size: 173351, elapsed ms: 0
2016-11-06 19:42:37,497 | DEBUG | main:ConcatNCopy | Appending smaller file: brand.csv
2016-11-06 19:42:37,498 | DEBUG | main:ConcatNCopy | Copy: brand.csv to /opt/pim/ETL/MCOM/etlc/output/site/brand.csv, size: 117929, elapsed ms: 1
2016-11-06 19:42:37,498 | DEBUG | main:ConcatNCopy | Appending smaller file: brand_constraint.csv
2016-11-06 19:42:37,498 | DEBUG | main:ConcatNCopy | Copy: brand_constraint.csv to /opt/pim/ETL/MCOM/etlc/output/site/brand_constraint.csv, size: 0, elapsed ms: 0
2016-11-06 19:42:37,498 | DEBUG | main:ConcatNCopy | Appending smaller file: brand_constraint_val.csv
2016-11-06 19:42:37,498 | DEBUG | main:ConcatNCopy | Copy: brand_constraint_val.csv to /opt/pim/ETL/MCOM/etlc/output/site/brand_constraint_val.csv, size: 0, elapsed ms: 0
2016-11-06 19:42:37,498 | DEBUG | main:ConcatNCopy | Appending smaller file: cat_pools.csv
2016-11-06 19:42:37,502 | DEBUG | main:ConcatNCopy | Copy: cat_pools.csv to /opt/pim/ETL/MCOM/etlc/output/site/cat_pools.csv, size: 4362512, elapsed ms: 4
2016-11-06 19:42:37,502 | DEBUG | main:ConcatNCopy | Appending smaller file: cat_prod.csv
2016-11-06 19:42:37,503 | DEBUG | main:ConcatNCopy | Copy: cat_prod.csv to /opt/pim/ETL/MCOM/etlc/output/site/cat_prod.csv, size: 54444, elapsed ms: 1
2016-11-06 19:42:37,503 | DEBUG | main:ConcatNCopy | Appending smaller file: catalog.csv
2016-11-06 19:42:37,503 | DEBUG | main:ConcatNCopy | Copy: catalog.csv to /opt/pim/ETL/MCOM/etlc/output/site/catalog.csv, size: 9310, elapsed ms: 0
2016-11-06 19:42:37,503 | DEBUG | main:ConcatNCopy | Appending smaller file: catalog_context.csv
2016-11-06 19:42:37,503 | DEBUG | main:ConcatNCopy | Copy: catalog_context.csv to /opt/pim/ETL/MCOM/etlc/output/site/catalog_context.csv, size: 31, elapsed ms: 0
2016-11-06 19:42:37,504 | DEBUG | main:ConcatNCopy | Appending smaller file: category.csv
2016-11-06 19:42:37,512 | DEBUG | main:ConcatNCopy | Copy: category.csv to /opt/pim/ETL/MCOM/etlc/output/site/category.csv, size: 9478833, elapsed ms: 8
2016-11-06 19:42:37,513 | DEBUG | main:ConcatNCopy | Appending smaller file: category_facet.csv
2016-11-06 19:42:37,543 | DEBUG | main:ConcatNCopy | Copy: category_facet.csv to /opt/pim/ETL/MCOM/etlc/output/site/category_facet.csv, size: 36649061, elapsed ms: 30
2016-11-06 19:42:37,544 | DEBUG | main:ConcatNCopy | Appending smaller file: contextual_category.csv
2016-11-06 19:42:37,547 | DEBUG | main:ConcatNCopy | Copy: contextual_category.csv to /opt/pim/ETL/MCOM/etlc/output/site/contextual_category.csv, size: 2614776, elapsed ms: 3

0 Karma

lakromani
Builder

Then this should do:
your search | rex "(?<file>\w+\.csv)"

Sames as rich7177 posted. So if this works, accept his answer.

PS some lines have more than one file name, this rex gets them all.

2016-11-06 19:42:36,977 | DEBUG | main:ConcatNCopy | Copy: attr_prod_brand_ship.csv attr_prod_copy_reg.csv attr_prod_data_source.csv attr_prod_forced_new.csv attr_prod_site_search.csv attr_prod_trigger_data.csv attr_prod_tuple_data.csv to
0 Karma

Richfez
SplunkTrust
SplunkTrust

Try..

 ... | rex "(?<MyCSVFile>\w+\.csv)" 

Change the name MyCSVFile to whatever you want to call it. Here you can see it in regex101.com.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...