Security

How to extract all sub strings ends with .csv in a string?

SumitPan
Explorer

Sorry I'm new to regex. I'm trying to get some meaning full data from the log files.

I want all the sub-strings ending with .csv in my log file at any given point of time. Below is the the log file preview. Any leads would be highly appreciated.

alt text

Tags (2)
0 Karma
1 Solution

SumitPan
Explorer

it fixed the problem.......

rex field=_raw ".*\s(?P.*\.csv)$" |search CSVFiles=*

View solution in original post

0 Karma

SumitPan
Explorer

it fixed the problem.......

rex field=_raw ".*\s(?P.*\.csv)$" |search CSVFiles=*

View solution in original post

0 Karma

lakromani
Builder

This rex does not work with the above data.
It can not be complete here?

PS you do not need to specify field=_raw, if omitted, _raw is used by default.

0 Karma

SumitPan
Explorer

I have tried both the rex and both seems to be working fine.

0 Karma

lakromani
Builder

(?P..csv) this does not extract anything. If it was more like (?<CSVFiles>\w+\.csv) it will work.

0 Karma

lakromani
Builder

It's better if you past the text in stead of a picture of the text.

0 Karma

SumitPan
Explorer

Below are the logs. Need to list down all files names ending with .csv. e.g.:
1. adn_attribute_set.csv
2. adn_navigation_attributes.csv

host=mdc1vr1002 sourcetype=MCOM_ETL_OUT

2016-11-06 19:42:35,800 | DEBUG | main:ConcatNCopy | Appending smaller file: adn_attribute_set.csv
2016-11-06 19:42:35,801 | DEBUG | main:ConcatNCopy | Copy: adn_attribute_set.csv to /opt/pim/ETL/MCOM/etlc/output/site/adn_attribute_set.csv, size: 0, elapsed ms: 1
2016-11-06 19:42:35,801 | DEBUG | main:ConcatNCopy | Appending smaller file: adn_navigation_attributes.csv
2016-11-06 19:42:35,801 | DEBUG | main:ConcatNCopy | Copy: adn_navigation_attributes.csv to /opt/pim/ETL/MCOM/etlc/output/site/adn_navigation_attributes.csv, size: 0, elapsed ms: 0
2016-11-06 19:42:35,809 | DEBUG | main:ConcatNCopy | Appending smaller file: archived_products.csv
2016-11-06 19:42:35,830 | DEBUG | main:ConcatNCopy | Copy: archived_products.csv to /opt/pim/ETL/MCOM/etlc/output/site/archived_products.csv, size: 2768026, elapsed ms: 21
2016-11-06 19:42:35,853 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_cat.csv
2016-11-06 19:42:36,043 | DEBUG | main:ConcatNCopy | Copy: attr_cat.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_cat.csv, size: 201223799, elapsed ms: 190
2016-11-06 19:42:36,043 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_category_exclusion.csv
2016-11-06 19:42:36,044 | DEBUG | main:ConcatNCopy | Copy: attr_category_exclusion.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_category_exclusion.csv, size: 16705, elapsed ms: 1
2016-11-06 19:42:36,045 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_page_media.csv
2016-11-06 19:42:36,079 | DEBUG | main:ConcatNCopy | Copy: attr_page_media.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_page_media.csv, size: 38563205, elapsed ms: 34
2016-11-06 19:42:36,125 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_prod_brand_ship.csv
2016-11-06 19:42:36,169 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_prod_copy_reg.csv
2016-11-06 19:42:36,359 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_prod_data_source.csv
2016-11-06 19:42:36,366 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_prod_forced_new.csv
2016-11-06 19:42:36,422 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_prod_site_search.csv
2016-11-06 19:42:36,773 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_prod_trigger_data.csv
2016-11-06 19:42:36,773 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_prod_tuple_data.csv
2016-11-06 19:42:36,977 | DEBUG | main:ConcatNCopy | Copy: attr_prod_brand_ship.csv attr_prod_copy_reg.csv attr_prod_data_source.csv attr_prod_forced_new.csv attr_prod_site_search.csv attr_prod_trigger_data.csv attr_prod_tuple_data.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_prod.csv, size: 876915462, elapsed ms: 852
2016-11-06 19:42:36,977 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_prod2.csv
2016-11-06 19:42:36,977 | DEBUG | main:ConcatNCopy | Copy: attr_prod2.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_prod2.csv, size: 0, elapsed ms: 0
2016-11-06 19:42:36,977 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_prod3.csv
2016-11-06 19:42:36,977 | DEBUG | main:ConcatNCopy | Copy: attr_prod3.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_prod3.csv, size: 0, elapsed ms: 0
2016-11-06 19:42:36,977 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_prod4.csv
2016-11-06 19:42:36,978 | DEBUG | main:ConcatNCopy | Copy: attr_prod4.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_prod4.csv, size: 0, elapsed ms: 1
2016-11-06 19:42:36,978 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_prod5.csv
2016-11-06 19:42:36,984 | DEBUG | main:ConcatNCopy | Copy: attr_prod5.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_prod5.csv, size: 6903099, elapsed ms: 6
2016-11-06 19:42:36,985 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_prod7.csv
2016-11-06 19:42:36,985 | DEBUG | main:ConcatNCopy | Copy: attr_prod7.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_prod7.csv, size: 0, elapsed ms: 1
2016-11-06 19:42:36,985 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_prod8.csv
2016-11-06 19:42:36,985 | DEBUG | main:ConcatNCopy | Copy: attr_prod8.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_prod8.csv, size: 0, elapsed ms: 0
2016-11-06 19:42:36,985 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_prod_colorway.csv
2016-11-06 19:42:36,985 | DEBUG | main:ConcatNCopy | Copy: attr_prod_colorway.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_prod_colorway.csv, size: 11624, elapsed ms: 0
2016-11-06 19:42:36,985 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_product_exclusion.csv
2016-11-06 19:42:36,991 | DEBUG | main:ConcatNCopy | Copy: attr_product_exclusion.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_product_exclusion.csv, size: 5340406, elapsed ms: 6
2016-11-06 19:42:36,991 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_promo.csv
2016-11-06 19:42:36,992 | DEBUG | main:ConcatNCopy | Copy: attr_promo.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_promo.csv, size: 577403, elapsed ms: 1
2016-11-06 19:42:36,992 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_req.csv
2016-11-06 19:42:36,992 | DEBUG | main:ConcatNCopy | Copy: attr_req.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_req.csv, size: 52738, elapsed ms: 0
2016-11-06 19:42:37,010 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_upc.csv
2016-11-06 19:42:37,495 | DEBUG | main:ConcatNCopy | Copy: attr_upc.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_upc.csv, size: 478738319, elapsed ms: 485
2016-11-06 19:42:37,495 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_vl.csv
2016-11-06 19:42:37,495 | DEBUG | main:ConcatNCopy | Copy: attr_vl.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_vl.csv, size: 21284, elapsed ms: 0
2016-11-06 19:42:37,496 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_vlitems.csv
2016-11-06 19:42:37,497 | DEBUG | main:ConcatNCopy | Copy: attr_vlitems.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_vlitems.csv, size: 1181621, elapsed ms: 1
2016-11-06 19:42:37,497 | DEBUG | main:ConcatNCopy | Appending smaller file: attribute.csv
2016-11-06 19:42:37,497 | DEBUG | main:ConcatNCopy | Copy: attribute.csv to /opt/pim/ETL/MCOM/etlc/output/site/attribute.csv, size: 173351, elapsed ms: 0
2016-11-06 19:42:37,497 | DEBUG | main:ConcatNCopy | Appending smaller file: brand.csv
2016-11-06 19:42:37,498 | DEBUG | main:ConcatNCopy | Copy: brand.csv to /opt/pim/ETL/MCOM/etlc/output/site/brand.csv, size: 117929, elapsed ms: 1
2016-11-06 19:42:37,498 | DEBUG | main:ConcatNCopy | Appending smaller file: brand_constraint.csv
2016-11-06 19:42:37,498 | DEBUG | main:ConcatNCopy | Copy: brand_constraint.csv to /opt/pim/ETL/MCOM/etlc/output/site/brand_constraint.csv, size: 0, elapsed ms: 0
2016-11-06 19:42:37,498 | DEBUG | main:ConcatNCopy | Appending smaller file: brand_constraint_val.csv
2016-11-06 19:42:37,498 | DEBUG | main:ConcatNCopy | Copy: brand_constraint_val.csv to /opt/pim/ETL/MCOM/etlc/output/site/brand_constraint_val.csv, size: 0, elapsed ms: 0
2016-11-06 19:42:37,498 | DEBUG | main:ConcatNCopy | Appending smaller file: cat_pools.csv
2016-11-06 19:42:37,502 | DEBUG | main:ConcatNCopy | Copy: cat_pools.csv to /opt/pim/ETL/MCOM/etlc/output/site/cat_pools.csv, size: 4362512, elapsed ms: 4
2016-11-06 19:42:37,502 | DEBUG | main:ConcatNCopy | Appending smaller file: cat_prod.csv
2016-11-06 19:42:37,503 | DEBUG | main:ConcatNCopy | Copy: cat_prod.csv to /opt/pim/ETL/MCOM/etlc/output/site/cat_prod.csv, size: 54444, elapsed ms: 1
2016-11-06 19:42:37,503 | DEBUG | main:ConcatNCopy | Appending smaller file: catalog.csv
2016-11-06 19:42:37,503 | DEBUG | main:ConcatNCopy | Copy: catalog.csv to /opt/pim/ETL/MCOM/etlc/output/site/catalog.csv, size: 9310, elapsed ms: 0
2016-11-06 19:42:37,503 | DEBUG | main:ConcatNCopy | Appending smaller file: catalog_context.csv
2016-11-06 19:42:37,503 | DEBUG | main:ConcatNCopy | Copy: catalog_context.csv to /opt/pim/ETL/MCOM/etlc/output/site/catalog_context.csv, size: 31, elapsed ms: 0
2016-11-06 19:42:37,504 | DEBUG | main:ConcatNCopy | Appending smaller file: category.csv
2016-11-06 19:42:37,512 | DEBUG | main:ConcatNCopy | Copy: category.csv to /opt/pim/ETL/MCOM/etlc/output/site/category.csv, size: 9478833, elapsed ms: 8
2016-11-06 19:42:37,513 | DEBUG | main:ConcatNCopy | Appending smaller file: category_facet.csv
2016-11-06 19:42:37,543 | DEBUG | main:ConcatNCopy | Copy: category_facet.csv to /opt/pim/ETL/MCOM/etlc/output/site/category_facet.csv, size: 36649061, elapsed ms: 30
2016-11-06 19:42:37,544 | DEBUG | main:ConcatNCopy | Appending smaller file: contextual_category.csv
2016-11-06 19:42:37,547 | DEBUG | main:ConcatNCopy | Copy: contextual_category.csv to /opt/pim/ETL/MCOM/etlc/output/site/contextual_category.csv, size: 2614776, elapsed ms: 3

0 Karma

lakromani
Builder

Then this should do:
your search | rex "(?<file>\w+\.csv)"

Sames as rich7177 posted. So if this works, accept his answer.

PS some lines have more than one file name, this rex gets them all.

2016-11-06 19:42:36,977 | DEBUG | main:ConcatNCopy | Copy: attr_prod_brand_ship.csv attr_prod_copy_reg.csv attr_prod_data_source.csv attr_prod_forced_new.csv attr_prod_site_search.csv attr_prod_trigger_data.csv attr_prod_tuple_data.csv to
0 Karma

Richfez
SplunkTrust
SplunkTrust

Try..

 ... | rex "(?<MyCSVFile>\w+\.csv)" 

Change the name MyCSVFile to whatever you want to call it. Here you can see it in regex101.com.

Take the 2021 Splunk Career Survey

Help us learn about how Splunk has
impacted your career by taking the 2021 Splunk Career Survey.

Earn $50 in Amazon cash!