topic Linebreaking and event date in text file in Getting Data In

Linebreaking and event date in text file

mcomfurf — Tue, 29 Sep 2020 07:09:49 GMT

We have a text file to log row counts for CSV files used in an ETL job. The format of the file is like this:

08/30/15 16:08:51
acme_file.csv: 0
coyote_file.csv: 0
anvil_file.csv: 88835
roadrunner_file.csv: 330333

I want to break this into one line per filename from line 2 onward, into fields {filename}.csv: {rowcount} Note that the first line in the .txt file is the datetime for all of the events.

I think the linebreaking is fairly straightforward via props.conf on my indexers, but that first line with the datetime is in a different format; I want to use that and apply it to all subsequent lines. Little help!

Re: Linebreaking and event date in text file

woodcock — Tue, 01 Sep 2015 21:53:13 GMT

Can you have the timestamp added to the filename? If so, you can use datetime.xml easily to do this.

Re: Linebreaking and event date in text file

mcomfurf — Tue, 29 Sep 2020 07:09:54 GMT

Yes, the timestamp is in the filename: 08-30-2015_162551__summary.txt

Re: Linebreaking and event date in text file

woodcock — Wed, 02 Sep 2015 01:04:02 GMT

Based on this clarification:

Yes, the timestamp is in the filename: 08-30-2015_162551__summary.txt

You can do it like this:

$SPLUNK_HOME/etc/apps/MyApp/default/datetime.xml

<define name="datetimefromfile" extract="month, day, year, hour, minute, second">
   <text><![CDATA[source::.*?(\d{2})-(\d{2})-(\d{4})_(\d{2})(\d{2})(\d{2})]]></text>
</define>
<timePatterns>
   <use name="datetimefromfile"/>
</timePatterns>
<datePatterns>
   <use name="datetimefromfile"/>
</datePatterns>
</datetime>

$SPLUNK_HOME/etc/apps/MyApp/default/props.conf:

[YourSourcetypeHere]
DATETIME_CONFIG = /etc/apps/MyApp/default/datetime.xml