Getting Data In

Linebreaking and event date in text file

mcomfurf
Path Finder

We have a text file to log row counts for CSV files used in an ETL job. The format of the file is like this:

08/30/15 16:08:51
acme_file.csv: 0
coyote_file.csv: 0
anvil_file.csv: 88835
roadrunner_file.csv: 330333

I want to break this into one line per filename from line 2 onward, into fields {filename}.csv: {rowcount} Note that the first line in the .txt file is the datetime for all of the events.

I think the linebreaking is fairly straightforward via props.conf on my indexers, but that first line with the datetime is in a different format; I want to use that and apply it to all subsequent lines. Little help!

0 Karma

woodcock
Esteemed Legend

Based on this clarification:

Yes, the timestamp is in the filename: 08-30-2015_162551__summary.txt

You can do it like this:

$SPLUNK_HOME/etc/apps/MyApp/default/datetime.xml

<define name="datetimefromfile" extract="month, day, year, hour, minute, second">
   <text><![CDATA[source::.*?(\d{2})-(\d{2})-(\d{4})_(\d{2})(\d{2})(\d{2})]]></text>
</define>
<timePatterns>
   <use name="datetimefromfile"/>
</timePatterns>
<datePatterns>
   <use name="datetimefromfile"/>
</datePatterns>
</datetime>

$SPLUNK_HOME/etc/apps/MyApp/default/props.conf:

[YourSourcetypeHere]
DATETIME_CONFIG = /etc/apps/MyApp/default/datetime.xml
0 Karma

woodcock
Esteemed Legend

Can you have the timestamp added to the filename? If so, you can use datetime.xml easily to do this.

0 Karma

mcomfurf
Path Finder

Yes, the timestamp is in the filename: 08-30-2015_162551__summary.txt

0 Karma
Get Updates on the Splunk Community!

Splunk Platform | Upgrading your Splunk Deployment to Python 3.9

Splunk initially announced the removal of Python 2 during the release of Splunk Enterprise 8.0.0, aiming to ...

From Product Design to User Insights: Boosting App Developer Identity on Splunkbase

co-authored by Yiyun Zhu & Dan Hosaka Engaging with the Community at .conf24 At .conf24, we revitalized the ...

Detect and Resolve Issues in a Kubernetes Environment

We’ve gone through common problems one can encounter in a Kubernetes environment, their impacts, and the ...