topic Re: How do I edit my props.conf for proper line breaking of my sample CSV log file? in Getting Data In

How do I edit my props.conf for proper line breaking of my sample CSV log file?

dcascione — Mon, 20 Jun 2016 17:59:16 GMT

I have a simple .csv log file that I'm trying to break with:

[software_summary]
LINE_BREAKER  = ([\r\n]+)
SHOULD_LINEMERGE = false

Here is a sample of the log:

Back to Index,
HOST INFORMATION,
Software build-2718055,10
Software build-3116895,15
Software build-2583090,35
Software 5.5.0 build-1746974,22

The two fields I'm interested in which are comma delimited are Software Build and Count. I'd like to see each line break out into its own log file. Thanks !!

Re: How do I edit my props.conf for proper line breaking of my sample CSV log file?

muebel — Tue, 29 Sep 2020 10:01:11 GMT

HI dcascione, I think a good angle on this would be to checkout the structured data options in props described here : http://docs.splunk.com/Documentation/Splunk/6.4.1/admin/Propsconf#Structured_Data_Header_Extraction_and_configuration

Essentially you could define FIELD_NAMES config to define the software_build and count fields, and PREAMBLE_REGEX config to disregard the initial couple lines

To stick with the initial idea of breaking out the events, it seems that your config should be effective for treating each line as an event (SHOULD_LINEMERGE=false)

One issue here could be that this config is being set on a universal forwarder, which wouldn't do linebreaking. Props definitions would need to be put on the upstream HF or Indexer.

Please let me know if this answers your question!

Re: How do I edit my props.conf for proper line breaking of my sample CSV log file?

dcascione — Mon, 20 Jun 2016 19:16:25 GMT

I was hoping to just line break the file in the props.conf and then build the field extractions using the UI - Thanks

Re: How do I edit my props.conf for proper line breaking of my sample CSV log file?

dcascione — Mon, 20 Jun 2016 19:44:04 GMT

According to the documentation, the stanza I added to the props.conf which includes (([\r\n]+), ) should break out each line into an event...Not sure why this is not working?

Defaults to ([\r\n]+), meaning data is broken into an event for each line, delimited by any number of carriage return or newline characters.

Re: How do I edit my props.conf for proper line breaking of my sample CSV log file?

ryanoconnor — Mon, 20 Jun 2016 19:55:36 GMT

The biggest issue I see with this file is that it's poorly formatted and not truly a well-formatted CSV file. See the screenshot below of when I saved the sample text you sent, saved it as software_summary.csv and opened it in Numbers on my mac.

If you did have it as a CSV then your props.conf would also include INDEXED_EXTRACTIONS = CSV and would handle this file much easier.

Re: How do I edit my props.conf for proper line breaking of my sample CSV log file?

muebel — Mon, 20 Jun 2016 20:01:34 GMT

I edited my original issue to address the event breaking difficulty

Re: How do I edit my props.conf for proper line breaking of my sample CSV log file?

dcascione — Tue, 29 Sep 2020 09:58:48 GMT

Yes the source file is a .csv - I just added the csv reference to the stanza - hopefully this will work! Thanks for the tip!!
[software_summary]
INDEXED_EXTRACTIONS = CSV
LINE_BREAKER = ([\r\n]+)
SHOULD_LINEMERGE = false

Re: How do I edit my props.conf for proper line breaking of my sample CSV log file?

ryanoconnor — Mon, 20 Jun 2016 20:16:49 GMT

Can you also modify the first couple lines? the CSV indexed extractions work best if the first line is a header for the CSV file.

 Back to Index,
 HOST INFORMATION,

Should be

 software_build, count

So your CSV would look more like

 software_build, count
 Software build-2718055,10
 Software build-3116895,15
 Software build-2583090,35
 Software 5.5.0 build-1746974,22

Re: How do I edit my props.conf for proper line breaking of my sample CSV log file?

dcascione — Mon, 20 Jun 2016 20:18:19 GMT

Will the change to the props.conf line break the pre-existing log files within the index, or just the new logs that are being ingested after the change?

Re: How do I edit my props.conf for proper line breaking of my sample CSV log file?

dcascione — Mon, 20 Jun 2016 20:30:40 GMT

I just reached out to the team who generates the logs to see if they can remove the very first "HOST INFORMATION" line. Thanks!

Re: How do I edit my props.conf for proper line breaking of my sample CSV log file?

muebel — Mon, 20 Jun 2016 20:37:06 GMT

event breaking is done at index time, more info here : http://docs.splunk.com/Documentation/Splunk/6.4.1/Admin/Configurationparametersandthedatapipeline

The already indexed events won't change.

Re: How do I edit my props.conf for proper line breaking of my sample CSV log file?

ryanoconnor — Mon, 20 Jun 2016 20:43:37 GMT

If possible, can you clean that index and re-index the file after you've made that change? Can you also post the updated version of the csv file once they've changed it so we can confirm it looks correct?

Thank you,
Ryan