topic Re: Why is line breaking inconsistent - File Monitoring - Roxio SecureBurn Log file - .txt in Getting Data In

Why is line breaking inconsistent - File Monitoring - Roxio SecureBurn Log file - .txt

danielansell — Thu, 07 Mar 2019 19:45:02 GMT

Everytime a CD is burned with Roxio SecureBurn, a txt file log of the cd is created. The format of the .txt log file is:

Date: Thu Mar 7 13:47:00 2019
Computer Name: ComputerName01
User Name: domain.accountname

Project includes 1 folder(s) and 2 file(s)

C:\Users\accountname\Desktop\TransferFolder\file1.txt e69f78a887b(rest of file hash)a35 3764543bytes 2019/3/7 08:13:15
C:\Users\accountname\Desktop\TransferFolder\file2.txt e69f78a887b(rest of file hash)a35 7764543bytes 2019/3/7 08:13:18
END OF FILE

My props.conf for the sourcetype I created includes:
SHOULD_LINEMERGE = false

Linebreaking is occurring inconsistently. Some of my events show up where each line is its own event, others include every bit of data in the .txt file as its own event. Any ideas - perhaps a more bulletproof way to force a break? Do Windows .txt files normally have inconsistancies with a carriage return or line breaking?

Re: Why is line breaking inconsistent - File Monitoring - Roxio SecureBurn Log file - .txt

nickhills — Fri, 08 Mar 2019 11:49:06 GMT

Forgive my ignorance - does each burn job result in a new file, or all all the job logs written into the same log?

Re: Why is line breaking inconsistent - File Monitoring - Roxio SecureBurn Log file - .txt

FrankVl — Fri, 08 Mar 2019 12:25:56 GMT

When you set SHOULD_LINEMERGE = false you also need to specify a linebreaker. And while you're at it, it is always good to specify time format config explicitly as well.

So, try adding this to your props.conf:

LINE_BREAKER = ([\r\n]*)Date:\s+\w+\s+\w+\s+\d+
TIME_PREFIX = Date:\s+
MAX_TIMESTAMP_LOOKAHEAD = 25
TIME_FORMAT = %a %b %e %H:%M:%S %Y

Re: Why is line breaking inconsistent - File Monitoring - Roxio SecureBurn Log file - .txt

danielansell — Fri, 08 Mar 2019 12:37:44 GMT

Each job results in a new file. As such, I intend to use the source field as a means to provide meaningful data (using either a transaction, or "by" to group data).

Re: Why is line breaking inconsistent - File Monitoring - Roxio SecureBurn Log file - .txt

danielansell — Fri, 08 Mar 2019 12:48:42 GMT

The ultimate goal is actually to extract data from each file burned to CD. When each .txt file is returned as a single event, I tried to extract data though using the rex command. The first result is stored in my "FileSize" field, but it does not continue on to the remaining lines.

So alternatively, if there is a more appropriate way to populate a FileSize, FileName, etc. field from each line, without breaking the event into single line events, that would be great as well.

Re: Why is line breaking inconsistent - File Monitoring - Roxio SecureBurn Log file - .txt

danielansell — Fri, 08 Mar 2019 13:01:31 GMT

I added to my props.conf and did not see a change in behavior. I'm still seeing the same - one of the .txt files has been split to 10 events where each line is its own event; while 2 other .txt files are still being returned as a multi-line event.

As I understand Splunk, changes in line breaking are applied at search time, correct?

Could there be an issue in how the .txt file is formatted?
I'm generally aware that there are some quirks with how Notepad.exe handles new lines (as evidenced by trying to modify any Splunk app in Notepad - there are never any linebreaks). Because of that I use another tool that handles the new lines appropriately - are there any tools (e.g. notepad++) that could identify what type of new line is present in my .txt file?

Re: Why is line breaking inconsistent - File Monitoring - Roxio SecureBurn Log file - .txt

FrankVl — Fri, 08 Mar 2019 13:03:15 GMT

That sounds like a matter of setting max_matches=0 in your rex command. By default the rex command stops after finding 1 match.

Re: Why is line breaking inconsistent - File Monitoring - Roxio SecureBurn Log file - .txt

danielansell — Fri, 08 Mar 2019 13:15:24 GMT

I typically use rex to test my extractions before building my field extractions in my props.conf.
Will a field extraction defined in a props.conf return multiple values - should a max_matches=0 be added to my props.conf?

If so, will I be able to perform stats functions on the multivalued field? For example, if I were to define a FileSize field in my props.conf as an extraction, and I'd like to return a total size for the burn job, would I be able to do the following:
...search... |eval DiscSize=sum(FileSize) by source

Re: Why is line breaking inconsistent - File Monitoring - Roxio SecureBurn Log file - .txt

FrankVl — Fri, 08 Mar 2019 13:34:08 GMT

You can use a REPORT extraction and configure the corresponding transforms.conf settings with MV_ADD=true.

Yes, something like that would work. Although the correct syntax would be: | stats sum(FileSize) by source not eval 🙂

But it might also be valid to split this into single line events upon indexing, as multi valued fields can be a bit difficult to work with sometimes. But then you will have to find a way of dealing with those header lines of the file (if at all interesting).

Re: Why is line breaking inconsistent - File Monitoring - Roxio SecureBurn Log file - .txt

FrankVl — Fri, 08 Mar 2019 13:34:41 GMT

Linebreaking happens at indextime. So to see the changes take effect, you need to restart and ingest fresh sample files.

Re: Why is line breaking inconsistent - File Monitoring - Roxio SecureBurn Log file - .txt

danielansell — Fri, 08 Mar 2019 16:18:32 GMT

That's my problem. I was expecting the props config to be interpreted at search time. The event I have that is broken up correctly was indexed after my props change. I didn't put two and two together with that.

I think I'll re-index the data and see if that serves my needs.

Thanks for the help - to include up above with the stats versus eval.

Re: Why is line breaking inconsistent - File Monitoring - Roxio SecureBurn Log file - .txt

mfw113 — Fri, 17 Jan 2020 21:59:16 GMT

Were you able to extract the file names and sizes from these logs, if so would you be willing to share?

Re: Why is line breaking inconsistent - File Monitoring - Roxio SecureBurn Log file - .txt

danielansell — Tue, 21 Jan 2020 21:38:06 GMT

For file details, my extraction looks like this:

(?P<FileName>.+)\s(?P<FileHash>[0-9a-fA-F]{40})\s+(?P<FileSize>\d+)bytes\s+(?P<FileModDate>\d{4}\/\d+\/\d+\s\d+:\d+:\d+