topic Re: How to write the regex for the removal of a header from a log file? in Splunk Search

How to write the regex for the removal of a header from a log file?

Bliide — Tue, 10 Feb 2015 00:03:32 GMT

I am trying to remove the header from a log file. I know that I need to put a stanza in props.conf on the forwarder and then create a transforms.conf like the following:

[skip_header_logfile]
REGEX = << 20-30 characters of your header line >>
DEST_KEY = queue
FORMAT = nullQueue

My problem is that my REGEX skills are horrible. My log file looks like this:

<Header>
  <Product>Microsoft SQL Server Reporting Services Version 2011.0110.2100.060 ((SQL11_RTM).120210-1917 )</Product>
  <Locale>English (United States)</Locale>
  <TimeZone>Central Standard Time</TimeZone>
  <Path>C:\Program Files\Microsoft SQL Server\MSRS11.MSSQLSERVER\Reporting Services\Logfiles\ReportServerService__01_06_2015_00_01_11.log</Path>
  <SystemName>SERVER</SystemName>
  <OSName>Microsoft Windows NT 6.1.7601 Service Pack 1</OSName>
  <OSVersion>6.1.7601</OSVersion>
  <ProcessID>10653</ProcessID>
  <Virtualization>None</Virtualization>
</Header>
library!WindowsService_112!195c!01/06/2015-00:01:11:: i INFO: Call to CleanBatch()

All that I need is REGEX that will select everything from to and of course include the header text, but I can not get it to work. Any help would be greatly appreciated.

Re: How to write the regex for the removal of a header from a log file?

sk314 — Tue, 10 Feb 2015 00:35:50 GMT

AFAIK, having regex expression go to nullQueue will discard entire event, not just the matched pattern. You should be looking for a SEDCMD.

Try this:

props.conf
[your_sourcetype]
SEDCMD-null = s/(?s)<Header>.*(?=<\/Header>)<\/Header>//

Having said so, proceed with caution as this works directly at index time and there is no way to get back lost data. I suggest you try the command on sample data before putting it into production.

Re: How to write the regex for the removal of a header from a log file?

Bliide — Tue, 10 Feb 2015 00:42:10 GMT

Thanks for the quick response, I will give it a shot and I am working with sample logs until I get it sorted.

Re: How to write the regex for the removal of a header from a log file?

Richfez — Sun, 01 Nov 2015 20:17:45 GMT

Converted this to an answer. Bliide, if you are still around could you check if this works and accept this answer, or otherwise comment back on what's still needed?