topic Re: props config for logs in Getting Data In

props config for logs

Strangertinz — Thu, 21 Dec 2023 20:04:23 GMT

Hi Splunk Community,

I am trying to create a props.conf for the sample log file below.
My goal is to
* Delete the Header tag and remove the data from being ingested.
* Break the individual events by starting with ( "library!WindowsService_98!..." OR "processing!ReportServer_0-127!" )
* Extracting time stamp such as ( "!11/26/2023-00:21:18::")

Heres the props.conf that I have so far but it is not working.
---------

[sourcetype_name]

disabled = false
SHOULD_LINEMERGE = false
MAX_TIMESTAMP_LOOKAHEAD = 80
TIME_FORMAT = %m/%d/%Y-%H:%M:%S
LINE_BREAKER = ([\r\n]+)library!
SEDCMD-null = (<Header>([\s\S]*?)<\/Header>)

disabled

------------------- sample log file -------------------------
<Header>
<Product>Microsoft SQL Server Reporting Services Version 2007.0100.6000.029 ((Random_value).18802-2848 )</Product>
<Locale>English (United States)</Locale>
<TimeZone>Central Daylight Time</TimeZone>
<Path>C:\Program Files\Microsoft SQL Server\MSRS10.MSSQLSERVER\Reporting Services\Logfiles\ReportServerService__11_26_2023_00_00_01.log</Path>
<SystemName>hostName01</SystemName>
<OSName>Microsoft Windows NT 6.2.9200</OSName>
<OSVersion>6.2.9200</OSVersion>
<ProcessID>3088</ProcessID>
</Header>library!WindowsService_98!1234!11/26/2023-00:00:01:: i INFO: Call to CleanBatch()
library!WindowsService_98!1234!11/26/2023-00:00:01:: i INFO: Cleaned 0 batch records, 0 policies, 0 sessions, 0 cache entries, 0 snapshots, 0 chunks, 0 running jobs, 0 persisted streams, 0 segments, 0 segment mappings.
library!WindowsService_98!1234!11/26/2023-00:00:01:: i INFO: Call to CleanBatch() ends
library!WindowsService_98!1218!11/26/2023-00:10:01:: i INFO: Call to CleanBatch()
library!WindowsService_98!1218!11/26/2023-00:10:01:: i INFO: Cleaned 0 batch records, 0 policies, 1 sessions, 0 cache entries, 1 snapshots, 14 chunks, 0 running jobs, 0 persisted streams, 9 segments, 9 segment mappings.
library!WindowsService_98!1218!11/26/2023-00:10:01:: i INFO: Call to CleanBatch() ends
library!WindowsService_98!d00!11/26/2023-00:20:01:: i INFO: Call to CleanBatch()
library!WindowsService_98!d00!11/26/2023-00:20:01:: i INFO: Cleaned 0 batch records, 0 policies, 0 sessions, 0 cache entries, 0 snapshots, 0 chunks, 0 running jobs, 0 persisted streams, 0 segments, 0 segment mappings.
library!WindowsService_98!d00!11/26/2023-00:20:01:: i INFO: Call to CleanBatch() ends
library!ReportServer_0-127!2558!11/26/2023-00:21:18:: i INFO: RenderForNewSession('/Hampton.Common.Reports/BOL')
processing!ReportServer_0-127!2558!11/26/2023-00:21:18:: e ERROR: Throwing Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: There is no data for the field at position 19., ;
Info: Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: There is no data for the field at position 19.
processing!ReportServer_0-127!2558!11/26/2023-00:21:18:: e ERROR: Throwing Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: There is no data for the field at position 54., ;
Info: Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: There is no data for the field at position 54.
processing!ReportServer_0-127!2558!11/26/2023-00:21:18:: e ERROR: Throwing Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: There is no data for the field at position 61., ;
Info: Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: There is no data for the field at position 61.
processing!ReportServer_0-127!2558!11/26/2023-00:21:18:: e ERROR: Throwing Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: There is no data for the field at position 62., ;
Info: Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: There is no data for the field at position 62.
processing!ReportServer_0-127!2558!11/26/2023-00:21:19:: e ERROR: Throwing Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: There is no data for the field at position 1., ;
Info: Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: There is no data for the field at position 1.
processing!ReportServer_0-127!2558!11/26/2023-00:21:19:: e ERROR: Throwing Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: There is no data for the field at position 2., ;
Info: Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: There is no data for the field at position 2.
processing!ReportServer_0-127!2558!11/26/2023-00:21:19:: e ERROR: Throwing Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: There is no data for the field at position 1., ;
Info: Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: There is no data for the field at position 1.
processing!ReportServer_0-127!2558!11/26/2023-00:21:19:: e ERROR: Throwing Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: There is no data for the field at position 2., ;
Info: Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: There is no data for the field at position 2.
library!WindowsService_98!1234!11/26/2023-00:30:01:: i INFO: Call to CleanBatch()
------------------- sample log file end -------------------------

Re: props config for logs

richgalloway — Thu, 21 Dec 2023 20:22:35 GMT

SEDCMD settings must contain either an s or y command not just a regex.

To properly extract a timestamp, the props stanza should contain TIME_PREFIX, TIME_FORMAT, and MAX_TIMESTAMP_LOOKAHEAD settings.

[sourcetype_name] disabled = false SHOULD_LINEMERGE = false MAX_TIMESTAMP_LOOKAHEAD = 80 TIME_FORMAT = %m/%d/%Y-%H:%M:%S TIME_PREFIX = \d! LINE_BREAKER = ([\r\n]+)library! SEDCMD-null = s/\<Header>[\s\S]*?\<\/Header>//g

You may have a problem with time zones, depending on the zones of the Splunk server and that in the data. Ideally, the time zone should be specified as part of the timestamp rather than as a separate element. The time zone should be a recognized abbreviation such as "CST" or "-0600". BTW, Central Daylight Time is not in effect in November.

Re: props config for logs

Strangertinz — Thu, 21 Dec 2023 22:49:25 GMT

Hi @richgalloway,

Thanks for getting back to me!

I tried the props.conf you proposed and got an error. Can you please try to upload the sample data with the sourcetype config you provided and see if you have any luck.

Re: props config for logs

richgalloway — Fri, 22 Dec 2023 01:05:35 GMT

What was the error?

Re: props config for logs

Strangertinz — Fri, 22 Dec 2023 01:14:34 GMT

@richgalloway

I am getting the error below. I can't even get Splunk to interpret the data as regular text.

Re: props config for logs

richgalloway — Fri, 22 Dec 2023 01:19:05 GMT

These settings may be cleaner, but I'm not sure what I'm trying to fix.

SHOULD_LINEMERGE=false LINE_BREAKER=()library! NO_BINARY_CHECK=true TIME_FORMAT=%m/%d/%Y-%H:%M:%S TIME_PREFIX=!\w{3,4}! SEDCMD-null=s/\<Header>[\s\S]*?\<\/Header>//g MAX_TIMESTAMP_LOOKAHEAD=80

Re: props config for logs

Strangertinz — Fri, 22 Dec 2023 01:35:28 GMT

@richgalloway
I am still getting the same error. Are you able to copy the sample data and ingest it into Splunk to see the errors I am getting?

Thanks

Re: props config for logs

richgalloway — Fri, 22 Dec 2023 13:09:05 GMT

I did copy the example text and ingest it successfully. I did not see the encoded text you see.

Re: props config for logs

Strangertinz — Fri, 22 Dec 2023 16:33:06 GMT

@richgalloway

Can you share a picture of the sourcetype along with the Splunk web screenshot? I am still getting errors on my end.

Thanks

Re: props config for logs

richgalloway — Fri, 22 Dec 2023 16:39:39 GMT

Re: props config for logs

Strangertinz — Fri, 22 Dec 2023 17:44:26 GMT

Thanks its fixed now

Re: props config for logs

Sridevi1 — Tue, 07 May 2024 14:14:36 GMT

Please change the sourcetype and try