Getting Data In

props config for logs

Strangertinz
Path Finder

Hi Splunk Community, 

 

I am trying to create a props.conf for the sample log file below. 
My goal is to 
    * Delete the Header tag and remove the data from being ingested. 
    * Break the individual events by starting with ( "library!WindowsService_98!..." OR "processing!ReportServer_0-127!" )
    * Extracting time stamp such as ( "!11/26/2023-00:21:18::")

 

Heres the props.conf that I have so far but it is not working. 
---------

[sourcetype_name]

disabled = false
SHOULD_LINEMERGE = false
MAX_TIMESTAMP_LOOKAHEAD = 80
TIME_FORMAT = %m/%d/%Y-%H:%M:%S
LINE_BREAKER = ([\r\n]+)library!
SEDCMD-null = (<Header>([\s\S]*?)<\/Header>)

 

disabled



------------------- sample log file -------------------------
<Header>
<Product>Microsoft SQL Server Reporting Services Version 2007.0100.6000.029 ((Random_value).18802-2848 )</Product>
<Locale>English (United States)</Locale>
<TimeZone>Central Daylight Time</TimeZone>
<Path>C:\Program Files\Microsoft SQL Server\MSRS10.MSSQLSERVER\Reporting Services\Logfiles\ReportServerService__11_26_2023_00_00_01.log</Path>
<SystemName>hostName01</SystemName>
<OSName>Microsoft Windows NT 6.2.9200</OSName>
<OSVersion>6.2.9200</OSVersion>
<ProcessID>3088</ProcessID>
</Header>library!WindowsService_98!1234!11/26/2023-00:00:01:: i INFO: Call to CleanBatch()
library!WindowsService_98!1234!11/26/2023-00:00:01:: i INFO: Cleaned 0 batch records, 0 policies, 0 sessions, 0 cache entries, 0 snapshots, 0 chunks, 0 running jobs, 0 persisted streams, 0 segments, 0 segment mappings.
library!WindowsService_98!1234!11/26/2023-00:00:01:: i INFO: Call to CleanBatch() ends
library!WindowsService_98!1218!11/26/2023-00:10:01:: i INFO: Call to CleanBatch()
library!WindowsService_98!1218!11/26/2023-00:10:01:: i INFO: Cleaned 0 batch records, 0 policies, 1 sessions, 0 cache entries, 1 snapshots, 14 chunks, 0 running jobs, 0 persisted streams, 9 segments, 9 segment mappings.
library!WindowsService_98!1218!11/26/2023-00:10:01:: i INFO: Call to CleanBatch() ends
library!WindowsService_98!d00!11/26/2023-00:20:01:: i INFO: Call to CleanBatch()
library!WindowsService_98!d00!11/26/2023-00:20:01:: i INFO: Cleaned 0 batch records, 0 policies, 0 sessions, 0 cache entries, 0 snapshots, 0 chunks, 0 running jobs, 0 persisted streams, 0 segments, 0 segment mappings.
library!WindowsService_98!d00!11/26/2023-00:20:01:: i INFO: Call to CleanBatch() ends
library!ReportServer_0-127!2558!11/26/2023-00:21:18:: i INFO: RenderForNewSession('/Hampton.Common.Reports/BOL')
processing!ReportServer_0-127!2558!11/26/2023-00:21:18:: e ERROR: Throwing Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: There is no data for the field at position 19., ;
Info: Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: There is no data for the field at position 19.
processing!ReportServer_0-127!2558!11/26/2023-00:21:18:: e ERROR: Throwing Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: There is no data for the field at position 54., ;
Info: Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: There is no data for the field at position 54.
processing!ReportServer_0-127!2558!11/26/2023-00:21:18:: e ERROR: Throwing Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: There is no data for the field at position 61., ;
Info: Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: There is no data for the field at position 61.
processing!ReportServer_0-127!2558!11/26/2023-00:21:18:: e ERROR: Throwing Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: There is no data for the field at position 62., ;
Info: Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: There is no data for the field at position 62.
processing!ReportServer_0-127!2558!11/26/2023-00:21:19:: e ERROR: Throwing Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: There is no data for the field at position 1., ;
Info: Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: There is no data for the field at position 1.
processing!ReportServer_0-127!2558!11/26/2023-00:21:19:: e ERROR: Throwing Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: There is no data for the field at position 2., ;
Info: Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: There is no data for the field at position 2.
processing!ReportServer_0-127!2558!11/26/2023-00:21:19:: e ERROR: Throwing Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: There is no data for the field at position 1., ;
Info: Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: There is no data for the field at position 1.
processing!ReportServer_0-127!2558!11/26/2023-00:21:19:: e ERROR: Throwing Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: There is no data for the field at position 2., ;
Info: Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: There is no data for the field at position 2.
library!WindowsService_98!1234!11/26/2023-00:30:01:: i INFO: Call to CleanBatch()
------------------- sample log file end -------------------------

0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

richgalloway_0-1703263170243.png

 

---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

SEDCMD settings must contain either an s or y command not just a regex.

To properly extract a timestamp, the props stanza should contain TIME_PREFIX, TIME_FORMAT, and MAX_TIMESTAMP_LOOKAHEAD settings.

[sourcetype_name]
disabled = false
SHOULD_LINEMERGE = false
MAX_TIMESTAMP_LOOKAHEAD = 80
TIME_FORMAT = %m/%d/%Y-%H:%M:%S
TIME_PREFIX = \d!
LINE_BREAKER = ([\r\n]+)library!
SEDCMD-null = s/\<Header>[\s\S]*?\<\/Header>//g

You may have a problem with time zones, depending on the zones of the Splunk server and that in the data.  Ideally, the time zone should be specified as part of the timestamp rather than as a separate element.  The time zone should be a recognized abbreviation such as "CST" or "-0600".  BTW, Central Daylight Time is not in effect in November.

---
If this reply helps you, Karma would be appreciated.
0 Karma

Strangertinz
Path Finder

Hi @richgalloway

 

Thanks for getting back to me!


I tried the props.conf you proposed and got an error. Can you please try to upload the sample data with the sourcetype config you provided and see if you have any luck. 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

These settings may be cleaner, but I'm not sure what I'm trying to fix.

SHOULD_LINEMERGE=false
LINE_BREAKER=()library!
NO_BINARY_CHECK=true
TIME_FORMAT=%m/%d/%Y-%H:%M:%S
TIME_PREFIX=!\w{3,4}!
SEDCMD-null=s/\<Header>[\s\S]*?\<\/Header>//g
MAX_TIMESTAMP_LOOKAHEAD=80
---
If this reply helps you, Karma would be appreciated.
0 Karma

Strangertinz
Path Finder

@richgalloway 
I am still getting the same error. Are you able to copy the sample data and ingest it into Splunk to see the errors I am getting? 

Thanks

0 Karma

richgalloway
SplunkTrust
SplunkTrust

I did copy the example text and ingest it successfully.  I did not see the encoded text you see.

---
If this reply helps you, Karma would be appreciated.
0 Karma

Strangertinz
Path Finder

@richgalloway 

Can you share a picture of the sourcetype along with the Splunk web screenshot? I am still getting errors on my end.


Thanks

0 Karma

richgalloway
SplunkTrust
SplunkTrust

richgalloway_0-1703263170243.png

 

---
If this reply helps you, Karma would be appreciated.

Strangertinz
Path Finder

Thanks its fixed now

richgalloway
SplunkTrust
SplunkTrust

What was the error?

---
If this reply helps you, Karma would be appreciated.
0 Karma

Strangertinz
Path Finder

@richgalloway 

I am getting the error below. I can't even get Splunk to interpret the data as regular text. 

Screen Shot 2023-12-21 at 7.11.32 PM.png

0 Karma

Sridevi1
New Member

Please change the sourcetype and try

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...