Getting Data In

How to create multiple Sourcetype for one source?

davemarianne
Engager

I have 4 different kind of logs that is coming from one source (sample logs are below). I would like to configure this in different sourcetypes so that the timestamps that Splunk will get is correct. My problem is they have different timestamp filed names and character count on where the time field are positioned.

A. It has timestamp coming from "time".

 

 

{ "count": 1, "total": 1, "minimum": 1, "maximum": 1, "average": 1, "resourceId": "KSJDIOU-43782JH3K28-28378KMK", "time": "2022-11-24T06:05:00.0000000Z", "metricName": "TotalBillable", "timeGrain": "MPT1DRIVE"}

 

 


B. It has timestamp coming from "EventTimestamp"

 

 

{ "Environment": "PROD", "Region": "SouthEast Asia", "ScaleUnit": "PRD-041", "TaskName": "ApplicationMetricsLog", "ActivityId": "89S7D-DS98-SDSDS", "SubscriptionId": "CKJD989897DS", "NamespaceName": "tm-uidso-prem-prd", "ActivityName": "ActiveConnections", "ResourceId": "KSJDIOU-43782JHFSDS3K28-28378KMK", "Outcome": "Success", "Protocol": "AMQP", "AuthType": "EntitySAS", "AuthId": "JKSDDI-55643", "NetworkType": "Public", "ClientIp": "1000.3425.0.2", "Count": 1, "Properties": "{\"EventTimestamp\":\"24/11/2022 06:10:05:7602\"}", "category": "MetricsLogs"}

 

 


C. It has timestamp coming from "time" but, time field is on a different character count from letter A.

 

 

{ "Deployment": "ksdjksdos1loio2klkl3", "time": "2022-11-24T06:04:00Z", "timeGrain": "GFT2KOIO", "resourceId": "KLSDASKOSO-3434-545-XCDS", "metricName": "GoStarted", "dimensions": "{\"Deployment\":\"767sd898ds8d9sdd9s\",\"Role\":\"maria.Home.upon\",\"RoleInstance\":\"maria.Home.upon_OUT_69\"}", "average": 1, "minimum": 1, "maximum": 1, "total": 1, "count": 1}

 

 

 
D.  It has timestamp coming from "time" but, time field is on a different character count from letter A and C.

 

 

{ "time": "2022-11-24T06:11:52.6825908Z", "resourceId": "dksjdks-sdsds-dsds-23232-3232s", "category": "FunctionLogs", "operationName": "Microsoft.Web/sites/functions/log", "level": "Informational", "location": "South America", "properties": {"appName":"func-dttysdvmj-eventstop-prd","roleInstance":"rollinginthedeep","message":"Response [sadlsad-d4343-dfsdf45-545dsd-sdsd] 200 OK (00.0s)\r\nETag:\"0xJYWEDFF6788DFSDF\"\r\nServer:Windows-Azure-Blob/1.0,Microsoft-HTTPAPI/2.0\r\nx-ms-request-id:dsds-8000000\r\nx-ms-client-request-id:sdsdsd0-dsdsdgfr1-454346fd76767gf\r\nx-ms-version:2020-08-04\r\nx-ms-lease-id:b51368e2-2d24-6c77-acab-78ced4658e79\r\nDate:Thu, 24 Nov 2022 06:11:52 GMT\r\nContent-Length:0\r\nLast-Modified:Mon, 17 Oct 2022 09:59:09 GMT\r\n","category":"Azure.Core.1","hostVersion":"467888.134263.2.1990097","hostInstanceId":"d57fdu6-kkew36-0000-dsf3-rgtty887gd","level":"Information","levelId":2,"processId":5976,"eventId":5,"eventName":"Response"}}

 

 

 
Thanks in advance.

Labels (3)
0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

if all those variations are in one file then you actually have only one source. Usually this means also one sourcetype. You have (at least) three options to fix it.

  1. Ask that developers write those different events to separate log files
  2. Use e.g. ingest_eval on props.conf to select correct timestamp field to _time
  3. Use props and transforms to change sourcetypes based on timestamp field

There should be quite many examples for last two options in community,

r. Ismo

 

davemarianne
Engager

Hi Isoutamo, they are actually in different source file but their location is in one source path. Is it possible to separate them in different sourcetypes?

0 Karma

isoutamo
SplunkTrust
SplunkTrust

What you are meaning by “different file but same path”? If those are separate files (file name) on same directory, then just assign own sourcetype for each file type. Just do this on your inputs.conf. 

0 Karma
Get Updates on the Splunk Community!

Set Up More Secure Configurations in Splunk Enterprise With Config Assist

This blog post is part 3 of 4 of a series on Splunk Assist. Click the links below to see the other ...

Observability Highlights | November 2022 Newsletter

 November 2022Observability CloudEnd Of Support Extension for SignalFx Smart AgentSplunk is extending the End ...

Enterprise Security Content Update (ESCU) v3.54.0

The Splunk Threat Research Team (STRT) recently released Enterprise Security Content Update (ESCU) v3.54.0 and ...