How to create multiple Sourcetype for one source?

[davemarianne]

I have 4 different kind of logs that is coming from one source (sample logs are below). I would like to configure this in different sourcetypes so that the timestamps that Splunk will get is correct. My problem is they have different timestamp filed names and character count on where the time field are positioned.

A. It has timestamp coming from "time".

{ "count": 1, "total": 1, "minimum": 1, "maximum": 1, "average": 1, "resourceId": "KSJDIOU-43782JH3K28-28378KMK", "time": "2022-11-24T06:05:00.0000000Z", "metricName": "TotalBillable", "timeGrain": "MPT1DRIVE"}

B. It has timestamp coming from "EventTimestamp"

{ "Environment": "PROD", "Region": "SouthEast Asia", "ScaleUnit": "PRD-041", "TaskName": "ApplicationMetricsLog", "ActivityId": "89S7D-DS98-SDSDS", "SubscriptionId": "CKJD989897DS", "NamespaceName": "tm-uidso-prem-prd", "ActivityName": "ActiveConnections", "ResourceId": "KSJDIOU-43782JHFSDS3K28-28378KMK", "Outcome": "Success", "Protocol": "AMQP", "AuthType": "EntitySAS", "AuthId": "JKSDDI-55643", "NetworkType": "Public", "ClientIp": "1000.3425.0.2", "Count": 1, "Properties": "{\"EventTimestamp\":\"24/11/2022 06:10:05:7602\"}", "category": "MetricsLogs"}

C. It has timestamp coming from "time" but, time field is on a different character count from letter A.

{ "Deployment": "ksdjksdos1loio2klkl3", "time": "2022-11-24T06:04:00Z", "timeGrain": "GFT2KOIO", "resourceId": "KLSDASKOSO-3434-545-XCDS", "metricName": "GoStarted", "dimensions": "{\"Deployment\":\"767sd898ds8d9sdd9s\",\"Role\":\"maria.Home.upon\",\"RoleInstance\":\"maria.Home.upon_OUT_69\"}", "average": 1, "minimum": 1, "maximum": 1, "total": 1, "count": 1}

 
D.  It has timestamp coming from "time" but, time field is on a different character count from letter A and C.

{ "time": "2022-11-24T06:11:52.6825908Z", "resourceId": "dksjdks-sdsds-dsds-23232-3232s", "category": "FunctionLogs", "operationName": "Microsoft.Web/sites/functions/log", "level": "Informational", "location": "South America", "properties": {"appName":"func-dttysdvmj-eventstop-prd","roleInstance":"rollinginthedeep","message":"Response [sadlsad-d4343-dfsdf45-545dsd-sdsd] 200 OK (00.0s)\r\nETag:\"0xJYWEDFF6788DFSDF\"\r\nServer:Windows-Azure-Blob/1.0,Microsoft-HTTPAPI/2.0\r\nx-ms-request-id:dsds-8000000\r\nx-ms-client-request-id:sdsdsd0-dsdsdgfr1-454346fd76767gf\r\nx-ms-version:2020-08-04\r\nx-ms-lease-id:b51368e2-2d24-6c77-acab-78ced4658e79\r\nDate:Thu, 24 Nov 2022 06:11:52 GMT\r\nContent-Length:0\r\nLast-Modified:Mon, 17 Oct 2022 09:59:09 GMT\r\n","category":"Azure.Core.1","hostVersion":"467888.134263.2.1990097","hostInstanceId":"d57fdu6-kkew36-0000-dsf3-rgtty887gd","level":"Information","levelId":2,"processId":5976,"eventId":5,"eventName":"Response"}}

 
Thanks in advance.

[isoutamo]

Hi

if all those variations are in one file then you actually have only one source. Usually this means also one sourcetype. You have (at least) three options to fix it.

1. Ask that developers write those different events to separate log files
2. Use e.g. ingest_eval on props.conf to select correct timestamp field to _time
3. Use props and transforms to change sourcetypes based on timestamp field

There should be quite many examples for last two options in community,

r. Ismo

[davemarianne]

Hi Isoutamo, they are actually in different source file but their location is in one source path. Is it possible to separate them in different sourcetypes?

[isoutamo]

What you are meaning by “different file but same path”? If those are separate files (file name) on same directory, then just assign own sourcetype for each file type. Just do this on your inputs.conf.