How to create multiple Sourcetype for one source?

davemarianne — Thu, 24 Nov 2022 06:30:05 GMT

I have 4 different kind of logs that is coming from one source (sample logs are below). I would like to configure this in different sourcetypes so that the timestamps that Splunk will get is correct. My problem is they have different timestamp filed names and character count on where the time field are positioned.

A. It has timestamp coming from "time".

{ "count": 1, "total": 1, "minimum": 1, "maximum": 1, "average": 1, "resourceId": "KSJDIOU-43782JH3K28-28378KMK", "time": "2022-11-24T06:05:00.0000000Z", "metricName": "TotalBillable", "timeGrain": "MPT1DRIVE"}

B. It has timestamp coming from "EventTimestamp"

{ "Environment": "PROD", "Region": "SouthEast Asia", "ScaleUnit": "PRD-041", "TaskName": "ApplicationMetricsLog", "ActivityId": "89S7D-DS98-SDSDS", "SubscriptionId": "CKJD989897DS", "NamespaceName": "tm-uidso-prem-prd", "ActivityName": "ActiveConnections", "ResourceId": "KSJDIOU-43782JHFSDS3K28-28378KMK", "Outcome": "Success", "Protocol": "AMQP", "AuthType": "EntitySAS", "AuthId": "JKSDDI-55643", "NetworkType": "Public", "ClientIp": "1000.3425.0.2", "Count": 1, "Properties": "{\"EventTimestamp\":\"24/11/2022 06:10:05:7602\"}", "category": "MetricsLogs"}

C. It has timestamp coming from "time" but, time field is on a different character count from letter A.

{ "Deployment": "ksdjksdos1loio2klkl3", "time": "2022-11-24T06:04:00Z", "timeGrain": "GFT2KOIO", "resourceId": "KLSDASKOSO-3434-545-XCDS", "metricName": "GoStarted", "dimensions": "{\"Deployment\":\"767sd898ds8d9sdd9s\",\"Role\":\"maria.Home.upon\",\"RoleInstance\":\"maria.Home.upon_OUT_69\"}", "average": 1, "minimum": 1, "maximum": 1, "total": 1, "count": 1}

D. It has timestamp coming from "time" but, time field is on a different character count from letter A and C.

{ "time": "2022-11-24T06:11:52.6825908Z", "resourceId": "dksjdks-sdsds-dsds-23232-3232s", "category": "FunctionLogs", "operationName": "Microsoft.Web/sites/functions/log", "level": "Informational", "location": "South America", "properties": {"appName":"func-dttysdvmj-eventstop-prd","roleInstance":"rollinginthedeep","message":"Response [sadlsad-d4343-dfsdf45-545dsd-sdsd] 200 OK (00.0s)\r\nETag:\"0xJYWEDFF6788DFSDF\"\r\nServer:Windows-Azure-Blob/1.0,Microsoft-HTTPAPI/2.0\r\nx-ms-request-id:dsds-8000000\r\nx-ms-client-request-id:sdsdsd0-dsdsdgfr1-454346fd76767gf\r\nx-ms-version:2020-08-04\r\nx-ms-lease-id:b51368e2-2d24-6c77-acab-78ced4658e79\r\nDate:Thu, 24 Nov 2022 06:11:52 GMT\r\nContent-Length:0\r\nLast-Modified:Mon, 17 Oct 2022 09:59:09 GMT\r\n","category":"Azure.Core.1","hostVersion":"467888.134263.2.1990097","hostInstanceId":"d57fdu6-kkew36-0000-dsf3-rgtty887gd","level":"Information","levelId":2,"processId":5976,"eventId":5,"eventName":"Response"}}

Thanks in advance.

Re: How to create multiple Sourcetype for one source?

isoutamo — Thu, 24 Nov 2022 06:51:06 GMT

if all those variations are in one file then you actually have only one source. Usually this means also one sourcetype. You have (at least) three options to fix it.

Ask that developers write those different events to separate log files
Use e.g. ingest_eval on props.conf to select correct timestamp field to _time
Use props and transforms to change sourcetypes based on timestamp field

There should be quite many examples for last two options in community,

r. Ismo

Re: How to create multiple Sourcetype for one source?

davemarianne — Thu, 24 Nov 2022 06:54:56 GMT

Hi Isoutamo, they are actually in different source file but their location is in one source path. Is it possible to separate them in different sourcetypes?

Re: How to create multiple Sourcetype for one source?

isoutamo — Thu, 24 Nov 2022 15:15:43 GMT

What you are meaning by “different file but same path”? If those are separate files (file name) on same directory, then just assign own sourcetype for each file type. Just do this on your inputs.conf.

topic How to create multiple Sourcetype for one source? in Getting Data In

How to create multiple Sourcetype for one source?

Re: How to create multiple Sourcetype for one source?

Re: How to create multiple Sourcetype for one source?

Re: How to create multiple Sourcetype for one source?