Splunk Data Stream Processor

What is the best way to get field from raw data?

Marttin123
New Member

how can i get date data in fields to use the splunk features.

What is the best way for it. The raw data are coming from a kafka stream:

e.g.:

  • MAX_CURRENT
  • BLOCK_POSITION_NUMBER
  • OPERATING_DISTANCE_KM

    row data:

{"body":"{\"namespace\":\"xxx.messages.sensor.sensortelemetrymessage\",\"payload\":{\"dataSource\":{\"endpointUrl\":\"opc.tcp://123.164.72.6:4840/\",\"id\":\"\",\"name\":\"\",\"route\":\"\"},\"data\":{\"key\":\"ns=3;s=\\\"TEST_CARRIER_DB\\\".\\\"CARRIER\\\"[105]\",\"value\":[{\"key\":\"TIMESTAMP\",\"value\":\"0001-01-01T00:00:00Z\",\"dataType\":\"DateTime\"},{\"key\":\"PLC_CARRIER\",\"value\":[{\"key\":\"OPERATION_MODE\",\"value\":\"0\",\"dataType\":\"Int16\"},{\"key\":\"TEMP_CABINET\",\"value\":\"0\",\"dataType\":\"Int16\"},{\"key\":\"F_PROG_SIG\",\"value\":[{\"key\":\"0\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"1\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"2\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"3\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"4\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"5\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"6\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"7\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"8\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"9\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"10\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"11\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"12\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"13\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"14\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"15\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"16\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"17\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"18\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"19\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"20\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"21\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"22\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"23\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"24\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"25\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"26\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"27\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"28\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"29\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"30\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"31\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"32\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"33\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"34\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"35\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"36\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"37\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"38\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"39\",\"value\":\"32\",\"dataType\":\"Byte\"}],\"dataType\":\"ByteCollection\"},{\"key\":\"PROG_DAT\",\"value\":[{\"key\":\"0\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"1\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"2\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"3\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"4\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"5\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"6\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"7\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"8\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"9\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"10\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"11\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"12\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"13\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"14\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"15\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"16\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"17\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"18\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"19\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"20\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"21\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"22\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"23\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"24\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"25\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"26\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"27\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"28\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"29\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"30\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"31\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"32\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"33\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"34\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"35\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"36\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"37\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"38\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"39\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"40\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"41\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"42\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"43\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"44\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"45\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"46\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"47\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"48\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"49\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"50\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"51\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"52\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"53\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"54\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"55\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"56\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"57\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"58\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"59\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"60\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"61\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"62\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"63\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"64\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"65\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"66\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"67\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"68\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"69\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"70\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"71\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"72\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"73\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"74\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"75\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"76\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"77\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"78\",\"value\":\"32\",\"dataType\":\"Byte\"},{\"key\":\"79\",\"value\":\"32\",\"dataType\":\"Byte\"}],\"dataType\":\"ByteCollection\"},{\"key\":\"CYCLETIME_AVERAGE\",\"value\":\"0\",\"dataType\":\"Int16\"}],\"dataType\":\"Struct<TEST_PLC_CARRIER_UDT>\"},{\"key\":\"CARRIER\",\"value\":[{\"key\":\"BLOCK_POSITION_NUMBER\",\"value\":\"0\",\"dataType\":\"Int16\"},{\"key\":\"OPERATING_HOURS\",\"value\":\"0\",\"dataType\":\"Int32\"}],\"dataType\":\"Struct<TEST_CARRIER_UDT>\"},{\"key\":\"DRIVE\",\"value\":[{\"key\":\"BMK\",\"value\":\"0\",\"dataType\":\"Int16\"},{\"key\":\"POSITION\",\"value\":\"0\",\"dataType\":\"Int32\"},{\"key\":\"SPEED\",\"value\":\"0\",\"dataType\":\"Int16\"},{\"key\":\"ACT_CURRENT\",\"value\":\"0\",\"dataType\":\"Int16\"},{\"key\":\"MAX_CURRENT\",\"value\":\"0\",\"dataType\":\"Int16\"},{\"key\":\"MAX_CURRENT_AVERAGE\",\"value\":\"0\",\"dataType\":\"Int16\"},{\"key\":\"ERRORCODE\",\"value\":\"0\",\"dataType\":\"Int16\"},{\"key\":\"OPERATING_DISTANCE_MM\",\"value\":\"0\",\"dataType\":\"Int32\"},{\"key\":\"OPERATING_DISTANCE_KM\",\"value\":\"0\",\"dataType\":\"Int32\"}],\"dataType\":\"Struct<TEST_FREQU_CONV_UDT>\"},{\"key\":\"USER\",\"value\":[{\"key\":\"BARCODE_DRIVE\",\"value\":\"0\",\"dataType\":\"Int16\"},{\"key\":\"BARCODE_DRIVE_MIN\",\"value\":\"0\",\"dataType\":\"Int16\"},{\"key\":\"WIFI_SIGNAL_STRENGTH\",\"value\":\"0\",\"dataType\":\"Int16\"},{\"key\":\"WIFI_CHANNEL\",\"value\":\"0\",\"dataType\":\"UInt16\"},{\"key\":\"TX_RATE\",\"value\":\"0\",\"dataType\":\"UInt16\"},{\"key\":\"SC_RESPONSE_TIME\",\"value\":\"0\",\"dataType\":\"UInt32\"}],\"dataType\":\"Struct<TEST_CARRIER_MAX_UDT_USER>\"}],\"status\":\"Good\",\"lastChangeTimestamp\":\"2022-08-14T06:04:02.9607079Z\",\"measurementTimestamp\":\"2022-08-14T06:04:02.9607079Z\",\"dataType\":\"Struct<TEST_CARRIER_MAX_UDT>\"}},\"id\":\"c9e9009b-f79b-44c8-9a44-ebd5a59b0814\",\"$schema\":\"https://xxxxx.blob.core.windows.net/schemas/message_schemas/2021-05-05/xxx.sensor.sensortelemetrymes...",\"metadata\":{\"timestamp\":\"2022-08-14T06:04:22.2257279Z\",\"correlationIds\":[],\"senderIdentifier\":{\"id\":\"aaaaa-1493-4f61-993b-e3fb046908aa\",\"name\":\"OPC UA Connector\",\"type\":\"gateway\",\"route\":\"\"},\"destinationIdentifiers\":[]}}","SCHEMA_MAPPER":"xxxx.TELEMETRY","enqueuedTime":"2022-08-14T06:04:22.225Z","YEAR":2022,"MONTH":8,"DAY":14,"HOUR":6}

 

Labels (2)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

What fields do you want to get out of that data?  What have you tried so far?

---
If this reply helps you, Karma would be appreciated.
0 Karma

Marttin123
New Member

This is what we get at the moment:

DAY

HOUR

MONTH

SCHEMA_MAPPER

YEAR

body

enqueredTime

 

 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Thank you, but that response answers neither of my questions.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...