Why is there an issue with Sourcetype Log2jmetrics_json?

[arunsoni]

Hello,

We are onboarding json formatted data and the data is written to a file on a server using a process called  Periodic Stats Logger. The file has initial header line where splunk is not recognizing as Json file.I want to exclude that line and want the data to be loaded to a metric index.

I created a metric index and also added null queue to exclude first header line but still I dont see any metric data coming into metric index.

Configuration on Indexer(Standalone):

Props.conf:

[log2metrics_json]
TRANSFORMS-t1=eliminate_header

Transforms.conf:

[eliminate_header]
REGEX= file
DEST_KEY=queue
FORMAT=nullQueue

Configuration on Forwarder:

Inputs.conf

[monitor:///opt/logs/dsstats.json]
sourcetype = log2metrics_json
index = pdmetrics
disabled = false

Props.conf

[log2metrics_json]
DATETIME_CONFIG =
INDEXED_EXTRACTIONS = json
LINE_BREAKER = ([\r\n]+)
METRIC-SCHEMA-TRANSFORMS = metric-schema:log2metrics_default_json
NO_BINARY_CHECK = true
category = Log to Metrics
pulldown_type = 1
description = JSON-formatted data. Log-to-metrics processing converts the numeric values in json keys into metric data points.

Sample Raw file:

# Log file created at 25/Jun/2022:23:59:22 -0400 after rotating from file dsstats.json.20220626035922Z
{"Datetime":"2022-06-26T03:59:23.000723Z","Num Ops in Progress (Avg)":0,"Num Ops in Progress (Max)":0,"All Ops":0,"All Avg Time (ms)":0.0,"All Ops/sec":0.0,"Search Ops":0,"Search Avg Time (ms)":0.0,"Search Ops/sec":0.0,"Modify Ops":0,"Modify Avg Time (ms)":0.0,"Modify Ops/sec":0.0,"Add Ops":0,"Add Avg Time (ms)":0.0,"Add Ops/sec":0.0,"Delete Ops":0,"Delete Avg Time (ms)":0.0,"Delete Ops/sec":0.0,"ModifyDN Ops":0,"ModifyDN Avg Time (ms)":0.0,"ModifyDN Ops/sec":0.0,"Bind Ops":0,"Bind Avg Time (ms)":0.0,"Bind Ops/sec":0.0,"Compare Ops":0,"Compare Avg Time (ms)":0.0,"Compare Ops/sec":0.0,"Extended Ops":0,"Extended Avg Time (ms)":0.0,"Extended Ops/sec":0.0,"Num Current Conn":389.0,"Num New Conn":0,"Backend userRoot Entry Count":957545.0,"Backend userRoot DB Cache % Full":11.0,"Backend userRoot Size on Disk (GB)":1.1374394875019789,"Backend userRoot Active Cleaner Threads (Avg)":0,"Backend userRoot Cleaner Backlog (Avg)":0,"Backend userRoot Cleaner Backlog (Min)":0,"Backend userRoot Cleaner Backlog (Max)":0,"Backend userRoot Nodes Evicted":0,"Backend userRoot Checkpoints":0,"Backend userRoot Average Checkpoint Duration":0,"Backend userRoot New DB Logs":0,"Replica dc=mediacom,dc=com Replication Backlog (Avg)":0,"Replica dc=mediacom,dc=com Replication Backlog (Min)":0,"Replica dc=mediacom,dc=com Replication Backlog (Max)":0,"Replica dc=mediacom,dc=com Sent Updates":0,"Replica dc=mediacom,dc=com Received Updates":0,"Replica dc=mediacom,dc=com Failed Replayed Updates":0,"Replica dc=mediacom,dc=com Unresolved Naming Conflicts":0,"Replica dc=mediacom,dc=com Replication Backlog Oldest Pending Change":0,"Backend replicationChanges DB Cache % Full":2.0,"Backend replicationChanges Active Cleaner Threads (Avg)":0,"Backend replicationChanges Cleaner Backlog (Avg)":0,"Backend replicationChanges Cleaner Backlog (Min)":0,"Backend replicationChanges Cleaner Backlog (Max)":0,"Worker Thread % Busy (Avg)":0,"Worker Thread % Busy (Max)":0,"Queue Size (Avg)":0,"Queue Size (Max)":0,"JVM Memory Used GB":12.2666015625,"JVM Memory Percent Full":56.0,"Memory Consumer Total GB":1.2536366451531649,"Memory Consumer Percentage of Total Memory":5.0,"g1-young-generation minor-collection g1-evacuation-pause Count":0,"g1-young-generation minor-collection metadata-gc-threshold Count":0,"g1-young-generation minor-collection g1-evacuation-pause Duration (ms)":0,"g1-young-generation minor-collection metadata-gc-threshold Duration (ms)":0,"g1-eden-space Live Bytes":0,"g1-survivor-space Live Bytes":0,"g1-old-gen Live Bytes":0,"g1-young-generation minor-collection gclocker-initiated-gc Count":0,"g1-young-generation minor-collection gclocker-initiated-gc Duration (ms)":0,"All Ops 0-1 ms":0,"All Ops 1-2 ms":0,"All Ops 2-3 ms":0,"All Ops 3-5 ms":0,"All Ops 5-10 ms":0,"All Ops 10-20 ms":0,"All Ops 20-30 ms":0,"All Ops 30-50 ms":0,"All Ops 50-100 ms":0,"All Ops 100-1000 ms":0,"All Ops > 1000 ms":0}
{"Datetime":"2022-06-26T03:59:24.000382Z","Num Ops in Progress (Avg)":0,"Num Ops in Progress (Max)":0,"All Ops":0,"All Avg Time (ms)":0.0,"All Ops/sec":0.0,"Search Ops":0,"Search Avg Time (ms)":0.0,"Search Ops/sec":0.0,"Modify Ops":0,"Modify Avg Time (ms)":0.0,"Modify Ops/sec":0.0,"Add Ops":0,"Add Avg Time (ms)":0.0,"Add Ops/sec":0.0,"Delete Ops":0,"Delete Avg Time (ms)":0.0,"Delete Ops/sec":0.0,"ModifyDN Ops":0,"ModifyDN Avg Time (ms)":0.0,"ModifyDN Ops/sec":0.0,"Bind Ops":0,"Bind Avg Time (ms)":0.0,"Bind Ops/sec":0.0,"Compare Ops":0,"Compare Avg Time (ms)":0.0,"Compare Ops/sec":0.0,"Extended Ops":0,"Extended Avg Time (ms)":0.0,"Extended Ops/sec":0.0,"Num Current Conn":389.0,"Num New Conn":0,"Backend userRoot Entry Count":957545.0,"Backend userRoot DB Cache % Full":11.0,"Backend userRoot Size on Disk (GB)":1.1374394875019789,"Backend userRoot Active Cleaner Threads (Avg)":0,"Backend userRoot Cleaner Backlog (Avg)":0,"Backend userRoot Cleaner Backlog (Min)":0,"Backend userRoot Cleaner Backlog (Max)":0,"Backend userRoot Nodes Evicted":0,"Backend userRoot Checkpoints":0,"Backend userRoot Average Checkpoint Duration":0,"Backend userRoot New DB Logs":0,"Replica dc=mediacom,dc=com Replication Backlog (Avg)":0,"Replica dc=mediacom,dc=com Replication Backlog (Min)":0,"Replica dc=mediacom,dc=com Replication Backlog (Max)":0,"Replica dc=mediacom,dc=com Sent Updates":0,"Replica dc=mediacom,dc=com Received Updates":0,"Replica dc=mediacom,dc=com Failed Replayed Updates":0,"Replica dc=mediacom,dc=com Unresolved Naming Conflicts":0,"Replica dc=mediacom,dc=com Replication Backlog Oldest Pending Change":0,"Backend replicationChanges DB Cache % Full":2.0,"Backend replicationChanges Active Cleaner Threads (Avg)":0,"Backend replicationChanges Cleaner Backlog (Avg)":0,"Backend replicationChanges Cleaner Backlog (Min)":0,"Backend replicationChanges Cleaner Backlog (Max)":0,"Worker Thread % Busy (Avg)":0,"Worker Thread % Busy (Max)":0,"Queue Size (Avg)":0,"Queue Size (Max)":0,"JVM Memory Used GB":12.3291015625,"JVM Memory Percent Full":56.0,"Memory Consumer Total GB":1.2536366451531649,"Memory Consumer Percentage of Total Memory":5.0,"g1-young-generation minor-collection g1-evacuation-pause Count":0,"g1-young-generation minor-collection metadata-gc-threshold Count":0,"g1-young-generation minor-collection g1-evacuation-pause Duration (ms)":0,"g1-young-generation minor-collection metadata-gc-threshold Duration (ms)":0,"g1-eden-space Live Bytes":0,"g1-survivor-space Live Bytes":0,"g1-old-gen Live Bytes":0,"g1-young-generation minor-collection gclocker-initiated-gc Count":0,"g1-young-generation minor-collection gclocker-initiated-gc Duration (ms)":0,"All Ops 0-1 ms":0,"All Ops 1-2 ms":0,"All Ops 2-3 ms":0,"All Ops 3-5 ms":0,"All Ops 5-10 ms":0,"All Ops 10-20 ms":0,"All Ops 20-30 ms":0,"All Ops 30-50 ms":0,"All Ops 50-100 ms":0,"All Ops 100-1000 ms":0,"All Ops > 1000 ms":0}
{"Datetime":"2022-06-26T03:59:25.000195Z","Num Ops in Progress (Avg)":0,"Num Ops in Progress (Max)":0,"All Ops":0,"All Avg Time (ms)":0.0,"All Ops/sec":0.0,"Search Ops":0,"Search Avg Time (ms)":0.0,"Search Ops/sec":0.0,"Modify Ops":0,"Modify Avg Time (ms)":0.0,"Modify Ops/sec":0.0,"Add Ops":0,"Add Avg Time (ms)":0.0,"Add Ops/sec":0.0,"Delete Ops":0,"Delete Avg Time (ms)":0.0,"Delete Ops/sec":0.0,"ModifyDN Ops":0,"ModifyDN Avg Time (ms)":0.0,"ModifyDN Ops/sec":0.0,"Bind Ops":0,"Bind Avg Time (ms)":0.0,"Bind Ops/sec":0.0,"Compare Ops":0,"Compare Avg Time (ms)":0.0,"Compare Ops/sec":0.0,"Extended Ops":0,"Extended Avg Time (ms)":0.0,"Extended Ops/sec":0.0,"Num Current Conn":389.0,"Num New Conn":0,"Backend userRoot Entry Count":957545.0,"Backend userRoot DB Cache % Full":11.0,"Backend userRoot Size on Disk (GB)":1.1374394875019789,"Backend userRoot Active Cleaner Threads (Avg)":0,"Backend userRoot Cleaner Backlog (Avg)":0,"Backend userRoot Cleaner Backlog (Min)":0,"Backend userRoot Cleaner Backlog (Max)":0,"Backend userRoot Nodes Evicted":0,"Backend userRoot Checkpoints":0,"Backend userRoot Average Checkpoint Duration":0,"Backend userRoot New DB Logs":0,"Replica dc=mediacom,dc=com Replication Backlog (Avg)":0,"Replica dc=mediacom,dc=com Replication Backlog (Min)":0,"Replica dc=mediacom,dc=com Replication Backlog (Max)":0,"Replica dc=mediacom,dc=com Sent Updates":0,"Replica dc=mediacom,dc=com Received Updates":0,"Replica dc=mediacom,dc=com Failed Replayed Updates":0,"Replica dc=mediacom,dc=com Unresolved Naming Conflicts":0,"Replica dc=mediacom,dc=com Replication Backlog Oldest Pending Change":0,"Backend replicationChanges DB Cache % Full":2.0,"Backend replicationChanges Active Cleaner Threads (Avg)":0,"Backend replicationChanges Cleaner Backlog (Avg)":0,"Backend replicationChanges Cleaner Backlog (Min)":0,"Backend replicationChanges Cleaner Backlog (Max)":0,"Worker Thread % Busy (Avg)":0,"Worker Thread % Busy (Max)":0,"Queue Size (Avg)":0,"Queue Size (Max)":0,"JVM Memory Used GB":12.4052734375,"JVM Memory Percent Full":56.0,"Memory Consumer Total GB":1.2536366451531649,"Memory Consumer Percentage of Total Memory":5.0,"g1-young-generation minor-collection g1-evacuation-pause Count":0,"g1-young-generation minor-collection metadata-gc-threshold Count":0,"g1-young-generation minor-collection g1-evacuation-pause Duration (ms)":0,"g1-young-generation minor-collection metadata-gc-threshold Duration (ms)":0,"g1-eden-space Live Bytes":0,"g1-survivor-space Live Bytes":0,"g1-old-gen Live Bytes":0,"g1-young-generation minor-collection gclocker-initiated-gc Count":0,"g1-young-generation minor-collection gclocker-initiated-gc Duration (ms)":0,"All Ops 0-1 ms":0,"All Ops 1-2 ms":0,"All Ops 2-3 ms":0,"All Ops 3-5 ms":0,"All Ops 5-10 ms":0,"All Ops 10-20 ms":0,"All Ops 20-30 ms":0,"All Ops 30-50 ms":0,"All Ops 50-100 ms":0,"All Ops 100-1000 ms":0,"All Ops > 1000 ms":0}

Please do the needful Help.Thanks in advance