Getting Data In

How can I get my TCP data into the metrics index?

walkerhound
Path Finder

I am trying to pull incoming tcp data into the Metrics Store using this information:

http://docs.splunk.com/Documentation/Splunk/7.0.0/Metrics/GetMetricsInOther#Get_metrics_in_from_clie...

Currently the data is coming in to an event index and the _raw field looks like this:
2017-11-29 18:43:22.807 +0000 Tag="Channel1.Device1.Tag1" Value="1523" Quality="good"

We would like to pull this into the Metrics Store with these fields
metric_name Tag
_value Value
_time _time

(We also want the Quality, but I'm ignoring that until I get the basics.)

Created a new port for the data to come in.

Source Sourcexxxxx
Source Type Typexxxxx
index indexxxxxx

The index (indexxxxx) is a metrics index

I edit configuration files as below and restart Splunk. I get two errors:

Index Processor: Metric value= is not valid for source=Sourcexxxxx, sourcetype=Typexxxxx, host=Hostxxxxx, index=index51115. Metric event data with an invalid metric value would not be indexed. Ensure the input metric data is not malformed.

Index Processor: Metric name is missing for source=Sourcexxxxx, sourcetype=Typexxxxx, host=Hostxxxxx, index=indexxxxxx. Metric event data without metric name is invalid and would not be indexed. Ensure the input metric data is not malformed.

What am I doing wrong? Thanks in advance for your help.

props.conf

[Typexxxxx]
    TIME_FORMAT = %F %H:%M:%S.%Q %z
    TRANSFORMS-test_metric_name = test_name
    TRANSFORMS-test_metric_value = test_value
    NO_BINARY_CHECK = true
    SHOULD_LINEMERGE = false
    pulldown_type = 1
    category = Metrics

transforms.conf
[test_name]
REGEX = "Channel1.Device1.(\S+)"
FORMAT = metric_name::$1
WRITE_META = true

[test_value]
    REGEX = Value="(\S+)"
    FORMAT = _value::$1
    WRITE_META = true

fields.conf
[metric_name]
INDEXED=true

    [_value]
    INDEXED=true
0 Karma

MuS
Legend

Hi walkerhound,

One thing I spot is that the docs mentions this value for the metric_name FORMAT to be FORMAT = metric_name::graphite.$1 maybe try to use FORMAT = metric_name::channel1.$1 in your case.

For the _value, try this regex Value=\"([^\"]+)\" or if you really have literary Value="1234" in your logs try this:

 [test_value]
 REGEX = (.+)
 SOURCE_KEY = Value
 FORMAT = _value::$1
 WRITE_META = true

Hope this helps ...

cheers, MuS

0 Karma

walkerhound
Path Finder

It turns out that my problem had to do with permissions. Splunk could not read the fields.conf file. When I corrected that problem, this worked.

0 Karma

MuS
Legend

Nice finding! How about writing that up as new answer and accept it so it can help other finding the right solution? I suspect you were doing this on a Windows Splunk instance?

0 Karma

walkerhound
Path Finder

Thank you for the suggestions. I experimented with several varieties of your ideas but I continue to get the same two errors indicating that the metric name and metric values are missing or unset.

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In November, the Splunk Threat Research Team had one release of new security content via the Enterprise ...

Index This | Divide 100 by half. What do you get?

November 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Stay Connected: Your Guide to December Tech Talks, Office Hours, and Webinars!

❄️ Celebrate the season with our December lineup of Community Office Hours, Tech Talks, and Webinars! ...