topic Re: Splunk refuses to ingest particular variable in Splunk Search

Splunk refuses to ingest particular variable

EricLloyd79 — Mon, 20 Nov 2017 17:47:19 GMT

I am attempting to retrieve the SNMP metrics ifHCInOctet and ifHCOutOctet.
I have a Python script that does this. When I used the Splunk python manually in command line, it works and prints the value.
When I try to ingest in by having the script run in Splunk Web, it refuses to print that particular variable.
The value is about 14 digits long. I create a key value pair "KV1=1234567910112".
The type of the variable when retrieved is an int. I change it to a string so I can concatenate it to the key value.
I have even tried to just have it print the type of that variable into an event in splunk and it won't do that. Any operation with that particular variable doesn't seem to work.
It gets even weirder...
When I retrieve ifInOctet and ifOutOctet, it works fine. I thought, okay, well these are smaller numbers so let me try a 14 digit number manually and put it into splunk and it worked fine. It whenever I try to read that particular variable that has been retrieved from the HC snmp metrics.
Further down the rabbit hole... I have two other scripts running and retrieving the HC snmp metrics with 14 digits in them and Splunk will write these to their events.
I'm at a loss. I cannot understand why Splunk refuses to ingest these particular variables on this script. Does anyone have any suggestions? BTW I am sure the script is running cause when I put something in manually along with the KV pair from HC metric, it will print the manually inserted information but refuses to print anything associated with the variable containing the HC snmp metric data.

Re: Splunk refuses to ingest particular variable

EricLloyd79 — Mon, 20 Nov 2017 18:47:18 GMT

As I explore deeper, it seems that one particular element is causing the problem.
I am adding together 4 different bandwidth values from servers: s1, s2, s3, and s4.
s1, s2 and s3 add up together fine and print into splunk as 10636157794287
s4 = 133395228938698

The combination of s1 is 14 digits and s4 is 15 digits. Could it be the size of the number that is causing the problem?

It seems the nature of this problem is in the int casting of the 15 digit number. I tried casting to a long and it still won't print in Splunk but when I cast it to a float, it will print in Splunk as a scientific number. This isn't the optimal way we want to record these number... is there a way to print it to Splunk with 15 digits in non-scientific format?

Re: Splunk refuses to ingest particular variable

niketn — Mon, 20 Nov 2017 20:39:48 GMT

@EricLloyd79, I would not expect Splunk to complain about data being inserted as it would consider it as string.

Even if it was a number Splunk should have picked up length of 15.

|  makeresults
|  eval data="133395228938698"
|  eval dataLength=len(data)
|  eval dataNum=tonumber(data)
|  eval dataNumLength=len(dataNum)

If the above number was 18 digits or more then you would start seeing rounding off of numeric data for example: try | eval data="133395228938630445"

Not sure if this is the right place but can you check in Python Script if somewhere data being ingested can be converted from number to string?

Re: Splunk refuses to ingest particular variable

EricLloyd79 — Mon, 20 Nov 2017 22:18:26 GMT

@niketnilay
Thanks for your response. I guess I didn't make it clear... Before I output the data to be ingested in splunk I transform it to a string. This is required as I concatenate it to. Key value (ie. "Kv1=" + var)

It's odd, following my example before I can actually output the value of s4 (which is 15 digits itself) and splunk will ingest it.
It's something about adding there fourth value to the others leaves the result in a form that splunk won't ingest ...
If you reply can you do so in an answer below instead of a comment?

Re: Splunk refuses to ingest particular variable

EricLloyd79 — Mon, 20 Nov 2017 23:58:33 GMT

Ah it seems Python is adding a trailing L that appears on large integers and for some reason Splunk won't ingest it! Now to find a way to get rid of that L!

Re: Splunk refuses to ingest particular variable

EricLloyd79 — Tue, 21 Nov 2017 00:21:41 GMT

That wasn't the answer. That was from me converting one of the values to a long but even without the L there it won't ingest into Splunk...

Re: Splunk refuses to ingest particular variable

mtulett_splunk — Tue, 21 Nov 2017 04:52:32 GMT

While I agree that this shouldn't be an issue, can you add the numbers up inside splunk, rather than inside your script?

I would expect any data to be ingested correctly if you have output in key="value" format (which is what you're returning, right?).

Re: Splunk refuses to ingest particular variable

niketn — Tue, 21 Nov 2017 05:04:49 GMT

@EricLloyd79, the reason why I kept it as a comment and not an answer was to keep it unanswered so that it remains flagged for others as well to resolve.

I am just speculating things and your trials seem to fail as well. Can you try a different approach of converting the input value to Hexadecimal in your Python code and then insert to Splunk? Later in Splunk perform operations on Hexadecimal and then convert to number afterwards to be displayed as end results. You can use printf() or tostring() function to perform this.

Re: Splunk refuses to ingest particular variable

mtulett_splunk — Tue, 29 Sep 2020 16:51:21 GMT

Just to add to this, I tried a scripted input that simply returns a large number (echo bytes="some_big_number") and was able to push very large numbers into splunk without issue - this might suggest it's a script issue, rather than a Splunk issue?

Re: Splunk refuses to ingest particular variable

niketn — Tue, 21 Nov 2017 05:42:41 GMT

@mtulett, tanks for that evidence. I think we all agree that issue is while ingesting from Python, since @EricLloyd79's attempt to fix data being treated as Long integer failed, I was proposing Python script to convert number to Hexadecimal, that way the same can be ingested and used within Splunk. If required it can be formatted back to number using formatting command before displaying to users.

Re: Splunk refuses to ingest particular variable

mtulett_splunk — Tue, 21 Nov 2017 06:05:02 GMT

I reworked it using a python script as the input, however it's still working correctly. I'm simply printing the output like so:

bytes = 65816843135136843513543434236856165465415641651651
print ("bytes=" +str(bytes))

Which gave me input with no issues:

Re: Splunk refuses to ingest particular variable

DalJeanis — Tue, 21 Nov 2017 13:17:36 GMT

At this point, it sure seems likely that the python script is the issue. Please add a comment with the python code and we'll see if we can help you debug it.

Re: Splunk refuses to ingest particular variable

EricLloyd79 — Tue, 21 Nov 2017 16:43:05 GMT

I considered adding the numbers in Splunk but some of the base SNMP values are actually not ingesting by themselves either.

Re: Splunk refuses to ingest particular variable

EricLloyd79 — Tue, 21 Nov 2017 16:48:00 GMT

Yeah Im finding less and less rhyme and reason to this surrounding the size of the number. Some of them will ingest with 12 digits, some with 15+ and some won't with 12 and some won't with 15. I have tried to see if there was special characters around the values Im trying to ingest but using python's repr function didn't show any. Im curious about switching the values to hex and then back... need to investigate. I think its something about the nature of the SNMP 64 bit values Im bringing in using the HC snmp metrics (ie. ifHCInOctets, ifHCOutOctets) But Im not sure of that either.

Re: Splunk refuses to ingest particular variable

EricLloyd79 — Tue, 21 Nov 2017 16:49:16 GMT

@niketnilay this is an intriguing prospect. Switching to hex and then back maybe will possibly strip the variable in a form that splunk will accept? They have to be ingested as numbers, though, not hex.

Re: Splunk refuses to ingest particular variable

EricLloyd79 — Tue, 21 Nov 2017 16:58:16 GMT

@DalJeanis
Here is the code, with variable names and IP addresses removed. Thanks for any help. Ill continue working on it today.

#!/root/anaconda2/python

import pprint

from pysnmp.entity.rfc3413.oneliner import cmdgen

foogw01in, foogw01out, foogw02in, foogw02out, foofw01in, foofw01out = ([] for i in range(6))

ipadds = {'foogw01’:’xx.xx.x.xxx’, 'foogw02’:’xx.xx.x.xxx’, 'foofw01’:’xx.xx.x.xxx’}

pp = pprint.PrettyPrinter(indent=4)




for server, i in ipadds.iteritems():
    cmdGen = cmdgen.CommandGenerator()

    errorIndication, errorStatus, errorIndex, varBindTable = cmdGen.nextCmd(
        cmdgen.CommunityData('cequint'),
        cmdgen.UdpTransportTarget((i, 161), timeout=2),
        cmdgen.MibVariable('IF-MIB', 'ifDescr'),
        cmdgen.MibVariable('IF-MIB', 'ifHCInOctets'),
        cmdgen.MibVariable('IF-MIB', 'ifHCOutOctets'),

       lookupNames=True,
        lookupValues=True
    )


    if errorIndication:
        print(errorIndication)
    else:
        if errorStatus:
                errorIndex and varBindTable[-1][int(errorIndex)-1] or '?'
        else:
            for varBindTableRow in varBindTable:
                for name, val in varBindTableRow:

                    name_f = name.prettyPrint()
                    name_i = name_f[8:]
                    name = name_i.split('.', 1)[0]
                    index = name_f[-1:]
                    val = val.prettyPrint()

                    if name == "ifDescr":
                        port = str(unicode(val))

                    elif name == 'ifHCInOctets':
                        InOctet = val

                    elif name == 'ifHCOutOctets':
                        OutOctet = val



                if server == "foogw01" and port in ("GigabitEthernet0/0/4", "GigabitEthernet0/0/5"):
                        foogw01in.append(InOctet)
                        foogw01out.append(OutOctet)
                elif server == "foogw02" and port == "GigabitEthernet0/0/4.3373":
                       foogw02in.append(InOctet)
                        foogw02out.append(OutOctet)
                elif server == "foofw01" and port in ("reth12", "reth13"):
                        foofw01in.append(InOctet)
                        foofw01out.append(OutOctet)


foo_to_bar_total = int(foogw01out[0]) + int(foogw01out[1]) + int(foogw02out[0]) + int(foofw01out[0])
foo_from_bar_total = int(foogw01in[0]) + int(foogw01in[1]) + int(foogw02in[0]) + int(foofw01in[0])
foo_to_blue = int(foofw01out[0])
foo_from_blue = int(foofw01in[0])



print(“FOO_TO_BAR_bw=" + str(foo_to_bar_total) + " "   “FOO_FROM_BAR_bw=" + str(foo_from_bar_total) + " " +  “FOO_TO_BAR_BLUE_bw=" + str(foo_to_blue) + " " + "foo_FROM_BAR_BLUE_bw=" + str(foo_from_blue) )

Re: Splunk refuses to ingest particular variable

EricLloyd79 — Tue, 21 Nov 2017 18:03:31 GMT

Yes I am returning format key=value. I can get it to return kv1="1238472743383741" but I need to have it without the quotes. (It returns that way if I return it as print(repr("kv1=" + val))

Re: Splunk refuses to ingest particular variable

EricLloyd79 — Tue, 21 Nov 2017 18:03:58 GMT

A sample of my code exists in a comment below.

Re: Splunk refuses to ingest particular variable

EricLloyd79 — Tue, 21 Nov 2017 18:21:08 GMT

@niketnilay
Thanks for the suggestion, it will indeed process the value as hex and display it but as per my use case is to use the PHP API to retrieve these values via a report. I dont think its acceptable for me to leave them in there are hex numbers and then transform them in the PHP API report we're creating I need them to be ingested as strings (without quotes). 😞 I even tried converting it back from hex to integer to string before ingesting and it refuses that value.

Re: Splunk refuses to ingest particular variable

EricLloyd79 — Tue, 21 Nov 2017 18:22:41 GMT

So, so far, it seems I can ingest these values as:
Strings with quotes (ie. kv1="83273791379381937")
Hexidecimal values
Scientific numbers

It refuses to ingest them as:
Integers
Strings without quotes

I need them to be ingested as strings without quotes. Just an update. back to the drawing board.