Splunk refuses to ingest particular variable

[EricLloyd79]

I am attempting to retrieve the SNMP metrics ifHCInOctet and ifHCOutOctet.
I have a Python script that does this. When I used the Splunk python manually in command line, it works and prints the value.
When I try to ingest in by having the script run in Splunk Web, it refuses to print that particular variable.
The value is about 14 digits long. I create a key value pair "KV1=1234567910112".
The type of the variable when retrieved is an int. I change it to a string so I can concatenate it to the key value.
I have even tried to just have it print the type of that variable into an event in splunk and it won't do that. Any operation with that particular variable doesn't seem to work.
It gets even weirder...
When I retrieve ifInOctet and ifOutOctet, it works fine. I thought, okay, well these are smaller numbers so let me try a 14 digit number manually and put it into splunk and it worked fine. It whenever I try to read that particular variable that has been retrieved from the HC snmp metrics.
Further down the rabbit hole... I have two other scripts running and retrieving the HC snmp metrics with 14 digits in them and Splunk will write these to their events.
I'm at a loss. I cannot understand why Splunk refuses to ingest these particular variables on this script. Does anyone have any suggestions? BTW I am sure the script is running cause when I put something in manually along with the KV pair from HC metric, it will print the manually inserted information but refuses to print anything associated with the variable containing the HC snmp metric data.

[EricLloyd79]

Please try to answer as an official answer so that when a solution is found I can award the karma points.

[DalJeanis]

At this point, it sure seems likely that the python script is the issue. Please add a comment with the python code and we'll see if we can help you debug it.

[EricLloyd79]

As I explore deeper, it seems that one particular element is causing the problem.
I am adding together 4 different bandwidth values from servers: s1, s2, s3, and s4.
s1, s2 and s3 add up together fine and print into splunk as 10636157794287
s4 = 133395228938698

The combination of s1 is 14 digits and s4 is 15 digits. Could it be the size of the number that is causing the problem?

It seems the nature of this problem is in the int casting of the 15 digit number. I tried casting to a long and it still won't print in Splunk but when I cast it to a float, it will print in Splunk as a scientific number. This isn't the optimal way we want to record these number... is there a way to print it to Splunk with 15 digits in non-scientific format?

[EricLloyd79]

@niketnilay
Thanks for your response. I guess I didn't make it clear... Before I output the data to be ingested in splunk I transform it to a string. This is required as I concatenate it to. Key value (ie. "Kv1=" + var)

It's odd, following my example before I can actually output the value of s4 (which is 15 digits itself) and splunk will ingest it.
It's something about adding there fourth value to the others leaves the result in a form that splunk won't ingest ...
If you reply can you do so in an answer below instead of a comment?

[mtulett_splunk]

Just to add to this, I tried a scripted input that simply returns a large number (echo bytes="some_big_number") and was able to push very large numbers into splunk without issue - this might suggest it's a script issue, rather than a Splunk issue?

[EricLloyd79]

Yeah Im finding less and less rhyme and reason to this surrounding the size of the number. Some of them will ingest with 12 digits, some with 15+ and some won't with 12 and some won't with 15. I have tried to see if there was special characters around the values Im trying to ingest but using python's repr function didn't show any. Im curious about switching the values to hex and then back... need to investigate. I think its something about the nature of the SNMP 64 bit values Im bringing in using the HC snmp metrics (ie. ifHCInOctets, ifHCOutOctets) But Im not sure of that either.

[EricLloyd79]

A sample of my code exists in a comment below.

[niketn]

@mtulett, tanks for that evidence. I think we all agree that issue is while ingesting from Python, since @EricLloyd79's attempt to fix data being treated as Long integer failed, I was proposing Python script to convert number to Hexadecimal, that way the same can be ingested and used within Splunk. If required it can be formatted back to number using formatting command before displaying to users.

[EricLloyd79]

@niketnilay
Thanks for the suggestion, it will indeed process the value as hex and display it but as per my use case is to use the PHP API to retrieve these values via a report. I dont think its acceptable for me to leave them in there are hex numbers and then transform them in the PHP API report we're creating I need them to be ingested as strings (without quotes). 😞 I even tried converting it back from hex to integer to string before ingesting and it refuses that value.

[EricLloyd79]

@niketnilay this is an intriguing prospect. Switching to hex and then back maybe will possibly strip the variable in a form that splunk will accept? They have to be ingested as numbers, though, not hex.

[mtulett_splunk]

I reworked it using a python script as the input, however it's still working correctly. I'm simply printing the output like so:

bytes = 65816843135136843513543434236856165465415641651651
print ("bytes=" +str(bytes))

Which gave me input with no issues:

[EricLloyd79]

Ah it seems Python is adding a trailing L that appears on large integers and for some reason Splunk won't ingest it! Now to find a way to get rid of that L!

[EricLloyd79]

That wasn't the answer. That was from me converting one of the values to a long but even without the L there it won't ingest into Splunk...

[niketn]

@EricLloyd79, the reason why I kept it as a comment and not an answer was to keep it unanswered so that it remains flagged for others as well to resolve.

I am just speculating things and your trials seem to fail as well. Can you try a different approach of converting the input value to Hexadecimal in your Python code and then insert to Splunk? Later in Splunk perform operations on Hexadecimal and then convert to number afterwards to be displayed as end results. You can use printf() or tostring() function to perform this.

[mtulett_splunk]

While I agree that this shouldn't be an issue, can you add the numbers up inside splunk, rather than inside your script?

I would expect any data to be ingested correctly if you have output in key="value" format (which is what you're returning, right?).

[EricLloyd79]

Yes I am returning format key=value. I can get it to return kv1="1238472743383741" but I need to have it without the quotes. (It returns that way if I return it as print(repr("kv1=" + val))

[EricLloyd79]

I considered adding the numbers in Splunk but some of the base SNMP values are actually not ingesting by themselves either.

[niketn]

@EricLloyd79, I would not expect Splunk to complain about data being inserted as it would consider it as string.

Even if it was a number Splunk should have picked up length of 15.

|  makeresults
|  eval data="133395228938698"
|  eval dataLength=len(data)
|  eval dataNum=tonumber(data)
|  eval dataNumLength=len(dataNum)

If the above number was 18 digits or more then you would start seeing rounding off of numeric data for example: try | eval data="133395228938630445"

Not sure if this is the right place but can you check in Python Script if somewhere data being ingested can be converted from number to string?