Splunk Search

Splunk refuses to ingest particular variable

EricLloyd79
Builder

I am attempting to retrieve the SNMP metrics ifHCInOctet and ifHCOutOctet.
I have a Python script that does this. When I used the Splunk python manually in command line, it works and prints the value.
When I try to ingest in by having the script run in Splunk Web, it refuses to print that particular variable.
The value is about 14 digits long. I create a key value pair "KV1=1234567910112".
The type of the variable when retrieved is an int. I change it to a string so I can concatenate it to the key value.
I have even tried to just have it print the type of that variable into an event in splunk and it won't do that. Any operation with that particular variable doesn't seem to work.
It gets even weirder...
When I retrieve ifInOctet and ifOutOctet, it works fine. I thought, okay, well these are smaller numbers so let me try a 14 digit number manually and put it into splunk and it worked fine. It whenever I try to read that particular variable that has been retrieved from the HC snmp metrics.
Further down the rabbit hole... I have two other scripts running and retrieving the HC snmp metrics with 14 digits in them and Splunk will write these to their events.
I'm at a loss. I cannot understand why Splunk refuses to ingest these particular variables on this script. Does anyone have any suggestions? BTW I am sure the script is running cause when I put something in manually along with the KV pair from HC metric, it will print the manually inserted information but refuses to print anything associated with the variable containing the HC snmp metric data.

0 Karma
1 Solution

EricLloyd79
Builder

The issue is resolved via a workaround.
I ended up rewriting my scripts in PHP, where it seemed the data was ingesting more readily. There was still one particular variable that would not ingest in the range of ~14202560152091+/-

I discovered if I added at least 3 digits to the front or back of this number, it would ingest. In the interest of time, I added 3 zeroes to the front of the number, effectively making it computationally the same yet able to be ingested.

View solution in original post

EricLloyd79
Builder

The issue is resolved via a workaround.
I ended up rewriting my scripts in PHP, where it seemed the data was ingesting more readily. There was still one particular variable that would not ingest in the range of ~14202560152091+/-

I discovered if I added at least 3 digits to the front or back of this number, it would ingest. In the interest of time, I added 3 zeroes to the front of the number, effectively making it computationally the same yet able to be ingested.

View solution in original post

DalJeanis
SplunkTrust
SplunkTrust

Okay, seems like I need to back up and check some basics. Since splunk treats everything as strings (until you get to the new metrics stuff) there shouldn't be any difference in how it treats a 13-17 digit long string, even if it IS made up of numbers.

So, I'm wondering about what CAN cause an ingestion error.

Are you using the . or the + concatenation operator?

With your 15 digit numbers starting with "13" and "14", I do notice that a signed, 6-digit integer can represent numbers up to about 140,737,488,355,327 -- which is suspiciously close to the exact range you are talking about. I wouldn't be surprised if you are encountering a silent overflow of some sort.

Pure string handling just can't differentiate between numbers of different lengths, so we have to be dealing with either (A) a numerical conversion, or (B) a misinterpretation of what is occurring.

If you want to hop onto the slack channel, you can post some of those huge items in a thread and we can see if we can debug it further.

0 Karma

EricLloyd79
Builder

I did find a workaround so I will post my workaround and accept that. Your suspicion about the 6-digit integer causing a silent overflow sounds very close what it might be. The reason I say this is because both on my Python script and my PHP script, the values that were being halted were in that 140,7... or 141,7... range although now that I think about it too there was another one in my Python script that was smaller than wouldn't ingest so I don't know.
I am using the "." operator to concatenate in PHP and was using the "+" operator to concatenate in Python. The PHP code seems to ingest easier.
Thanks for your help.

0 Karma

EricLloyd79
Builder

I think I have found a workaround. Its not pretty but it works for our purposes and retains the functionality of stats and other mathematical operations in Splunk.
Since the one particular kv pair requires me to add 3 digits to it for it be ingested, I simply added 3 zeroes to the front of it. So now it ingests and and it retains its computational integrity in Splunk.

I have to ingest two more scripts after the holiday so hopefully those will go in smoothly. Not really sure who to award these karma points to since no one really answered in the correct answer section.

0 Karma

mtulett_splunk
Splunk Employee
Splunk Employee

I don't really think there's an 'answer' here - it would seem to be something specific to your environment, but at least you've found a workaround. Maybe write up a quick summary of how you fixed it and post that, then accept it as your own answer?

niketnilay
Legend

I would agree with the same!

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

niketnilay
Legend

@EricLloyd79, while I am not able to find the question there was similar issue faced by someone more than 6 months back. Comparison between large numbers was failing because numbers got rounded off with same values for nearby sequence numbers.

The option use was to convert to text and prefix zeros to make same length and then perform comparison.
Converting to hexadecimal would also have worked in similar fashion. However, you can stick to text with prefixed 0 if that solves your needs.

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

EricLloyd79
Builder

So, this gets even more mysterious.
I rewrote the entire script in PHP thinking it was a Python issue and the values I calculated Splunk STILL would not ingest.
:(
Oddly enough If I concatenate the value onto itself in PHP it will ingest.
So, kv1=14202560152091
will not ingest but
kv1=1420256015209114202560152091
does.
It makes absolutely no sense.

0 Karma

EricLloyd79
Builder

Apparently, itll ingest when I concatenate a minimum of three numbers to it. Only numbers. If I try to concatenate two it refuses to ingest. Im completely baffled.

0 Karma

mtulett_splunk
Splunk Employee
Splunk Employee

As a test, can you go back to basics and ingest each metric separately, then do the addition of the stats inside splunk with eval? This saves all the calculations and data munging within the script.

I agree though, this is really bizarre.

Alternatively, can you run your script with a cron entry and write to an output file, then follow that with Splunk?

0 Karma

EricLloyd79
Builder

In answer to your first part, yes, I have been ingesting them individually and theres a particular one that won't ingest, but I dont see any difference between its value and the others.
Your other idea might be plausible though yet, I would be concerned about eventual growth of the size of the log file its creating - I'd have to have it on rotation.
Thanks for the ideas. With the PHP version of the script, I only have one of the metrics left that refuses to ingest.

0 Karma

micahkemp
Champion

Can you have your python script write to a text file, so you can see what the script does versus what Splunk does?

0 Karma

EricLloyd79
Builder

Sure. This is what the text output:

FOO_TO_BAR_bw=42633986553287 FOO_FROM_BAR_bw=14247833943534 FOO_TO_BLUE_bw=319351982811404 FOO_FROM_BLUE_bw=164771268078805

The output to Splunk didn't exist. The value "FOO_FROM_BAR_bw" is the cultprit. Beacuse those are all on one line, since that once doesn't ingest, the whole line doesn't ingest.

0 Karma

micahkemp
Champion

Can you post the output of bin/splunk btool transforms list and bin/splunk btool props list? I'm wondering if there is some NULLQUEUE filtering in place that you weren't aware of or forgot about.

0 Karma

EricLloyd79
Builder

Transforms List: Thats a little too long to post here (actually 9900 characters long to be exact)
Props List: That is also too long by even more characters... Is there something in specific I can look for?

0 Karma

micahkemp
Champion

Do you see any TRANSFORM- entries in the sourcetype for this input in props.conf?

0 Karma

EricLloyd79
Builder

@DalJeanis
Here is the code, with variable names and IP addresses removed. Thanks for any help. Ill continue working on it today.


#!/root/anaconda2/python

import pprint

from pysnmp.entity.rfc3413.oneliner import cmdgen

foogw01in, foogw01out, foogw02in, foogw02out, foofw01in, foofw01out = ([] for i in range(6))

ipadds = {'foogw01’:’xx.xx.x.xxx’, 'foogw02’:’xx.xx.x.xxx’, 'foofw01’:’xx.xx.x.xxx’}

pp = pprint.PrettyPrinter(indent=4)




for server, i in ipadds.iteritems():
    cmdGen = cmdgen.CommandGenerator()

    errorIndication, errorStatus, errorIndex, varBindTable = cmdGen.nextCmd(
        cmdgen.CommunityData('cequint'),
        cmdgen.UdpTransportTarget((i, 161), timeout=2),
        cmdgen.MibVariable('IF-MIB', 'ifDescr'),
        cmdgen.MibVariable('IF-MIB', 'ifHCInOctets'),
        cmdgen.MibVariable('IF-MIB', 'ifHCOutOctets'),

       lookupNames=True,
        lookupValues=True
    )


    if errorIndication:
        print(errorIndication)
    else:
        if errorStatus:
                errorIndex and varBindTable[-1][int(errorIndex)-1] or '?'
        else:
            for varBindTableRow in varBindTable:
                for name, val in varBindTableRow:

                    name_f = name.prettyPrint()
                    name_i = name_f[8:]
                    name = name_i.split('.', 1)[0]
                    index = name_f[-1:]
                    val = val.prettyPrint()

                    if name == "ifDescr":
                        port = str(unicode(val))

                    elif name == 'ifHCInOctets':
                        InOctet = val

                    elif name == 'ifHCOutOctets':
                        OutOctet = val



                if server == "foogw01" and port in ("GigabitEthernet0/0/4", "GigabitEthernet0/0/5"):
                        foogw01in.append(InOctet)
                        foogw01out.append(OutOctet)
                elif server == "foogw02" and port == "GigabitEthernet0/0/4.3373":
                       foogw02in.append(InOctet)
                        foogw02out.append(OutOctet)
                elif server == "foofw01" and port in ("reth12", "reth13"):
                        foofw01in.append(InOctet)
                        foofw01out.append(OutOctet)


foo_to_bar_total = int(foogw01out[0]) + int(foogw01out[1]) + int(foogw02out[0]) + int(foofw01out[0])
foo_from_bar_total = int(foogw01in[0]) + int(foogw01in[1]) + int(foogw02in[0]) + int(foofw01in[0])
foo_to_blue = int(foofw01out[0])
foo_from_blue = int(foofw01in[0])



print(“FOO_TO_BAR_bw=" + str(foo_to_bar_total) + " "   “FOO_FROM_BAR_bw=" + str(foo_from_bar_total) + " " +  “FOO_TO_BAR_BLUE_bw=" + str(foo_to_blue) + " " + "foo_FROM_BAR_BLUE_bw=" + str(foo_from_blue) )

0 Karma

EricLloyd79
Builder

So, so far, it seems I can ingest these values as:
Strings with quotes (ie. kv1="83273791379381937")
Hexidecimal values
Scientific numbers

It refuses to ingest them as:
Integers
Strings without quotes

I need them to be ingested as strings without quotes. Just an update. back to the drawing board.

0 Karma

niketnilay
Legend

@EricLloyd79, if you are able to insert as string with quotes that should also work in splunk without any change required. Is there any reason you dont want to do it this way if it is inserting to Splunk?

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

EricLloyd79
Builder

Well I tried to do stats functions on the values when they were strings with the quotes on them and it didn't seem to work. Plus, my boss has told me I need should ingest them without the quotes around them but I can possibly convince him otherwise if I absolutely have to. I have only one value from the PHP scripts which will not ingest unless I concatenate at least 3 digits to it, so I am so close to completion.

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!