topic Re: Field showing an additional and not visible value --"none"-- under timestamp field in Getting Data In

Field showing an additional and not visible value --"none"-- under timestamp field

mgarciar — Fri, 15 Sep 2017 20:18:32 GMT

Hi all,

I have a problem with a field call "timestamp".

I have created a custom python script and added as "Data input". The script is executed every 5 minutes and makes an API call, parse the json response and send it to the indexer.

This is a sample raw event:

{"rev_pingdeath_count": 0, "fwd_tiny_count": 0, "dst_address": "X.X.X.X", "timestamp": "2017-09-15T16:05:00.000Z", "start_timestamp": "1505491512000000", "fwd_cwr_count": 0, "user_src_location": null, "rev_synrst_count": 0, "fwd_xmas_count": 0, "server_app_latency_usec": 0, "rev_rst_count": 0, "dst_is_internal": "true", "user_dst_businessUnit": null, "fwd_bytes": 1022, "fwd_synfin_count": 0, "rev_ack_count": 8, "user_dst_pod": null, "total_perceived_latency_usec": 0, "bandwidth_bytes_per_second": "0", "user_src_department": null, "user_src_businessUnit": null, "rev_psh_count": 4, "dst_hostname": "appServerXXXX", "user_src_lifecycle": null, "rev_pkts": 8, "fwd_nc_count": 0, "src_address": "Y.Y.Y.Y", "rev_finnoack_count": 0, "user_src_pod": null, "dst_enforcement_epg_name": [], "rev_nc_count": 0, "rev_cwr_count": 0, "user_src_datacenter": null, "fwd_synrst_count": 0, "fwd_ack_count": 7, "srtt_available": "SRTT_NONE", "rev_allzero_count": 0}

There is only one timestamp field on each event, as far I have been able to see, but when I do a ** index=main source=mysource | head 1 | table timestamp** I get the following data:

Where is the none value coming from ??. This none is present in every single event.

Splunk version 6.5.2, single instance.

Thanks and regards,

Re: Field showing an additional and not visible value --"none"-- under timestamp field

s2_splunk — Fri, 15 Sep 2017 20:29:33 GMT

Cannot reproduce, please share your props.conf for this sourcetype

Re: Field showing an additional and not visible value --"none"-- under timestamp field

mgarciar — Tue, 29 Sep 2020 15:46:20 GMT

props.conf
[mysourcetype]
DATETIME_CONFIG = CURRENT
MAX_EVENTS = 1000
TRUNCATE = 100000

But I did find something interesting ... I did a export raw events and this is what I got:

_raw field contains the event as in the original message
_time fields shows fine
timestamp field shows none

Re: Field showing an additional and not visible value --"none"-- under timestamp field

mgarciar — Fri, 15 Sep 2017 21:14:04 GMT

Sorry actually shows both values: none and the actual field value

Re: Field showing an additional and not visible value --"none"-- under timestamp field

gcusello — Sat, 16 Sep 2017 06:54:20 GMT

Hi mgarciar,
I think that you used "timestamp" field as timestamp in log ingestion
so try to use index=main source=mysource | head 1 | table _time
Bye.
Giuseppe

Re: Field showing an additional and not visible value --"none"-- under timestamp field

mgarciar — Sat, 16 Sep 2017 12:50:58 GMT

Splunk is using the current time to set the _time field value and not timestamp.

Re: Field showing an additional and not visible value --"none"-- under timestamp field

gcusello — Sat, 16 Sep 2017 16:18:14 GMT

It should be better to configure your input to use timestamp field as timestamp,
Anyway try

index=main source=mysource | head 1 | rename timestamp AS _time | table _time

Bye.
Giuseppe

Re: Field showing an additional and not visible value --"none"-- under timestamp field

Sukisen1981 — Tue, 29 Sep 2020 15:46:34 GMT

just tried to use your sample in a text input, it worked fine, below is how the event looks like when i ingest it through a text file...see and compare against your event
{ [-]
bandwidth_bytes_per_second: 0

dst_address: X.X.X.X

dst_enforcement_epg_name: [ [+]
]

dst_hostname: appServerXXXX

dst_is_internal: true

fwd_ack_count: 7

fwd_bytes: 1022

fwd_cwr_count: 0

fwd_nc_count: 0

fwd_synfin_count: 0

fwd_synrst_count: 0

fwd_tiny_count: 0

fwd_xmas_count: 0

rev_ack_count: 8

rev_allzero_count: 0

rev_cwr_count: 0

rev_finnoack_count: 0

rev_nc_count: 0

rev_pingdeath_count: 0

rev_pkts: 8

rev_psh_count: 4

rev_rst_count: 0

rev_synrst_count: 0

server_app_latency_usec: 0

src_address: Y.Y.Y.Y

srtt_available: SRTT_NONE

start_timestamp: 1505491512000000

timestamp: 2017-09-15T16:05:00.000Z

total_perceived_latency_usec: 0

user_dst_businessUnit: null

user_dst_pod: null

user_src_businessUnit: null

user_src_datacenter: null

user_src_department: null

user_src_lifecycle: null

user_src_location: null

user_src_pod: null

}

And when i ran the query | table timestamp,_time , this is what i get:
timestamp _time
2017-09-15T16:05:00.000Z 2017-09-15 21:35:00

so mgarciar _timee is not really the current time , is it?
I think if you do the event compare between my text ingestion and your event you will get some clues

Re: Field showing an additional and not visible value --"none"-- under timestamp field

gcusello — Mon, 18 Sep 2017 06:54:34 GMT

Hi mgarciar,,
could you explain:
you ingested this event from a text file in index=main using as TimeStamp the current time;
you extracted the "timestamp" field using a regex (like this timestamp: (?<timestamp>.*)\s+total );
after you run my search:

index=main source=mysource | head 1 | rename timestamp AS _time | table timestamp _time

and you have different values between timestamp and _time?

I performed the same test and I had the same values, could you repeat this full test?

Bye.
Giuseppe

Re: Field showing an additional and not visible value --"none"-- under timestamp field

mgarciar — Mon, 18 Sep 2017 13:48:22 GMT

Hi,
I'm ingesting events from a Python script that is calling an API. This script writes to splunk the events as shown on my original post. When I do a search, like this:

index=main source=mysource | head 1

I can see Splunk returns a single event, everything looking good. But when add:

| table timestamp

where timestamp is a field from the json formatted event it returns two values, none and the actual date time value (as shown on the original picture). The none is not visible anywhere.

Re: Field showing an additional and not visible value --"none"-- under timestamp field

gcusello — Mon, 18 Sep 2017 13:53:53 GMT

Hi mgarciar,
in your python script try to use this search:

index=main source=mysource 
| head 1 
| rex "timestamp: (?<timestamp>.*)\s+total" 
| table timestamp

Bye.
Giuseppe

Re: Field showing an additional and not visible value --"none"-- under timestamp field

mgarciar — Mon, 18 Sep 2017 14:03:54 GMT

Hi Giuseppe,
Thank you for your suggestions. That will probably allow me to extract the correct value at search time.
But the problem persists right?, the question is more: why I'm getting that none value inserted on the field? Is it that timestamp is a reserved field maybe?

Re: Field showing an additional and not visible value --"none"-- under timestamp field

gcusello — Mon, 18 Sep 2017 14:08:47 GMT

Hi mgarciar,
probably you didn't correctly extracted the timestamp field, try to re configure field extraction enabled for the role that you use in python script and probably you'll not have the problem.

if you're satisfied by this answer, please accept or upvote it.

Bye.
Giuseppe

Re: Field showing an additional and not visible value --"none"-- under timestamp field

mgarciar — Mon, 18 Sep 2017 14:14:44 GMT

There is not field extraction at all. The value is "as is". That's the value coming from the Api (with the exception of the none). The _time field is just set to be the system date/time (for now).
There is not field extraction, props, transform, anything related with the "timestamp" field.

Re: Field showing an additional and not visible value --"none"-- under timestamp field

gcusello — Mon, 18 Sep 2017 14:19:00 GMT

try to add a field extraction for timestamp field using my regex, in this way you should use this field also in script.
Bye.
Giuseppe

Re: Field showing an additional and not visible value --"none"-- under timestamp field

gcusello — Wed, 20 Sep 2017 11:18:23 GMT

If this answer satisfies your question, please accept or upvote it.
Bye.
Giuseppe