Getting Data In

Is there any way to do checkpoints or avoid duplicate events when make a REST API call?

ansif
Motivator

I am building an addon for Networker using REST API input.

Not able to configure checkpoints because REST API not support greater than or lesser than symbols to get last 5 mins data or so.

Is there any way to do checkpoints or avoid duplicate events when make a REST API call?

Labels (1)
0 Karma

mydog8it
Builder

The Dell EMC NetWorkers getting started with API documentation speaks to time filtering in chapter 4 page 35:
https://www.dellemc.com/pl-pl/collaterals/unauth/technical-guides-support-information/products/data-...

0 Karma

ansif
Motivator

This works with time ranges. How can I give this as checkpoint?

0 Karma

mydog8it
Builder

If you want to truly make this a checkpoint of the message, you might consider writing a checkpoint module that you script runs after it downloads the data. I have done something similar in powershell, but not python. I took a list of two columns, concatenated each pair and then hashed them. I then searched the hashes for previous matches.
You could do the same thing with the last message read. Hash all the messages and save and save the last one in a batch as a checkpoint. When you pull the next batch of messages only forward the ones that come in after your checkpoint hash is matched.

0 Karma

ansif
Motivator

Where to keep REST API response or which module I need to modify to call other script that takes the response JSON and do check ?

0 Karma

dmarling
Builder

If you could provide the rest call that is failing and the failure message that would help us answer your question. My first guess is if you are attempting to pass in a greater than or less than symbol in the body of the transmission, you most likely need to make it a utf-8 url encoded greater than: %3E or less than: %3C

If this comment/answer was helpful, please up vote it. Thank you.
0 Karma

ansif
Motivator

The REST API itself not supports to query timestamp greater than or lesser than.

For this situation I need help to adjust in script to create a checkpoint of id's that is indexed already.

https://${nw_server_hostname}:9090/nwrestapi/v3/global/${endpoint}?q=completionStatus: "Failed"

This end point gives me backup failed events like below:

{
"count":6,
"jobs":[
{
"adhocJob":false,
"command":"/xxx/xxxx/xxxxxxx -C DefaultReportHomeTask",
"completionStatus":"Failed",
"dependentJobIds":[
0
],
"endTime":"2020-01-16T08:00:00+08:00",
"exitCode":1,
"id":221372,
"itemIdLong":221372,
"links":[
{
"href":"https://{nw_server_hostname}:9090/nwrestapi/v3/global/jobs/221372",
"rel":"item"
}
],
"name":"DefaultReportHomeTask",
"ndmp":false,
"parentJobId":0,
"previousJobId":0,
"rootParentJobId":0,
"runOnHost":"{nw_server_hostname}.xxxxxxx.com",
"siblingJobIds":[
],
"startTime":"2020-01-16T08:00:00+08:00",
"state":"Completed",
"stopped":true,
"tenant":"",
"type":"task job"
},
{
"adhocJob":false,
"command":"/xxx/xxxx/xxxxxxx -C DefaultNsrclientfixTask",
"completionStatus":"Failed",
"dependentJobIds":[
0
],
"endTime":"2020-01-16T07:00:00+08:00",
"exitCode":1,
"id":221371,
"itemIdLong":221371,
"links":[
{
"href":"https://{nw_server_hostname}:9090/nwrestapi/v3/global/jobs/221371",
"rel":"item"
}
],
"name":"DefaultNsrclientfixTask",
"ndmp":false,
"parentJobId":0,
"previousJobId":0,
"rootParentJobId":0,
"runOnHost":"{nw_server_hostname}.xxxxxxx.com",
"siblingJobIds":[
],
"startTime":"2020-01-16T07:00:00+08:00",
"state":"Completed",
"stopped":true,
"tenant":"",
"type":"task job"
},
{
"adhocJob":false,
"command":"/xxx/xxxx/xxxxxxx -C DefaultReportHomeTask",
"completionStatus":"Failed",
"dependentJobIds":[
0
],
"endTime":"2020-01-15T08:00:00+08:00",
"exitCode":1,
"id":221124,
"itemIdLong":221124,
"links":[
{
"href":"https://{nw_server_hostname}:9090/nwrestapi/v3/global/jobs/221124",
"rel":"item"
}
],
"name":"DefaultReportHomeTask",
"ndmp":false,
"parentJobId":0,
"previousJobId":0,
"rootParentJobId":0,
"runOnHost":"{nw_server_hostname}.xxxxxxx.com",
"siblingJobIds":[
],
"startTime":"2020-01-15T08:00:00+08:00",
"state":"Completed",
"stopped":true,
"tenant":"",
"type":"task job"
},
{
"adhocJob":false,
"command":"/xxx/xxxx/xxxxxxx -C DefaultNsrclientfixTask",
"completionStatus":"Failed",
"dependentJobIds":[
0
],
"endTime":"2020-01-15T07:00:00+08:00",
"exitCode":1,
"id":221123,
"itemIdLong":221123,
"links":[
{
"href":"https://{nw_server_hostname}:9090/nwrestapi/v3/global/jobs/221123",
"rel":"item"
}
],
"name":"DefaultNsrclientfixTask",
"ndmp":false,
"parentJobId":0,
"previousJobId":0,
"rootParentJobId":0,
"runOnHost":"{nw_server_hostname}.xxxxxxx.com",
"siblingJobIds":[
],
"startTime":"2020-01-15T07:00:00+08:00",
"state":"Completed",
"stopped":true,
"tenant":"",
"type":"task job"
},
{
"adhocJob":false,
"command":"/xxx/xxxx/xxxxxxx -C DefaultReportHomeTask",
"completionStatus":"Failed",
"dependentJobIds":[
0
],
"endTime":"2020-01-14T08:00:00+08:00",
"exitCode":1,
"id":220875,
"itemIdLong":220875,
"links":[
{
"href":"https://{nw_server_hostname}:9090/nwrestapi/v3/global/jobs/220875",
"rel":"item"
}
],
"name":"DefaultReportHomeTask",
"ndmp":false,
"parentJobId":0,
"previousJobId":0,
"rootParentJobId":0,
"runOnHost":"{nw_server_hostname}.xxxxxxx.com",
"siblingJobIds":[
],
"startTime":"2020-01-14T08:00:00+08:00",
"state":"Completed",
"stopped":true,
"tenant":"",
"type":"task job"
},
{
"adhocJob":false,
"command":"/xxx/xxxx/xxxxxxx -C DefaultNsrclientfixTask",
"completionStatus":"Failed",
"dependentJobIds":[
0
],
"endTime":"2020-01-14T07:00:00+08:00",
"exitCode":1,
"id":220874,
"itemIdLong":220874,
"links":[
{
"href":"https://{nw_server_hostname}:9090/nwrestapi/v3/global/jobs/220874",
"rel":"item"
}
],
"name":"DefaultNsrclientfixTask",
"ndmp":false,
"parentJobId":0,
"previousJobId":0,
"rootParentJobId":0,
"runOnHost":"{nw_server_hostname}.xxxxxxx.com",
"siblingJobIds":[
],
"startTime":"2020-01-14T07:00:00+08:00",
"state":"Completed",
"stopped":true,
"tenant":"",
"type":"task job"
}
]
}

I can only do a check point with endTime field.

But cant make a call with "endTime" greater than some timestamp.The vendor REST API is not supported with greater than or less than symbols(https://www.support.nec.co.jp/DownLoad.aspx?file=document%2Fdocu91962_NetWorker-18.2-REST-API-Gettin...)

Is there any way to do a checkpoint so that I can make sure that no duplicate events I am getting everytime make a REST API call.

NB:- id field is unique for each event. Is ther a way to do checkpoint using id field?

0 Karma

jamsfreak
New Member

Nice responde

 

but 

 

How to monitoring jobs with completionStatus = Failed and Abandoned

How to endpoint is made?

 

https://${nw_server_hostname}:9090/nwrestapi/v3/global/${endpoint}?q=completionStatus: "Failed" ????

 

Failed and status abandone

Try it in your lab and post the json response here

 

 

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...