I have a python script which makes an API call and get the events . Number of events, its collecting are correct however its adding duplicate entries per field. Can you please assist, I am ?
Here is my script
response = helper.send_http_request(rest_url, 'GET' ,parameters=queryParam, payload=None,headers=headers, cookies=None,verify=False, cert=None, timeout=None, use_proxy=False)
r_headers = response.headers
r_json = response.json()
r_status = response.status_code
if r_status !=200:
response.raise_for_status()
final_result = []
for _file in r_json:
responseStr=''
fileid = str(_file["fileid"])
state = helper.get_check_point(str(fileid))
if state is None:
final_result.append(_file)
helper.save_check_point(str(str(fileid)), "Indexed")
event=helper.new_event(json.dumps(final_result), time=None, host=None, index=None, source=None, sourcetype=None, done=True, unbroken=True)
ew.write_event(event)
response:
[
{
"fileid": "abc.txt",
"source": "source1",
"destination": "dest1",
"servername": "server1",
},
{
"fileid": "xyz.txt",
"source": "source2",
"destination": "dest2",
"servername": "server2",
}
]
Response after collecting data to Index looks as below:
| fileid | source | destination | servername |
"abc.txt abc.txt | source1 source1 | dest1 dest1 | server1 server1 |
xyz.txt xyz.txt | source2 source2 | dest2 dest2 | server2 server2 |
