I am updating a CSV on disk via the search api using outputlookup. Each time I run my script using the same source CSV, the CSV on disk is updated correctly. It shows the correct number of rows.
When I try to read the CSV via UI or the /jobs & /export API it doesn't return the correct number of rows approximately 40% of the time.
I run my script 10 times, and 6 times it's a perfect match, the row count is what I expect it to be, 4 times it fails with random number of rows returned. Each time, the number of rows in the CSV on disk is correct though. From what I can see, I am not hitting quotas or limits and it doesn't seem to be related to system load. There isn't anything obvious to me in any logs either.
My flow is like this:
API call with append=f to overwrite the CSV/set column names
Multiple API calls with append=t to add data (10 rows of data per call)
Final search to ensure the expected number of rows match
My search to validate is like this, and this is where the inconsistencies happen:
| inputlookup my_new_file.csv | table <column name, column name....>
My CSV is currently 110 columns, and 5625 rows.
When it works, search returns the correct number of rows each time. When it doesn't work, it consistently returns the wrong number. To clarify, if i get 500 rows instead of the expected 5625, every time i search i will get 500 rows until I re-run my script to update the CSV on disk.
Thanks
... View more