Create a batch job

The batch endpoint will take a file hash then fetch and parse that file into locations. Upon receiving the file hash, it will create a job in our jobs table with the status scheduled and return the hash of that job. There will be an additional endpoint to fetch the status of a job and then a further endpoint to fetch the results from a job, this endpoint should be paginated and limited to 500 records per page.

Once a job has been added to the jobs table, it should also dispatch a job to our queue, that job should fetch the file from our files service, update the status of the job to processing and then for each row in our file add a another job to our queue to process that line, such that every line in a CSV or JSON file we fetch gets its own job. When all the jobs are complete we should update the status of the job in our database to complete . When each job has parsed its row, it should update the completed or failed values for the job, it should also add an entry to the job_record table with the id of the location and the id of the job.

The endpoint to fetch results for a job should only return results when the status of the job is complete

When a batch job is created we should trigger the event location.batch.scheduled with the job hash and the file hash in the payload. When an event starts it should trigger the location.batch.started event with the job hash and file hash in the payload. When a row job parses a location it should trigger the event location.batch.tick with the job hash and the location_hash of the location it created (of fetched), if a parse failed it should trigger the location.batch.error with the job hash and a reason as the payload. The reason should just be row failed . When the batch has finished processing it should dispatch the event location.batch.completed with the job hash and file hash in the payload. If a job totally fails it should dispatch the event location.batch.failed with the job hash and file hash in the payload. All events should also have a timestamp.

We will only parse files in CSV and JSON format. The CSV should contain a single column, with no header, with the values to lookup. The JSON should be a JSON array with one value per row of the array. If a job can’t be processed you should update the status of the job to failed .

This endpoint requires an Admin JWT

Add the parameter

Authorization

to Headers，whose value is to concatenate the Token after the Bearer.

Example:

Authorization: Bearer ********************

application/json

Bodyapplication/json

Example

{
    "hash": "string",
    "file_hash": "string",
    "status": "scheduled",
    "rows": 0,
    "completed": 0,
    "failed": 0,
    "started_at": "2019-08-24T14:15:22.123Z",
    "finished_at": "2019-08-24T14:15:22.123Z"
}

Create a batch job

Request

Request Code Samples

Responses