Backend API Documentation
This project contains a private API backend for processing videos and images, written in Python using FastAPI.
Flow of operations
Start a job by calling one of the /process_* endpoints.
Job created with tasks associated with it will be returned along with total frame counts to use for credit deduction.
Approve the job with /approve_job endpoint. This will add tasks of the job to the Backbone queue for processing.
Use /job_status query job status. Status codes returned by this endpoint are as follows:
| status | description |
|---|---|
| 0 | awaiting i.e not approved |
| 1-99 | sent to backbone |
| 100 | all tasks complete (might have tasks with error) |
If a job has multiple tasks their progress is calculated as a non-weighted average of their progress.
Use /task_status query task status. Status codes returned by this endpoint are as follows:
| status | description |
|---|---|
| 0 | awaiting |
| 1-99 | processing |
| 100 | task completed successfully |
| 200-300 | error. processing failed |
Once 100 is returned for a job, the job has been completed and the processed files can be retrieved from path of the output_file for each task.
There are 4 endpoints that can be used to input data for processing which are detailed below.
/process_file/process_video/process_image/process_image_upload
Calling one of these endpoints with valid inputs will create a processing job and a task for each valid image/video file given.
Assuming an archive file with two videos inside is requested, two task folders with their UUIDs as names and a parent job folder will be created under jobs directory. Task folders only contain the associated unprocessed input file inside. The created job is also inserted into the mongoDB/jobs collection.
jobs
└── 002cf9d5-9dbd-49c1-ab8b-70148edc36c4 job_id
├── b7d41449-2e80-4ed8-913d-aa821bfce8e7 task_id 1
│ └── sample.mp4
└── f09cb301-db57-4073-b79a-3efff3e93fd4 task_id 2
└── sample-10s.mp4
Next step is to approve this request via the /approve_job endpoint.
Once approved the backend will add approved job's tasks to the tasks collection.
The backbone constantly checks for changes to the tasks collection and sends the input file to a suitable worker for processing.
Workers update task progress on tasks collection every %10. If an error has occurred during processing the task_status is updated with a value between 200-300. If the status is 100, the task processing was successful.
After both files are processed, the file structure should look similar to this:
jobs
└── 002cf9d5-9dbd-49c1-ab8b-70148edc36c4
├── b7d41449-2e80-4ed8-913d-aa821bfce8e7
│ ├── sample.mp4
│ └── sample_output.mp4
└── f09cb301-db57-4073-b79a-3efff3e93fd4
├── sample-10s.mp4
└── sample-10s_output.mp4
The job process can be manually checked using the /job_status endpoint.
Alternatively, there are 4 webhook services provided by the API for getting job/task status updates.
/webhook/notify_on_task_update//webhook/notify_on_task_completion//webhook/notify_on_job_update//webhook/notify_on_job_completion/
Job/Task IDs can be subscribed to for updates with a return URL.
Once the associated Job/Task has a progress update, the API will POST Job/Task details as JSON to the return URL.
Note that the POST request will be only sent once per subscription to _completion endpoints and will be removed after.
Similary, a subscription on _update endpoints will be removed once the task is completed or if an error occurs.
/process_file
Process file(s) with given parameter settings in;
-
A supported video in format:
['.mp4', '.mkv', '.webm', '.ts', '.mov', '.avi'] -
A supported image in format:
['.jpg', '.png', '.jpeg', '.bmp'] -
An archive file in one of the supported formats:
['.zip','.rar','.7z']
If the file is an archive file, it will be extracted and all the files in the archive will be processed.
Nesting in archive is not supported. Files must be in the root directory of the archive.
Processing won't start until the job is approved with /approve_job
/process_video
Process video file(s) with given parameter settings in;
-
A supported video in format:
['.mp4', '.mkv', '.webm', '.ts', '.mov', '.avi'] -
An archive file in one of the supported formats:
['.zip','.rar','.7z']
If the file is an archive file, it will be extracted and all the files in the archive will be processed.
Nesting in archive is not supported. Files must be in the root directory of the archive.
Processing won't start until the job is approved with /approve_job
/process_image
Process image file(s) with given parameter settings in;
-
A supported image in format:
['.jpg', '.png', '.jpeg', '.bmp'] -
An archive file in one of the supported formats:
['.zip','.rar','.7z']
If the file is an archive file, it will be extracted and all the files in the archive will be processed.
Nesting in archive is not supported. Files must be in the root directory of the archive.
Processing won't start until the job is approved with /approve_job
/process_image_upload
Upload and process the uploaded image file with given parameter settings in a supported image in format: ['.jpg', '.png', '.jpeg', '.bmp']
Supports single image file upload only.
Processing won't start until the job is approved with /approve_job