BadgerDoc is a ML Delivery Platform made to make delivery process of Machine Learning Solutions visible to customer, managers and ML team. The primary goal of the platform is to visualize ML model delivery cycle - data annotation, model training and result visualization.
The platform has rich functionality in access and data management, annotation setups, and pipeline composition. Access management is based on Keycloak, which is integrated with EPAM Active Directory. Data can be uploaded in batches, organized into datasets as well as uploaded as a single file. ML pipeline can be applied to a dataset, which will trigger batch processing, or to a single document. BadgerDoc is capable of annotating large datasets by many annotators. It has algorithms for task distribution, validation roles, several validation setups and will have multicoverage of files by annotators in nearest future.
BadgerDoc also has steady growing number of pre-trained models available for users, which can be assembled into pipelines through visual editor.
Having such a rich functionality, BadgerDoc can be used for implementing full ML development cycle, as well as for rapid prototyping, demonstrating EPAM expertise in ML and even for large annotation project when preliminary annoation is available.
For now, BadgerDoc is working with vectorized and scanned documents, but it has capability of image annotation.
Run the following command to build the base image:
make build_base
After the base image is built, it is recommended to clean up any temporary files generated during the build process. To do this, run the following command:
make clean
Easiest way to build microservices is to run make build_all command, right after that,
it's possible run docker-compose to serve BadgerDoc in local mode.
If it's required to build separate microservice, just run make build_{microservice} command,
for instance: make build_users to build or rebuild users
After all services are built, you need to create .env file in root folder. You may just copy from example: cp .env.example .env
Time to run:
docker-compose -f docker-compose-dev.yaml up -d
Now services are running, but to start using BadgerDoc, some additional configuration steps are required
It's a good idea to automate this section
Important! This is not secure configuration, follow KeyCloak best practices to setup on production environment
-
Login into Keycloak using url http://127.0.0.1:8082/auth and
admin:adminas credentials -
Go to Realm Settings -> Keys and disable
RSA-OAEPalgorithm. It will help to avoid issue explainded here jpadilla/pyjwt#722 -
Add tenant attribute to
adminuser, go to Users -> selectadmin-> go to Attributes -> create attributtenants:local -
Go to Clients -> admin-cli -> Mappers -> Create and fill form with following values:
| Param | Value |
|---|---|
| Protocol | openid-connect |
| Name | tenants |
| Mapper Type | User Attribute |
| User Attribute | tenants |
| Token Claim Name | tenants |
| Claim JSON Type | string |
| Add to ID token | On |
| Add to access token | On |
| Add to userinfo | On |
| Multivalued | On |
| Aggregate attribute values | On |
-
Go to Client Scopes -> Find
rolesand select in list -> Scope -> Addadminto Assigned Roles, then go to Mappers and ensure that only 2 mappers exists:realm rolesandclient roles. Delete all other mappers -
Go to Clients -> Create -> Fill form and save
| Param | Value |
|---|---|
| Client ID | badgerdoc-internal |
| Client Protocol | openid-connect |
- Go to Cliens -> Find
badgerdoc-internal-> change settingsAccess Type: Confidential, setService Accounts EnabledtoOn, then save. Now you can Credentials tab, open it and copy Secret
Now Client ID and Secret must be set to .env as KEYCLOAK_SYSTEM_USER_CLIENT and KEYCLOAK_SYSTEM_USER_SECRET
- Go to Clients -> Find
badgerdoc-internal-> Service Account Roles -> Client Roles -> master-realm -> Findview-usersandview-identity-providersin Available Roles and add to Assigned Roles
Time to reload docker-compose, because .env was changed:
docker-compose -f docker-compose-dev.yaml up -d
- Install all required dependencies for a microservice using a packaging tool like Pipenv/Poetry depending on the microservice you are about to set up (we will use Pipenv and "assets" service for this example):
cd assets && pipenv install --dev
- Install dependencies from "lib" folder:
pipenv shell && pip install -e ../lib/filter_lib ../lib/tenants