Skip to content

Allow for batched models to work in trt pipelines#6

Open
finlay-hudson wants to merge 1 commit intoplaygroundai-legacy:nihanth/batchingfrom
finlay-hudson:finlay/allow_batching
Open

Allow for batched models to work in trt pipelines#6
finlay-hudson wants to merge 1 commit intoplaygroundai-legacy:nihanth/batchingfrom
finlay-hudson:finlay/allow_batching

Conversation

@finlay-hudson
Copy link

@finlay-hudson finlay-hudson commented Feb 12, 2024

This PR is to allow for batch mode to work with the TRT models in both img2img and txt2img pipelines. It mimics batching in default pytorch method, whereby all images have to be of consistent sizes throughout the batch.

Speeds

Tested on 50 iterations, base model, txt2img pipeline

Batch Size Total Time (s) Time Per Image (s)
1 5.26 5.26
4 19.78 4.94
8 38.39 4.79

Tested on 37 iterations, base model, img2img pipeline

Batch Size Total Time (s) Time Per Image (s)
1 4.09 4.09
4 15.45 3.85
8 30.25 3.78

Tested on 37 iterations, refiner model, img2img pipeline

Batch Size Total Time (s) Time Per Image (s)
1 4.48 4.48
4 17.24 4.31
8 33.53 4.19

@height
Copy link

height bot commented Feb 12, 2024

Link Height tasks by mentioning a task ID in the pull request title or commit messages, or description and comments with the keyword link (e.g. "Link T-123").

💡Tip: You can also use "Close T-X" to automatically close a task when the pull request is merged.

@finlay-hudson finlay-hudson marked this pull request as ready for review February 12, 2024 14:21
@nhnt11 nhnt11 changed the base branch from tyler/spectral_2 to nihanth/batching February 29, 2024 18:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant