-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathindex.html
More file actions
345 lines (307 loc) · 16.3 KB
/
index.html
File metadata and controls
345 lines (307 loc) · 16.3 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<!-- Meta tags for social media banners, these should be filled in appropriatly as they are your "business card" -->
<!-- Replace the content tag with appropriate information -->
<meta name="description" content="DESCRIPTION META TAG">
<meta property="og:title" content="TCTS & FAS"/>
<meta property="og:description" content="Text-Conditioned Sampling Framework for Text-to-Image Generation with Masked Generative Models"/>
<meta property="og:url" content="URL OF THE WEBSITE"/>
<!-- Path to banner image, should be in the path listed below. Optimal dimenssions are 1200X630-->
<!-- meta property="og:image" content="static/image/your_banner_image.png"/ -->
<!--meta property="og:image:width" content="1200"/-->
<!--meta property="og:image:height" content="630"/ -->
<meta name="twitter:title" content="TWITTER BANNER TITLE META TAG">
<meta name="twitter:description" content="TWITTER BANNER DESCRIPTION META TAG">
<!-- Path to banner image, should be in the path listed below. Optimal dimenssions are 1200X600-->
<meta name="twitter:image" content="static/images/your_twitter_banner_image.png">
<meta name="twitter:card" content="summary_large_image">
<!-- Keywords for your paper to be indexed by-->
<meta name="keywords" content="KEYWORDS SHOULD BE PLACED HERE">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>TCTS_FAS</title>
<!-- link rel="icon" type="image/x-icon" href="static/images/favicon.ico"-->
<link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro"
rel="stylesheet">
<link rel="stylesheet" href="static/css/bulma.min.css">
<link rel="stylesheet" href="static/css/bulma-carousel.min.css">
<link rel="stylesheet" href="static/css/bulma-slider.min.css">
<link rel="stylesheet" href="static/css/fontawesome.all.min.css">
<link rel="stylesheet"
href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css">
<link rel="stylesheet" href="static/css/index.css">
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script>
<script src="https://documentcloud.adobe.com/view-sdk/main.js"></script>
<script defer src="static/js/fontawesome.all.min.js"></script>
<script src="static/js/bulma-carousel.min.js"></script>
<script src="static/js/bulma-slider.min.js"></script>
<script src="static/js/index.js"></script>
</head>
<body>
<section class="hero">
<div class="hero-body">
<div class="container is-max-desktop">
<div class="columns is-centered">
<div class="column has-text-centered">
<h1 class="title is-1 publication-title">Text-Conditioned Sampling Framework for Text-to-Image Generation with Masked Generative Models</h1>
<div class="is-size-5 publication-authors">
<!-- Paper authors -->
<span class="author-block">
Jaewoong Lee</a><sup>*,1</sup>,</span>
<span class="author-block">
<!--a href="SECOND AUTHOR PERSONAL LINK" target="_blank"-->
Sangwon Jang</a><sup>*,2</sup>,</span>
<span class="author-block">
Jaehyeong Jo</a><sup>1</sup>,</span>
<span class="author-block">
Jaehong Yoon</a><sup>1</sup>,</span>
<span class="author-block">
Yunji Kim</a><sup>3</sup>,</span>
<span class="author-block">
Jin-Hwa Kim</a><sup>3</sup>,</span>
<span class="author-block">
Jung-Woo Ha</a><sup>3</sup>,</span>
<span class="author-block">
Sung Ju Hwang</a><sup>1</sup>
</span>
</div>
<div class="is-size-5 publication-authors">
<span class="author-block"><sup>1 </sup>KAIST, <sup>2 </sup>Yonsei University, <sup>3 </sup>NAVER AI Lab<br>ICCV 2023</span>
<span class="eql-cntrb"><small><br><sup>*</sup>Indicates Equal Contribution</small></span>
</div>
<div class="column has-text-centered">
<div class="publication-links">
<!-- Arxiv PDF link -->
<span class="link-block">
<a href="https://arxiv.org/pdf/2304.01515.pdf" target="_blank"
class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="fas fa-file-pdf"></i>
</span>
<span>Paper</span>
</a>
</span>
<!-- Supplementary PDF link -->
<span class="link-block">
<a href="static/pdfs/supplementary_material.pdf" target="_blank"
class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="fas fa-file-pdf"></i>
</span>
<span>Supplementary</span>
</a>
</span>
<!-- Github link -->
<span class="link-block">
<a href="https://github.com/YOUR REPO HERE" target="_blank"
class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="fab fa-github"></i>
</span>
<span>Code(Coming soon)</span>
</a>
</span>
<!-- ArXiv abstract Link -->
<span class="link-block">
<a href="https://arxiv.org/abs/2304.01515" target="_blank"
class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="ai ai-arxiv"></i>
</span>
<span>arXiv</span>
</a>
</span>
</div>
</div>
</div>
</div>
</div>
</div>
</section>
<!-- Paper abstract -->
<section class="section hero is-light">
<div class="container is-max-desktop">
<div class="columns is-centered has-text-centered">
<div class="column is-four-fifths">
<h2 class="title is-3">Abstract</h2>
<div class="content has-text-justified">
<p>
Token-based masked generative models are gaining popularity for their fast inference time with parallel decoding. While recent token-based approaches achieve competitive performance to diffusion-based models, their generation performance is still suboptimal as they sample multiple tokens simultaneously without considering the dependence among them. We empirically investigate this problem and propose a learnable sampling model, Text-Conditioned Token Selection (TCTS), to select optimal tokens via localized supervision with text information. TCTS improves not only the image quality but also the semantic alignment of the generated images with the given texts. To further improve the image quality, we introduce a cohesive sampling strategy, Frequency Adaptive Sampling (FAS), to each group of tokens divided according to the self-attention maps. We validate the efficacy of TCTS combined with FAS with various generative tasks, demonstrating that it significantly outperforms the baselines in image-text alignment and image quality. Our text-conditioned sampling framework further reduces the original inference time by more than 50% without modifying the original generative model.
</p>
</div>
</div>
</div>
</div>
</section>
<!-- End paper abstract -->
<!-- Image carousel -->
<section class="hero is-small">
<div class="hero-body">
<div class="container">
<h2 class="title">Motivation</h2>
<div id="results-carousel" class="carousel results-carousel">
<div class="item">
<!-- Your image here -->
<img src="static/images/plane2.png" alt="MY ALT TEXT"
style="display: block; margin: 0 auto; height:500px;"
/>
<h2 class="subtitle has-text-centered">
<b>Reconstructed image using x̂<sub>0</sub><sup>(t)</sup> each step during diffusion using the fixed and revocable method.</b><br>
Only the revocable method is able to edit the tokens to generate a plane according to the text
“<i>A view of the end of an airplane in the sky over mountains</i>”.
</h2>
</div>
<div class="item">
<!-- Your image here -->
<img src="static/images/Birds.png" alt="MY ALT TEXT"
style="display: block; margin: 0 auto; height:500px;"
/>
<h2 class="subtitle has-text-centered">
<b>Reconstructed images using x̂<sub>0</sub><sup>(t)</sup> each step during diffusion using the revocable method with and without FAS.</b><br>
The backgrounds of the images in the top row are over-simplified while our proposed FAS prevents this, as shown in the bottom row.
The text is “<i>This small bird is greyish in color with flecks of yellow on the back and breast, and a bit of white on the belly</i>”.
</h2>
</div>
</div>
</div>
</div>
</section>
<!-- End image carousel -->
<!-- Image carousel -->
<section class="hero is-small">
<div class="hero-body">
<div class="container">
<h2 class="title">Method</h2>
<div id="results-carousel" class="carousel results-carousel">
<div class="item">
<!-- Your image here -->
<img src="static/images/figure1.png" alt="MY ALT TEXT"
style="display: block; margin: 0 auto; height:500px;"
/>
<h2 class="subtitle has-text-centered">
<b>Overall generation framework of proposed TCTS (Text-Conditioned Token Selection) and FAS (Frequency Adaptive Sampling).</b> <br> After the generator predicts the tokens, TCTS exploits the text condition to detect misaligned tokens and outputs the score map. Meanwhile, FAS splits the tokens according to the frequency using the self-attention map from the generator, performing revocable sampling to high-frequency split and persistent sampling to low-frequency split. The adaptive sampling predicts x̂<sub>0</sub> and decide a few of the locations to mask according to x<sub>t</sub>. The token maps produced by FAS is combined with the score map to predict x<sub>t-1</sub>.
</h2>
</div>
<div class="item">
<!-- Your image here -->
<img src="static/images/test.png" alt="MY ALT TEXT"
style="display: block; width:1632px;"
/>
<h2 class="subtitle has-text-centered">
<b>Generated samples on MS-COCO dataset and evaluation graph of various sampling methods showing their trade-off.</b> <br>Uniform sampling is a fixed strategy with notably poor text alignment compared to other methods. Random revoke sampling is a revocable strategy with improved text alignment. Ours is TCTS combined with FAS, where both the image quality and the text alignment are significantly better compared to those of baselines. Metrics are measured on all 40K images with their corresponding single caption. The classifier-free guidance scale was fixed at 5 for all sampling methods.
</h2>
</div>
</div>
</div>
</div>
</section>
<!-- End image carousel -->
<!-- Image carousel -->
<section class="hero is-small">
<div class="hero-body">
<div class="container">
<h2 class="title">Experiments</h2>
<div id="results-carousel" class="carousel results-carousel">
<div class="item">
<!-- Your image here -->
<img src="static/images/mask_free.png" alt="MY ALT TEXT"
style="display: block; margin: 0 auto; height:400px;"
/>
<h2 class="subtitle has-text-centered">
<b>Examples of mask-free editing samples with cross-attention map.</b> <br>Motivated by the operation of self-attention maps in frequency adaptive sampling, we leverage a cross-attention map corresponding to the word of the object that is to be changed, giving weights to resample tokens so that the corresponding locations can be resampled. The cross-attention map is multiplied to the score map of TCTS to perform weighted sampling.
</h2>
</div>
<div class="item">
<!-- Your image here -->
<img src="static/images/Appendix_mask_free.png" alt="MY ALT TEXT"
style="display: block; margin: 0 auto; height:400px;"
/>
<h2 class="subtitle has-text-centered">
<b>Comparing mask-free object editing samples with and without cross-attention map weighted sampling.</b> <br>Starting from the image on the left, the result images every 20 steps of editing with 30% masking ratio. <b>Top</b>: Failure case without weighted sampling, <b>Bottom</b>: Results with weighted sampling.
</h2>
</div>
<div class="item">
<!-- Your image here -->
<img src="static/images/image_refine2.png" alt="MY ALT TEXT"
style="display: block; margin: 0 auto; height:400px;"
/>
<h2 class="subtitle has-text-centered">
<b>Results of image refinement using TCTS.</b><br> <b>Top</b>: Original samples, <b>Bottom</b>: Refined images for 8-steps with TCTS.
</h2>
</div>
</div>
</div>
</div>
</section>
<!-- End image carousel -->
<section class="hero is-small">
<div class="hero-body">
<!-- Your image here -->
<img src="static/images/Appendix_step_perf.png" alt="MY ALT TEXT"
style="display: block; margin: 0 auto; height:300px;"
/>
<h2 class="subtitle has-text-centered">
<b>Performance comparison of each method at different steps. </b><br>In our experiments, we fixed classifier-free guidance to 5. When
we use FAS method, it was possible to lower the FID score while maintaining text alignment.
</h2>
</div>
</section>
<section class="hero is-small">
<div class="hero-body">
<!-- Your image here -->
<img src="static/images/Appendix_time_perf.png" alt="MY ALT TEXT"
style="display: block; margin: 0 auto; height:300px;"
/>
<h2 class="subtitle has-text-centered">
<b>Comparison of our model and the baseline in performance over generation time. </b><br>In our experiments, we fixed classifier-free guidance to 5.
</h2>
</div>
</section>
<!-- Paper poster -->
<section class="hero is-small is-light">
<div class="hero-body">
<div class="container">
<h2 class="title">Poster</h2>
<iframe src="static/pdfs/iccv23_poster.pdf" width="100%" height="550">
</iframe>
</div>
</div>
</section>
<!--End paper poster -->
<!--BibTex citation -->
<section class="section" id="BibTeX">
<div class="container is-max-desktop content">
<h2 class="title">BibTeX</h2>
<pre><code>@misc{lee2023textconditioned,
title={Text-Conditioned Sampling Framework for Text-to-Image Generation with Masked Generative Models},
author={Jaewoong Lee and Sangwon Jang and Jaehyeong Jo and Jaehong Yoon and Yunji Kim and Jin-Hwa Kim and Jung-Woo Ha and Sung Ju Hwang},
year={2023},
eprint={2304.01515},
archivePrefix={arXiv},
primaryClass={cs.LG}
}
</code></pre>
</div>
</section>
<!--End BibTex citation -->
<footer class="footer">
<div class="container">
<div class="columns is-centered">
<div class="column is-8">
<div class="content">
<p>
This page was built using the <a href="https://github.com/eliahuhorwitz/Academic-project-page-template" target="_blank">Academic Project Page Template</a> which was adopted from the <a href="https://nerfies.github.io" target="_blank">Nerfies</a> project page.
You are free to borrow the of this website, we just ask that you link back to this page in the footer. <br> This website is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/" target="_blank">Creative
Commons Attribution-ShareAlike 4.0 International License</a>.
</p>
</div>
</div>
</div>
</div>
</footer>
<!-- Statcounter tracking code -->
<!-- You can add a tracker to track page visits by creating an account at statcounter.com -->
<!-- End of Statcounter Code -->
</body>
</html>