{"version":2,"relationshipURLBase":"https://codingai-lec03.pages.dev/course/notebook-blocks/f0aeadb1c331456f96ceb967cfb16a4c.html","relationshipResolution":"Resolve relative relationship URLs against relationshipURLBase; complete records and absolute URLs are in blocks.","title":"CLIP 使用已配对的图像与文字，同时训练两种编码器","url":"https://codingai-lec03.pages.dev/course/notebook-blocks/f0aeadb1c331456f96ceb967cfb16a4c.html","markdown":"https://codingai-lec03.pages.dev/course/notebook-blocks/f0aeadb1c331456f96ceb967cfb16a4c.md","pages":[{"id":"reading-opening","title":"CLIP 使用已配对的图像与文字，同时训练两种编码器","unit":"CLIP 使用已配对的图像与文字，同时训练两种编码器","unitId":"notebook-blocks/f0aeadb1c331456f96ceb967cfb16a4c","chapter":"course","number":1}],"aliases":[{"oldId":"b-be00d7336e89b03df1575dfb","targetId":"b-be00d7336e89b03df1575dfb","page":"reading-opening","operation":"move","beforeSHA256":"343fb71807f077069b41c6495a8824ea4d353c740175b5775ccc5145b3bf19ca","afterSHA256":"343fb71807f077069b41c6495a8824ea4d353c740175b5775ccc5145b3bf19ca","reason":"Same source body in authored semantic page"},{"oldId":"b-b5ae6e130c5f5f2113f85351","targetId":"b-b5ae6e130c5f5f2113f85351","page":"reading-opening","operation":"move","beforeSHA256":"f733ba1a67125501cad1752e2618be51ac26e0e42b144557f26eea1ed3540a82","afterSHA256":"f733ba1a67125501cad1752e2618be51ac26e0e42b144557f26eea1ed3540a82","reason":"Same source body in authored semantic page"},{"oldId":"b-6bdd0d448452375de65d5137","targetId":"b-6bdd0d448452375de65d5137","page":"reading-opening","operation":"move","beforeSHA256":"42cba31deccba2a350b9b23bd0ca1dcaa4c4c5cb9d35c773e1fcb89124788ccd","afterSHA256":"42cba31deccba2a350b9b23bd0ca1dcaa4c4c5cb9d35c773e1fcb89124788ccd","reason":"Same source body in authored semantic page"},{"oldId":"b-380df9d9ebc8c9d96a34484f","targetId":"b-380df9d9ebc8c9d96a34484f","page":"reading-opening","operation":"move","beforeSHA256":"41c70c2c07b82678ff0a60f2d009b6afa750c5c691d109e754a69abf23868b50","afterSHA256":"41c70c2c07b82678ff0a60f2d009b6afa750c5c691d109e754a69abf23868b50","reason":"Same source body in authored semantic page"},{"oldId":"b-aa57811c0f0db1abb017137e","targetId":"b-aa57811c0f0db1abb017137e","page":"reading-opening","operation":"move","beforeSHA256":"7ca98a60191efe95460cc789edd63b4dcd46b8e818d377021613b48bd73a2582","afterSHA256":"7ca98a60191efe95460cc789edd63b4dcd46b8e818d377021613b48bd73a2582","reason":"Same source body in authored semantic page"},{"oldId":"b-12238ea6a86b904efd927631","targetId":"b-12238ea6a86b904efd927631","page":"reading-opening","operation":"move","beforeSHA256":"8cbcdacec033c71a8883d97e2d9160a22887dde95646104c6a75b045185b1614","afterSHA256":"8cbcdacec033c71a8883d97e2d9160a22887dde95646104c6a75b045185b1614","reason":"Same source body in authored semantic page"},{"oldId":"b-e10dc3e574540aae596d8b26","targetId":"b-e10dc3e574540aae596d8b26","page":"reading-opening","operation":"move","beforeSHA256":"a22143a5d7eb7b02fb2f6b604dc5b90425bbfb2492a3e768276483cb768bb024","afterSHA256":"a22143a5d7eb7b02fb2f6b604dc5b90425bbfb2492a3e768276483cb768bb024","reason":"Same source body in authored semantic page"}],"blocks":[{"id":"b-be00d7336e89b03df1575dfb","type":"p","scope":"document","url":"https://codingai-lec03.pages.dev/course/notebook-blocks/f0aeadb1c331456f96ceb967cfb16a4c.html#b-be00d7336e89b03df1575dfb","content":"[CLS 03 · 视觉与语言：任务、训练与接口](/course/notebooks/680da245145447dd88983aabd136fd24.html)","sha256":"343fb71807f077069b41c6495a8824ea4d353c740175b5775ccc5145b3bf19ca","chapter":{"id":"course","title":"CLIP 使用已配对的图像与文字，同时训练两种编码器"},"unit":{"id":"notebook-blocks/f0aeadb1c331456f96ceb967cfb16a4c","title":"CLIP 使用已配对的图像与文字，同时训练两种编码器"},"page":{"id":"reading-opening","title":"CLIP 使用已配对的图像与文字，同时训练两种编码器","number":1},"sectionTitle":"CLIP 使用已配对的图像与文字，同时训练两种编码器","currentQuestion":"CLIP 使用已配对的图像与文字，同时训练两种编码器","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":null,"nextBlock":"b-b5ae6e130c5f5f2113f85351","prerequisites":[],"relatedBlocks":[]},{"id":"b-b5ae6e130c5f5f2113f85351","type":"h1","scope":"document","url":"https://codingai-lec03.pages.dev/course/notebook-blocks/f0aeadb1c331456f96ceb967cfb16a4c.html#b-b5ae6e130c5f5f2113f85351","content":"# CLIP 使用已配对的图像与文字，同时训练两种编码器","sha256":"f733ba1a67125501cad1752e2618be51ac26e0e42b144557f26eea1ed3540a82","chapter":{"id":"course","title":"CLIP 使用已配对的图像与文字，同时训练两种编码器"},"unit":{"id":"notebook-blocks/f0aeadb1c331456f96ceb967cfb16a4c","title":"CLIP 使用已配对的图像与文字，同时训练两种编码器"},"page":{"id":"reading-opening","title":"CLIP 使用已配对的图像与文字，同时训练两种编码器","number":1},"sectionTitle":"CLIP 使用已配对的图像与文字，同时训练两种编码器","currentQuestion":"CLIP 使用已配对的图像与文字，同时训练两种编码器","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-be00d7336e89b03df1575dfb","nextBlock":"b-6bdd0d448452375de65d5137","prerequisites":[{"id":"b-be00d7336e89b03df1575dfb","url":"#b-be00d7336e89b03df1575dfb","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-6bdd0d448452375de65d5137","type":"p","scope":"document","url":"https://codingai-lec03.pages.dev/course/notebook-blocks/f0aeadb1c331456f96ceb967cfb16a4c.html#b-6bdd0d448452375de65d5137","content":"主线阅读与优先实验","sha256":"42cba31deccba2a350b9b23bd0ca1dcaa4c4c5cb9d35c773e1fcb89124788ccd","chapter":{"id":"course","title":"CLIP 使用已配对的图像与文字，同时训练两种编码器"},"unit":{"id":"notebook-blocks/f0aeadb1c331456f96ceb967cfb16a4c","title":"CLIP 使用已配对的图像与文字，同时训练两种编码器"},"page":{"id":"reading-opening","title":"CLIP 使用已配对的图像与文字，同时训练两种编码器","number":1},"sectionTitle":"CLIP 使用已配对的图像与文字，同时训练两种编码器","currentQuestion":"CLIP 使用已配对的图像与文字，同时训练两种编码器","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-b5ae6e130c5f5f2113f85351","nextBlock":"b-380df9d9ebc8c9d96a34484f","prerequisites":[{"id":"b-b5ae6e130c5f5f2113f85351","url":"#b-b5ae6e130c5f5f2113f85351","role":"section-title"},{"id":"b-be00d7336e89b03df1575dfb","url":"#b-be00d7336e89b03df1575dfb","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-380df9d9ebc8c9d96a34484f","type":"p","scope":"document","url":"https://codingai-lec03.pages.dev/course/notebook-blocks/f0aeadb1c331456f96ceb967cfb16a4c.html#b-380df9d9ebc8c9d96a34484f","content":"caption历史、VQA、CLIP候选、关系控制、GLIP/LLaVA与生成桥梁","sha256":"41c70c2c07b82678ff0a60f2d009b6afa750c5c691d109e754a69abf23868b50","chapter":{"id":"course","title":"CLIP 使用已配对的图像与文字，同时训练两种编码器"},"unit":{"id":"notebook-blocks/f0aeadb1c331456f96ceb967cfb16a4c","title":"CLIP 使用已配对的图像与文字，同时训练两种编码器"},"page":{"id":"reading-opening","title":"CLIP 使用已配对的图像与文字，同时训练两种编码器","number":1},"sectionTitle":"CLIP 使用已配对的图像与文字，同时训练两种编码器","currentQuestion":"CLIP 使用已配对的图像与文字，同时训练两种编码器","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-6bdd0d448452375de65d5137","nextBlock":"b-aa57811c0f0db1abb017137e","prerequisites":[{"id":"b-b5ae6e130c5f5f2113f85351","url":"#b-b5ae6e130c5f5f2113f85351","role":"section-title"},{"id":"b-be00d7336e89b03df1575dfb","url":"#b-be00d7336e89b03df1575dfb","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-aa57811c0f0db1abb017137e","type":"p","scope":"document","url":"https://codingai-lec03.pages.dev/course/notebook-blocks/f0aeadb1c331456f96ceb967cfb16a4c.html#b-aa57811c0f0db1abb017137e","content":"CLIP 使用已配对的图像与文字，同时训练两种编码器。训练后，可以把“a photo of a cup”等候选描述编码成文字向量，与图像向量比较，做分类或在已有集合中检索。[CLIP](https://proceedings.mlr.press/v139/radford21a.html)","sha256":"7ca98a60191efe95460cc789edd63b4dcd46b8e818d377021613b48bd73a2582","chapter":{"id":"course","title":"CLIP 使用已配对的图像与文字，同时训练两种编码器"},"unit":{"id":"notebook-blocks/f0aeadb1c331456f96ceb967cfb16a4c","title":"CLIP 使用已配对的图像与文字，同时训练两种编码器"},"page":{"id":"reading-opening","title":"CLIP 使用已配对的图像与文字，同时训练两种编码器","number":1},"sectionTitle":"CLIP 使用已配对的图像与文字，同时训练两种编码器","currentQuestion":"CLIP 使用已配对的图像与文字，同时训练两种编码器","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-380df9d9ebc8c9d96a34484f","nextBlock":"b-12238ea6a86b904efd927631","prerequisites":[{"id":"b-b5ae6e130c5f5f2113f85351","url":"#b-b5ae6e130c5f5f2113f85351","role":"section-title"},{"id":"b-be00d7336e89b03df1575dfb","url":"#b-be00d7336e89b03df1575dfb","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-12238ea6a86b904efd927631","type":"p","scope":"document","url":"https://codingai-lec03.pages.dev/course/notebook-blocks/f0aeadb1c331456f96ceb967cfb16a4c.html#b-12238ea6a86b904efd927631","content":"这项能力来自配对学习：图像向量与文字向量本来并不天然处在可比较的坐标系。图文配对包含人的文字和数据选择，是外部学习信号；不使用固定类别标签不等于完全没有监督。","sha256":"8cbcdacec033c71a8883d97e2d9160a22887dde95646104c6a75b045185b1614","chapter":{"id":"course","title":"CLIP 使用已配对的图像与文字，同时训练两种编码器"},"unit":{"id":"notebook-blocks/f0aeadb1c331456f96ceb967cfb16a4c","title":"CLIP 使用已配对的图像与文字，同时训练两种编码器"},"page":{"id":"reading-opening","title":"CLIP 使用已配对的图像与文字，同时训练两种编码器","number":1},"sectionTitle":"CLIP 使用已配对的图像与文字，同时训练两种编码器","currentQuestion":"CLIP 使用已配对的图像与文字，同时训练两种编码器","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-aa57811c0f0db1abb017137e","nextBlock":"b-e10dc3e574540aae596d8b26","prerequisites":[{"id":"b-b5ae6e130c5f5f2113f85351","url":"#b-b5ae6e130c5f5f2113f85351","role":"section-title"},{"id":"b-be00d7336e89b03df1575dfb","url":"#b-be00d7336e89b03df1575dfb","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-e10dc3e574540aae596d8b26","type":"p","scope":"document","url":"https://codingai-lec03.pages.dev/course/notebook-blocks/f0aeadb1c331456f96ceb967cfb16a4c.html#b-e10dc3e574540aae596d8b26","content":"设红杯、蓝杯、书三张图分别与三段描述配对。把归一化表示记为 uᵢ 和 vⱼ，组成 `Sᵢⱼ=uᵢᵀvⱼ`。每一行回答“这张图配哪段文字”；每一列回答“这段文字配哪张图”。正确配对位于对角线，两种方向的分类损失一起参与训练。","sha256":"a22143a5d7eb7b02fb2f6b604dc5b90425bbfb2492a3e768276483cb768bb024","chapter":{"id":"course","title":"CLIP 使用已配对的图像与文字，同时训练两种编码器"},"unit":{"id":"notebook-blocks/f0aeadb1c331456f96ceb967cfb16a4c","title":"CLIP 使用已配对的图像与文字，同时训练两种编码器"},"page":{"id":"reading-opening","title":"CLIP 使用已配对的图像与文字，同时训练两种编码器","number":1},"sectionTitle":"CLIP 使用已配对的图像与文字，同时训练两种编码器","currentQuestion":"CLIP 使用已配对的图像与文字，同时训练两种编码器","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-12238ea6a86b904efd927631","nextBlock":null,"prerequisites":[{"id":"b-b5ae6e130c5f5f2113f85351","url":"#b-b5ae6e130c5f5f2113f85351","role":"section-title"},{"id":"b-be00d7336e89b03df1575dfb","url":"#b-be00d7336e89b03df1575dfb","role":"problem-context"}],"relatedBlocks":[]}]}