{"version":2,"relationshipURLBase":"https://codingai-lec03.pages.dev/course/blocks/clip-pairing-objective.html","relationshipResolution":"Resolve relative relationship URLs against relationshipURLBase; complete records and absolute URLs are in blocks.","title":"CLIP：文字怎样从标签表变成学习信号","url":"https://codingai-lec03.pages.dev/course/blocks/clip-pairing-objective.html","markdown":"https://codingai-lec03.pages.dev/course/blocks/clip-pairing-objective.md","pages":[{"id":"reading-opening","title":"CLIP：文字怎样从标签表变成学习信号","unit":"已知图文配对怎样变成可用的文字候选接口？","unitId":"clip-pairing-to-candidates","chapter":"III","number":1}],"aliases":[{"oldId":"b-a47bed85c88f90780f52117d","targetId":"b-a47bed85c88f90780f52117d","page":"reading-opening","operation":"move","beforeSHA256":"b9bb5a334110c030d452874b578d5905680730e16987f1d38076d8ceb8d12429","afterSHA256":"b9bb5a334110c030d452874b578d5905680730e16987f1d38076d8ceb8d12429","reason":"Same source body in authored semantic page"},{"oldId":"b-46aca8528edb0f03c2619a19","targetId":"b-46aca8528edb0f03c2619a19","page":"reading-opening","operation":"move","beforeSHA256":"0d653233ba094cef5ff2f1272d447699b1f92f65dbfb71291aa002592f5e7e70","afterSHA256":"0d653233ba094cef5ff2f1272d447699b1f92f65dbfb71291aa002592f5e7e70","reason":"Same source body in authored semantic page"},{"oldId":"b-3dae5b1e6a7c855ca6474bd6","targetId":"b-3dae5b1e6a7c855ca6474bd6","page":"reading-opening","operation":"move","beforeSHA256":"b9680e25d4e832aad1b22242c17efb2bca353c25fa2991448a62994107fab30e","afterSHA256":"b9680e25d4e832aad1b22242c17efb2bca353c25fa2991448a62994107fab30e","reason":"Same source body in authored semantic page"},{"oldId":"b-fb3d470dc0320cbde6ed29ad","targetId":"b-fb3d470dc0320cbde6ed29ad","page":"reading-opening","operation":"move","beforeSHA256":"740e3b6ade773aea1d7a912a98f2a7c09f126fc75b8b574ca9d886cfe6c40342","afterSHA256":"740e3b6ade773aea1d7a912a98f2a7c09f126fc75b8b574ca9d886cfe6c40342","reason":"Same source body in authored semantic page"},{"oldId":"b-9c34a030426cd5f02bd2a94d","targetId":"b-9c34a030426cd5f02bd2a94d","page":"reading-opening","operation":"move","beforeSHA256":"bf1d8692ca1aa8689a2052607dbacad6413518129e70c31345ccf4c368ef2502","afterSHA256":"bf1d8692ca1aa8689a2052607dbacad6413518129e70c31345ccf4c368ef2502","reason":"Same source body in authored semantic page"},{"oldId":"b-90ba74ff5ce8f07c93d52f2f","targetId":"b-90ba74ff5ce8f07c93d52f2f","page":"reading-opening","operation":"move","beforeSHA256":"3efe21234482d01574f0859dc902c0d8f0ab78268142b8dfaf90b6d5dfca8633","afterSHA256":"3efe21234482d01574f0859dc902c0d8f0ab78268142b8dfaf90b6d5dfca8633","reason":"Same source body in authored semantic page"},{"oldId":"b-45e07e6bf54ae22cb22f43a7","targetId":"b-45e07e6bf54ae22cb22f43a7","page":"reading-opening","operation":"rewrite","beforeSHA256":"e5c461feeff3d0e7cb72fb5fd506e4b72acacaf90987531d3b0041cf1091c951","afterSHA256":"30397dedc39e5afabb8f77b423da8dd6c6f6445b39790c72fdc4dbb7a1700ffd","reason":"Same source identity with figure semantics or reading context enriched"},{"oldId":"b-2491d99ffcf9192bfdbd13b3","targetId":"b-2491d99ffcf9192bfdbd13b3","page":"reading-opening","operation":"move","beforeSHA256":"2b9ccef7e1220e2250cceff1abd8bdd736b42f8da5a223f26b029f44b962e807","afterSHA256":"2b9ccef7e1220e2250cceff1abd8bdd736b42f8da5a223f26b029f44b962e807","reason":"Same source body in authored semantic page"}],"blocks":[{"id":"b-a47bed85c88f90780f52117d","type":"h1","scope":"clip-pairing-objective","url":"https://codingai-lec03.pages.dev/course/blocks/clip-pairing-objective.html#b-a47bed85c88f90780f52117d","content":"# CLIP：文字怎样从标签表变成学习信号","sha256":"b9bb5a334110c030d452874b578d5905680730e16987f1d38076d8ceb8d12429","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"clip-pairing-to-candidates","title":"已知图文配对怎样变成可用的文字候选接口？"},"page":{"id":"reading-opening","title":"CLIP：文字怎样从标签表变成学习信号","number":1},"sectionTitle":"CLIP：文字怎样从标签表变成学习信号","currentQuestion":"CLIP：文字怎样从标签表变成学习信号","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":null,"nextBlock":"b-46aca8528edb0f03c2619a19","prerequisites":[{"id":"b-46aca8528edb0f03c2619a19","url":"#b-46aca8528edb0f03c2619a19","role":"problem-context"}],"relatedBlocks":[{"id":"b-45e07e6bf54ae22cb22f43a7","url":"#b-45e07e6bf54ae22cb22f43a7","relation":"figure-and-explanation"}]},{"id":"b-46aca8528edb0f03c2619a19","type":"p","scope":"clip-pairing-objective","url":"https://codingai-lec03.pages.dev/course/blocks/clip-pairing-objective.html#b-46aca8528edb0f03c2619a19","content":"CLIP 选择配对识别作为学习任务。两套编码器各自处理图片和文字，配对资料指出哪些对应应更容易被选出。与图像自监督对比的联系是训练关系的来源不同；文字监督拓展语义供给，下一页才用矩阵落实损失。","sha256":"0d653233ba094cef5ff2f1272d447699b1f92f65dbfb71291aa002592f5e7e70","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"clip-pairing-to-candidates","title":"已知图文配对怎样变成可用的文字候选接口？"},"page":{"id":"reading-opening","title":"CLIP：文字怎样从标签表变成学习信号","number":1},"sectionTitle":"CLIP：文字怎样从标签表变成学习信号","currentQuestion":"CLIP：文字怎样从标签表变成学习信号","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-a47bed85c88f90780f52117d","nextBlock":"b-3dae5b1e6a7c855ca6474bd6","prerequisites":[{"id":"b-a47bed85c88f90780f52117d","url":"#b-a47bed85c88f90780f52117d","role":"section-title"}],"relatedBlocks":[{"id":"b-45e07e6bf54ae22cb22f43a7","url":"#b-45e07e6bf54ae22cb22f43a7","relation":"figure-and-explanation"}]},{"id":"b-3dae5b1e6a7c855ca6474bd6","type":"p","scope":"clip-pairing-objective","url":"https://codingai-lec03.pages.dev/course/blocks/clip-pairing-objective.html#b-3dae5b1e6a7c855ca6474bd6","content":"监督分类先规定一个类别表，再让模型预测表中的索引。图文配对可以提供更丰富的语义：文字可能同时包含对象、属性、动作和场景。CLIP 选择一个可扩展的任务——在一批图文中识别原有配对——把这些描述用于训练可比较的表示。","sha256":"b9680e25d4e832aad1b22242c17efb2bca353c25fa2991448a62994107fab30e","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"clip-pairing-to-candidates","title":"已知图文配对怎样变成可用的文字候选接口？"},"page":{"id":"reading-opening","title":"CLIP：文字怎样从标签表变成学习信号","number":1},"sectionTitle":"CLIP：文字怎样从标签表变成学习信号","currentQuestion":"CLIP：文字怎样从标签表变成学习信号","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-46aca8528edb0f03c2619a19","nextBlock":"b-fb3d470dc0320cbde6ed29ad","prerequisites":[{"id":"b-a47bed85c88f90780f52117d","url":"#b-a47bed85c88f90780f52117d","role":"section-title"},{"id":"b-46aca8528edb0f03c2619a19","url":"#b-46aca8528edb0f03c2619a19","role":"problem-context"}],"relatedBlocks":[{"id":"b-45e07e6bf54ae22cb22f43a7","url":"#b-45e07e6bf54ae22cb22f43a7","relation":"figure-and-explanation"}]},{"id":"b-fb3d470dc0320cbde6ed29ad","type":"p","scope":"clip-pairing-objective","url":"https://codingai-lec03.pages.dev/course/blocks/clip-pairing-objective.html#b-fb3d470dc0320cbde6ed29ad","content":"设图像编码器为 $f$，文字编码器为 $g$，归一化后得到 $v_i=f(I_i)/\\|f(I_i)\\|$、$u_j=g(T_j)/\\|g(T_j)\\|$。两个编码器分别处理各自输入，然后用点积 $S_{ij}=v_i^Tu_j$ 比较。没有先让图像生成一句话，再把两句话相互比较。","sha256":"740e3b6ade773aea1d7a912a98f2a7c09f126fc75b8b574ca9d886cfe6c40342","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"clip-pairing-to-candidates","title":"已知图文配对怎样变成可用的文字候选接口？"},"page":{"id":"reading-opening","title":"CLIP：文字怎样从标签表变成学习信号","number":1},"sectionTitle":"CLIP：文字怎样从标签表变成学习信号","currentQuestion":"CLIP：文字怎样从标签表变成学习信号","sourceSlides":[],"knowledgeBlock":null,"formulas":[{"latex":"f","definitionContext":"CLIP：文字怎样从标签表变成学习信号"},{"latex":"g","definitionContext":"CLIP：文字怎样从标签表变成学习信号"},{"latex":"v_i=f(I_i)/\\|f(I_i)\\|","definitionContext":"CLIP：文字怎样从标签表变成学习信号"},{"latex":"u_j=g(T_j)/\\|g(T_j)\\|","definitionContext":"CLIP：文字怎样从标签表变成学习信号"},{"latex":"S_{ij}=v_i^Tu_j","definitionContext":"CLIP：文字怎样从标签表变成学习信号"}],"previousBlock":"b-3dae5b1e6a7c855ca6474bd6","nextBlock":"b-9c34a030426cd5f02bd2a94d","prerequisites":[{"id":"b-a47bed85c88f90780f52117d","url":"#b-a47bed85c88f90780f52117d","role":"section-title"},{"id":"b-46aca8528edb0f03c2619a19","url":"#b-46aca8528edb0f03c2619a19","role":"problem-context"}],"relatedBlocks":[{"id":"b-45e07e6bf54ae22cb22f43a7","url":"#b-45e07e6bf54ae22cb22f43a7","relation":"figure-and-explanation"}]},{"id":"b-9c34a030426cd5f02bd2a94d","type":"p","scope":"clip-pairing-objective","url":"https://codingai-lec03.pages.dev/course/blocks/clip-pairing-objective.html#b-9c34a030426cd5f02bd2a94d","content":"与第一部分的图像对比学习相连：两张增强视图的正关系来自同一原图；CLIP 的正关系来自资料中的图文配对。文字已经包含人的选择和表达，因此“没有固定类别标签”不等于“没有人类监督”。","sha256":"bf1d8692ca1aa8689a2052607dbacad6413518129e70c31345ccf4c368ef2502","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"clip-pairing-to-candidates","title":"已知图文配对怎样变成可用的文字候选接口？"},"page":{"id":"reading-opening","title":"CLIP：文字怎样从标签表变成学习信号","number":1},"sectionTitle":"CLIP：文字怎样从标签表变成学习信号","currentQuestion":"CLIP：文字怎样从标签表变成学习信号","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-fb3d470dc0320cbde6ed29ad","nextBlock":"b-90ba74ff5ce8f07c93d52f2f","prerequisites":[{"id":"b-a47bed85c88f90780f52117d","url":"#b-a47bed85c88f90780f52117d","role":"section-title"},{"id":"b-46aca8528edb0f03c2619a19","url":"#b-46aca8528edb0f03c2619a19","role":"problem-context"}],"relatedBlocks":[{"id":"b-45e07e6bf54ae22cb22f43a7","url":"#b-45e07e6bf54ae22cb22f43a7","relation":"figure-and-explanation"}]},{"id":"b-90ba74ff5ce8f07c93d52f2f","type":"p","scope":"clip-pairing-objective","url":"https://codingai-lec03.pages.dev/course/blocks/clip-pairing-objective.html#b-90ba74ff5ce8f07c93d52f2f","content":"训练完保留两套编码器，新图像与新文字可以进入同一比较接口。使用者能更换候选描述或建立检索库，这是方法的正面能力。它还没有提供下一词预测分布或框回归输出；若新任务需要这些输出，要另行定义模块、目标和训练。","sha256":"3efe21234482d01574f0859dc902c0d8f0ab78268142b8dfaf90b6d5dfca8633","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"clip-pairing-to-candidates","title":"已知图文配对怎样变成可用的文字候选接口？"},"page":{"id":"reading-opening","title":"CLIP：文字怎样从标签表变成学习信号","number":1},"sectionTitle":"CLIP：文字怎样从标签表变成学习信号","currentQuestion":"CLIP：文字怎样从标签表变成学习信号","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-9c34a030426cd5f02bd2a94d","nextBlock":"b-45e07e6bf54ae22cb22f43a7","prerequisites":[{"id":"b-a47bed85c88f90780f52117d","url":"#b-a47bed85c88f90780f52117d","role":"section-title"},{"id":"b-46aca8528edb0f03c2619a19","url":"#b-46aca8528edb0f03c2619a19","role":"problem-context"}],"relatedBlocks":[{"id":"b-45e07e6bf54ae22cb22f43a7","url":"#b-45e07e6bf54ae22cb22f43a7","relation":"figure-and-explanation"}]},{"id":"b-45e07e6bf54ae22cb22f43a7","type":"figure","scope":"clip-pairing-objective","url":"https://codingai-lec03.pages.dev/course/blocks/clip-pairing-objective.html#b-45e07e6bf54ae22cb22f43a7","content":"图内标签与数值：图像编码器；“红衣人拿着篮球，；旁边站着蓝衣人。”；文字编码器；文字 → 向量；比较；训练答案：这一图像与这一段文字配成一对；向量条纹仅标示接口，未作定量含义；CLIP 学习图文匹配，不在此分支生成完整句子。\n\n配对关系先于模型给定；两套编码器学会把配对送入可比较空间。 查看对应图解状态\n\n- 图像编码器\n- “红衣人拿着篮球，\n- 旁边站着蓝衣人。”\n- 文字编码器\n- 文字 → 向量\n- 比较\n- 训练答案：这一图像与这一段文字配成一对\n- 向量条纹仅标示接口，未作定量含义；CLIP 学习图文匹配，不在此分支生成完整句子。配对关系先于模型给定；两套编码器学会把配对送入可比较空间。 [查看对应图解状态](/#/l3-r2-p56/0)[查看完整图解](/#/l3-r2-p56/0)","sha256":"30397dedc39e5afabb8f77b423da8dd6c6f6445b39790c72fdc4dbb7a1700ffd","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"clip-pairing-to-candidates","title":"已知图文配对怎样变成可用的文字候选接口？"},"page":{"id":"reading-opening","title":"CLIP：文字怎样从标签表变成学习信号","number":1},"sectionTitle":"CLIP：文字怎样从标签表变成学习信号","currentQuestion":"CLIP：文字怎样从标签表变成学习信号","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"figure":{"caption":"配对关系先于模型给定；两套编码器学会把配对送入可比较空间。 查看对应图解状态","labels":["图像编码器","“红衣人拿着篮球，","旁边站着蓝衣人。”","文字编码器","文字 → 向量","比较","训练答案：这一图像与这一段文字配成一对","向量条纹仅标示接口，未作定量含义；CLIP 学习图文匹配，不在此分支生成完整句子。"],"description":"配对关系先于模型给定；两套编码器学会把配对送入可比较空间。 查看对应图解状态图像编码器“红衣人拿着篮球，旁边站着蓝衣人。”文字编码器文字 → 向量比较训练答案：这一图像与这一段文字配成一对向量条纹仅标示接口，未作定量含义；CLIP 学习图文匹配，不在此分支生成完整句子。","evidenceIdentity":"配对关系先于模型给定；两套编码器学会把配对送入可比较空间。 查看对应图解状态","images":[{"src":"/assets/display/512610a019c560ff0384e86a.webp","alt":""}]},"previousBlock":"b-90ba74ff5ce8f07c93d52f2f","nextBlock":"b-2491d99ffcf9192bfdbd13b3","prerequisites":[{"id":"b-a47bed85c88f90780f52117d","url":"#b-a47bed85c88f90780f52117d","role":"section-title"},{"id":"b-46aca8528edb0f03c2619a19","url":"#b-46aca8528edb0f03c2619a19","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-2491d99ffcf9192bfdbd13b3","type":"h2","scope":"clip-pairing-objective","url":"https://codingai-lec03.pages.dev/course/blocks/clip-pairing-objective.html#b-2491d99ffcf9192bfdbd13b3","content":"## 继续阅读与实验","sha256":"2b9ccef7e1220e2250cceff1abd8bdd736b42f8da5a223f26b029f44b962e807","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"clip-pairing-to-candidates","title":"已知图文配对怎样变成可用的文字候选接口？"},"page":{"id":"reading-opening","title":"CLIP：文字怎样从标签表变成学习信号","number":1},"sectionTitle":"CLIP：文字怎样从标签表变成学习信号","currentQuestion":"CLIP：文字怎样从标签表变成学习信号","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-45e07e6bf54ae22cb22f43a7","nextBlock":null,"prerequisites":[{"id":"b-a47bed85c88f90780f52117d","url":"#b-a47bed85c88f90780f52117d","role":"section-title"},{"id":"b-46aca8528edb0f03c2619a19","url":"#b-46aca8528edb0f03c2619a19","role":"problem-context"}],"relatedBlocks":[{"id":"b-45e07e6bf54ae22cb22f43a7","url":"#b-45e07e6bf54ae22cb22f43a7","relation":"figure-and-explanation"}]}]}