{"version":2,"relationshipURLBase":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p52.html","relationshipResolution":"Resolve relative relationship URLs against relationshipURLBase; complete records and absolute URLs are in blocks.","title":"认识一个物体，为什么未必会把它说出来？","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p52.html","markdown":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p52.md","pages":[{"id":"caption-composition-gap","title":"找出物体与词语之间尚未学会的使用关系","unit":"图像怎样约束一句新生成的描述？","unitId":"caption-visual-conditioning","chapter":"III","number":1}],"aliases":[{"oldId":"explanation","targetId":"caption-composition-gap","operation":"split-explanation-range"},{"oldId":"b-5faa1a4858f1d377beed3668","targetId":"b-5faa1a4858f1d377beed3668","page":"caption-composition-gap","operation":"move","beforeSHA256":"d71cf15e707c8a3674b4c90a599efee783cb84da5b8c8236a1d01c3add7b4958","afterSHA256":"d71cf15e707c8a3674b4c90a599efee783cb84da5b8c8236a1d01c3add7b4958","reason":"Same source body in authored semantic page"},{"oldId":"b-19b50ddabd3f9e1980449010","targetId":"b-19b50ddabd3f9e1980449010","page":"caption-composition-gap","operation":"move","beforeSHA256":"5e5e7645e509f59152e76eda1e526a8d8b7351dba0e911b09904adcc5e3cc50b","afterSHA256":"5e5e7645e509f59152e76eda1e526a8d8b7351dba0e911b09904adcc5e3cc50b","reason":"Same source body in authored semantic page"},{"oldId":"b-57450bcdb34f112e54acdb2f","targetId":"b-57450bcdb34f112e54acdb2f","page":"caption-composition-gap","operation":"move","beforeSHA256":"7f63704d805edecaecdf1d96b13560ed33ec21e13105d30a9da9fe6493863d3d","afterSHA256":"7f63704d805edecaecdf1d96b13560ed33ec21e13105d30a9da9fe6493863d3d","reason":"Same source body in authored semantic page"},{"oldId":"b-d927cfb52c3f924a500f5628","targetId":"b-d927cfb52c3f924a500f5628","page":"caption-composition-gap","operation":"rewrite","beforeSHA256":"bedf8cc8fd637bbe0a2540d263f7b1b9fc655a9cf7118d267012ed568800dacb","afterSHA256":"26cb6a9eabc1d4e3852bf26bbb3b368468205c5ac04354fb468a75196138b991","reason":"Same source identity with figure semantics or reading context enriched"},{"oldId":"b-41722f663a5b34afb28833bf","targetId":"b-41722f663a5b34afb28833bf","page":"caption-composition-gap","operation":"move","beforeSHA256":"786ce10a6b32a492259a3dccd0a54e61cd41d28a34382fb7b5386c5f03b7ad69","afterSHA256":"786ce10a6b32a492259a3dccd0a54e61cd41d28a34382fb7b5386c5f03b7ad69","reason":"Same source body in authored semantic page"},{"oldId":"b-cfdfc4336b0d8f6fddc6672f","targetId":"b-cfdfc4336b0d8f6fddc6672f","page":"caption-composition-gap","operation":"move","beforeSHA256":"428ea566a1bb1ccb4c1ff47dca53ec0864f38b3da42a71787418d7b7593e3a3c","afterSHA256":"428ea566a1bb1ccb4c1ff47dca53ec0864f38b3da42a71787418d7b7593e3a3c","reason":"Same source body in authored semantic page"},{"oldId":"b-a1ed0bc94e3cf20cea2f656a","targetId":"b-a1ed0bc94e3cf20cea2f656a","page":"caption-composition-gap","operation":"move","beforeSHA256":"9799de1de93eb4eba0831374981a597cde29ca27b5d09834d06daa04e1d87ed7","afterSHA256":"9799de1de93eb4eba0831374981a597cde29ca27b5d09834d06daa04e1d87ed7","reason":"Same source body in authored semantic page"},{"oldId":"b-7a3f26390c0510d0285059cb","targetId":"b-7a3f26390c0510d0285059cb","page":"caption-composition-gap","operation":"move","beforeSHA256":"2eb20432e38bdbf76b853ac229b1a484296a9f4e03bfc13f04eec1f6b166b5c0","afterSHA256":"2eb20432e38bdbf76b853ac229b1a484296a9f4e03bfc13f04eec1f6b166b5c0","reason":"Same source body in authored semantic page"},{"oldId":"b-72d1b0f41080c867bfe36520","targetId":"b-72d1b0f41080c867bfe36520","page":"caption-composition-gap","operation":"move","beforeSHA256":"3cd30c1374864eca90570c520a8358763c29f4142490f2d6090db4bfe1fb213f","afterSHA256":"3cd30c1374864eca90570c520a8358763c29f4142490f2d6090db4bfe1fb213f","reason":"Same source body in authored semantic page"},{"oldId":"b-97c26f9371af40c489cea434","targetId":"b-97c26f9371af40c489cea434","page":"caption-composition-gap","operation":"move","beforeSHA256":"75ba8dac172d55602c23a0bebea1b1f45763290930c94cc0256fc6a64e90714c","afterSHA256":"75ba8dac172d55602c23a0bebea1b1f45763290930c94cc0256fc6a64e90714c","reason":"Same source body in authored semantic page"},{"oldId":"b-3ca345e56ff79e20723b5831","targetId":"b-3ca345e56ff79e20723b5831","page":"caption-composition-gap","operation":"move","beforeSHA256":"d157be2058eab4edd7ef1474fdf09f94e9d9fb6bf033fbdd8aafc6d22e6713b8","afterSHA256":"d157be2058eab4edd7ef1474fdf09f94e9d9fb6bf033fbdd8aafc6d22e6713b8","reason":"Same source body in authored semantic page"},{"oldId":"b-2aee9349597bc5b1add6b6d8","targetId":"b-2aee9349597bc5b1add6b6d8","page":"caption-composition-gap","operation":"move","beforeSHA256":"f16be0424762a44344338696f06ef266c4571d15367ae2cbbd28fac98ac7e9cb","afterSHA256":"f16be0424762a44344338696f06ef266c4571d15367ae2cbbd28fac98ac7e9cb","reason":"Same source body in authored semantic page"},{"oldId":"b-df6d6c4e8c827dcf011f2307","targetId":"b-df6d6c4e8c827dcf011f2307","page":"caption-composition-gap","operation":"move","beforeSHA256":"b25685f04cd1b885f3a5999e6a912460c7424021cdc5b2190584242086a82681","afterSHA256":"b25685f04cd1b885f3a5999e6a912460c7424021cdc5b2190584242086a82681","reason":"Same source body in authored semantic page"},{"oldId":"b-71599dc8e1a484c3aa0873cd","targetId":"b-71599dc8e1a484c3aa0873cd","page":"caption-composition-gap","operation":"move","beforeSHA256":"833db77ab8059985a97f56f7da522d6c3a3035acb7ae45bbc6c1e3d6b4eb516b","afterSHA256":"833db77ab8059985a97f56f7da522d6c3a3035acb7ae45bbc6c1e3d6b4eb516b","reason":"Same source body in authored semantic page"},{"oldId":"b-00a3b42ff2acd44cbb156686","targetId":"b-00a3b42ff2acd44cbb156686","page":"caption-composition-gap","operation":"move","beforeSHA256":"252e0d78c2f8254c4ea6c68570f356138a76148e11618a452c7c8058143cfc35","afterSHA256":"252e0d78c2f8254c4ea6c68570f356138a76148e11618a452c7c8058143cfc35","reason":"Same source body in authored semantic page"},{"oldId":"b-b18ea5ef57c414f26698e65f","targetId":"b-b18ea5ef57c414f26698e65f","page":"caption-composition-gap","operation":"move","beforeSHA256":"cf44fcc62782fb4f6bdc42c16c5187d57556a03cb7ce6065e7bd9c36d8e7d733","afterSHA256":"cf44fcc62782fb4f6bdc42c16c5187d57556a03cb7ce6065e7bd9c36d8e7d733","reason":"Same source body in authored semantic page"},{"oldId":"b-f73de35e43955b7758df2036","targetId":"b-f73de35e43955b7758df2036","page":"caption-composition-gap","operation":"move","beforeSHA256":"3f12361d24fb375c4fa9212b387c4035be03d494433b2fe50e44ceac7c08a7e5","afterSHA256":"3f12361d24fb375c4fa9212b387c4035be03d494433b2fe50e44ceac7c08a7e5","reason":"Same source body in authored semantic page"},{"oldId":"b-7f20f3d9480171df79c8f19c","targetId":"b-7f20f3d9480171df79c8f19c","page":"caption-composition-gap","operation":"move","beforeSHA256":"da8fefb58c7f9e5e864411adbd618eef5d3c41c618a32ba1af2e8d7d7cf2b4fc","afterSHA256":"da8fefb58c7f9e5e864411adbd618eef5d3c41c618a32ba1af2e8d7d7cf2b4fc","reason":"Same source body in authored semantic page"},{"oldId":"caption-composition-gap-title","targetId":"b-61efc94fd1eb67bfac6ff4f0","operation":"retain-source-identity"}],"blocks":[{"id":"b-61efc94fd1eb67bfac6ff4f0","type":"h2","scope":"caption-composition-gap","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p52.html#b-61efc94fd1eb67bfac6ff4f0","content":"## 找出物体与词语之间尚未学会的使用关系","sha256":"4f7ccdf55455ae87558421f834ce48eeb0146569ffa1c30a21d6c0011ab3b7e7","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"caption-visual-conditioning","title":"图像怎样约束一句新生成的描述？"},"page":{"id":"caption-composition-gap","title":"找出物体与词语之间尚未学会的使用关系","number":1},"sectionTitle":"找出物体与词语之间尚未学会的使用关系","currentQuestion":"从训练目标到自由生成，怎样检查模型实际使用了图像？","sourceSlides":["l3-r2-p52"],"knowledgeBlock":null,"formulas":[],"previousBlock":null,"nextBlock":"b-5faa1a4858f1d377beed3668","prerequisites":[{"id":"b-57450bcdb34f112e54acdb2f","url":"#b-57450bcdb34f112e54acdb2f","role":"problem-context"}],"relatedBlocks":[{"id":"b-d927cfb52c3f924a500f5628","url":"#b-d927cfb52c3f924a500f5628","relation":"figure-and-explanation"}]},{"id":"b-5faa1a4858f1d377beed3668","type":"h2","scope":"explanation","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p52.html#b-5faa1a4858f1d377beed3668","content":"## 解释、算例与证据","sha256":"d71cf15e707c8a3674b4c90a599efee783cb84da5b8c8236a1d01c3add7b4958","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"caption-visual-conditioning","title":"图像怎样约束一句新生成的描述？"},"page":{"id":"caption-composition-gap","title":"找出物体与词语之间尚未学会的使用关系","number":1},"sectionTitle":"找出物体与词语之间尚未学会的使用关系","currentQuestion":"从训练目标到自由生成，怎样检查模型实际使用了图像？","sourceSlides":["l3-r2-p52"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-61efc94fd1eb67bfac6ff4f0","nextBlock":"b-19b50ddabd3f9e1980449010","prerequisites":[{"id":"b-61efc94fd1eb67bfac6ff4f0","url":"#b-61efc94fd1eb67bfac6ff4f0","role":"section-title"},{"id":"b-57450bcdb34f112e54acdb2f","url":"#b-57450bcdb34f112e54acdb2f","role":"problem-context"}],"relatedBlocks":[{"id":"b-d927cfb52c3f924a500f5628","url":"#b-d927cfb52c3f924a500f5628","relation":"figure-and-explanation"}]},{"id":"b-19b50ddabd3f9e1980449010","type":"h1","scope":"document","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p52.html#b-19b50ddabd3f9e1980449010","content":"# P52 · 认识一个物体，为什么未必会把它说出来？","sha256":"5e5e7645e509f59152e76eda1e526a8d8b7351dba0e911b09904adcc5e3cc50b","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"caption-visual-conditioning","title":"图像怎样约束一句新生成的描述？"},"page":{"id":"caption-composition-gap","title":"找出物体与词语之间尚未学会的使用关系","number":1},"sectionTitle":"找出物体与词语之间尚未学会的使用关系","currentQuestion":"从训练目标到自由生成，怎样检查模型实际使用了图像？","sourceSlides":["l3-r2-p52"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-5faa1a4858f1d377beed3668","nextBlock":"b-57450bcdb34f112e54acdb2f","prerequisites":[{"id":"b-5faa1a4858f1d377beed3668","url":"#b-5faa1a4858f1d377beed3668","role":"section-title"},{"id":"b-57450bcdb34f112e54acdb2f","url":"#b-57450bcdb34f112e54acdb2f","role":"problem-context"}],"relatedBlocks":[{"id":"b-d927cfb52c3f924a500f5628","url":"#b-d927cfb52c3f924a500f5628","relation":"figure-and-explanation"}]},{"id":"b-57450bcdb34f112e54acdb2f","type":"p","scope":"document","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p52.html#b-57450bcdb34f112e54acdb2f","content":"本页的缺口发生在“知识如何共同使用”：视觉认识某物体、文字存在对应词，并不自动提供描述时的连接规则。DCC 的新颖对象限定在配对描述资料，而非所有数据。课程零件 K 是解释用例，理解这一限定后再回看 LLaVA 的接口学习。","sha256":"7f63704d805edecaecdf1d96b13560ed33ec21e13105d30a9da9fe6493863d3d","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"caption-visual-conditioning","title":"图像怎样约束一句新生成的描述？"},"page":{"id":"caption-composition-gap","title":"找出物体与词语之间尚未学会的使用关系","number":1},"sectionTitle":"找出物体与词语之间尚未学会的使用关系","currentQuestion":"从训练目标到自由生成，怎样检查模型实际使用了图像？","sourceSlides":["l3-r2-p52"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-19b50ddabd3f9e1980449010","nextBlock":"b-d927cfb52c3f924a500f5628","prerequisites":[{"id":"b-19b50ddabd3f9e1980449010","url":"#b-19b50ddabd3f9e1980449010","role":"section-title"}],"relatedBlocks":[{"id":"b-d927cfb52c3f924a500f5628","url":"#b-d927cfb52c3f924a500f5628","relation":"figure-and-explanation"}]},{"id":"b-d927cfb52c3f924a500f5628","type":"figure","scope":"document","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p52.html#b-d927cfb52c3f924a500f5628","content":"图内标签与数值：课程构造：视觉类别“零件 K”已见，配对描述中未见；类别：零件 K；文字材料；“零件 K 安装在支架上。”；“工人检查了零件 K。”；这个词已出现在文字里；配对描述训练；图像 ↔ 句子；已有其他对象的配对；缺少零件 K 的配对；把两边的知识组合成描述；DCC 的问题：在配对描述中未见，仍能否说出？；DCC（CVPR 2016）研究设定的教学转写；零件 K 不是原论文样本或原论文输出。\n\n独立的零件类别图像、包含零件词的文字语料与缺少该类别的配对描述训练集，组成DCC要解决的组合缺口。\n\n- 课程构造：视觉类别“零件 K”已见，配对描述中未见\n- 类别：零件 K\n- 文字材料\n- “零件 K 安装在支架上。”\n- “工人检查了零件 K。”\n- 这个词已出现在文字里\n- 配对描述训练\n- 图像 ↔ 句子\n- 已有其他对象的配对\n- 缺少零件 K 的配对\n- 把两边的知识组合成描述\n- DCC 的问题：在配对描述中未见，仍能否说出？\n- DCC（CVPR 2016）研究设定的教学转写；零件 K 不是原论文样本或原论文输出。独立的零件类别图像、包含零件词的文字语料与缺少该类别的配对描述训练集，组成DCC要解决的组合缺口。 [查看对应图解状态](/#/l3-r2-p52/0)[查看完整图解](/course/figures/l3-r2-p52-0.html)","sha256":"26cb6a9eabc1d4e3852bf26bbb3b368468205c5ac04354fb468a75196138b991","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"caption-visual-conditioning","title":"图像怎样约束一句新生成的描述？"},"page":{"id":"caption-composition-gap","title":"找出物体与词语之间尚未学会的使用关系","number":1},"sectionTitle":"找出物体与词语之间尚未学会的使用关系","currentQuestion":"从训练目标到自由生成，怎样检查模型实际使用了图像？","sourceSlides":["l3-r2-p52"],"knowledgeBlock":null,"formulas":[],"figure":{"caption":"独立的零件类别图像、包含零件词的文字语料与缺少该类别的配对描述训练集，组成DCC要解决的组合缺口。 查看对应图解状态","labels":["课程构造：视觉类别“零件 K”已见，配对描述中未见","类别：零件 K","文字材料","“零件 K 安装在支架上。”","“工人检查了零件 K。”","这个词已出现在文字里","配对描述训练","图像 ↔ 句子","已有其他对象的配对","缺少零件 K 的配对","把两边的知识组合成描述","DCC 的问题：在配对描述中未见，仍能否说出？","DCC（CVPR 2016）研究设定的教学转写；零件 K 不是原论文样本或原论文输出。"],"description":"独立的零件类别图像、包含零件词的文字语料与缺少该类别的配对描述训练集，组成DCC要解决的组合缺口。课程构造：视觉类别“零件 K”已见，配对描述中未见类别：零件 K文字材料“零件 K 安装在支架上。”“工人检查了零件 K。”这个词已出现在文字里配对描述训练图像 ↔ 句子已有其他对象的配对缺少零件 K 的配对把两边的知识组合成描述DCC 的问题：在配对描述中未见，仍能否说出？DCC（CVPR 2016）研究设定的教学转写；零件 K 不是原论文样本或原论文输出。","evidenceIdentity":"独立的零件类别图像、包含零件词的文字语料与缺少该类别的配对描述训练集，组成DCC要解决的组合缺口。 查看对应图解状态","images":[{"src":"/assets/display/8757616f3f92bbccd2d8db9c.webp","alt":""}]},"previousBlock":"b-57450bcdb34f112e54acdb2f","nextBlock":"b-41722f663a5b34afb28833bf","prerequisites":[{"id":"b-19b50ddabd3f9e1980449010","url":"#b-19b50ddabd3f9e1980449010","role":"section-title"},{"id":"b-57450bcdb34f112e54acdb2f","url":"#b-57450bcdb34f112e54acdb2f","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-41722f663a5b34afb28833bf","type":"h2","scope":"narrative","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p52.html#b-41722f663a5b34afb28833bf","content":"## 课堂讲述","sha256":"786ce10a6b32a492259a3dccd0a54e61cd41d28a34382fb7b5386c5f03b7ad69","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"caption-visual-conditioning","title":"图像怎样约束一句新生成的描述？"},"page":{"id":"caption-composition-gap","title":"找出物体与词语之间尚未学会的使用关系","number":1},"sectionTitle":"找出物体与词语之间尚未学会的使用关系","currentQuestion":"从训练目标到自由生成，怎样检查模型实际使用了图像？","sourceSlides":["l3-r2-p52"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-d927cfb52c3f924a500f5628","nextBlock":"b-cfdfc4336b0d8f6fddc6672f","prerequisites":[{"id":"b-19b50ddabd3f9e1980449010","url":"#b-19b50ddabd3f9e1980449010","role":"section-title"},{"id":"b-57450bcdb34f112e54acdb2f","url":"#b-57450bcdb34f112e54acdb2f","role":"problem-context"}],"relatedBlocks":[{"id":"b-d927cfb52c3f924a500f5628","url":"#b-d927cfb52c3f924a500f5628","relation":"figure-and-explanation"}]},{"id":"b-cfdfc4336b0d8f6fddc6672f","type":"p","scope":"narrative","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p52.html#b-cfdfc4336b0d8f6fddc6672f","content":"DCC 问了一个很有意思的问题：视觉端可能已经认识某类物体，语言材料里也存在这个词，但描述系统没有见过它的图文配对，能否把两种知识组合起来？\n这里的新颖不是“世界上从未出现过这个物体”，而是相关组合没有出现在配对描述训练里。研究把独立的视觉知识、文字知识和图文组合分开，才有办法检查连接能力。\n这对我们很重要。两个组件分别掌握了一些知识，不等于接到一起就已经会共同使用。我们后面看到视觉编码器接入语言模型，也要问同样的问题：接口的形状对上了，什么关系还需要数据与训练去建立？","sha256":"428ea566a1bb1ccb4c1ff47dca53ec0864f38b3da42a71787418d7b7593e3a3c","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"caption-visual-conditioning","title":"图像怎样约束一句新生成的描述？"},"page":{"id":"caption-composition-gap","title":"找出物体与词语之间尚未学会的使用关系","number":1},"sectionTitle":"找出物体与词语之间尚未学会的使用关系","currentQuestion":"从训练目标到自由生成，怎样检查模型实际使用了图像？","sourceSlides":["l3-r2-p52"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-41722f663a5b34afb28833bf","nextBlock":"b-a1ed0bc94e3cf20cea2f656a","prerequisites":[{"id":"b-41722f663a5b34afb28833bf","url":"#b-41722f663a5b34afb28833bf","role":"section-title"},{"id":"b-57450bcdb34f112e54acdb2f","url":"#b-57450bcdb34f112e54acdb2f","role":"problem-context"}],"relatedBlocks":[{"id":"b-d927cfb52c3f924a500f5628","url":"#b-d927cfb52c3f924a500f5628","relation":"figure-and-explanation"}]},{"id":"b-a1ed0bc94e3cf20cea2f656a","type":"h3","scope":"caption-compositional-gap","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p52.html#b-a1ed0bc94e3cf20cea2f656a","content":"### DCC：分别认识物体和词语，为什么还不会组合描述","sha256":"9799de1de93eb4eba0831374981a597cde29ca27b5d09834d06daa04e1d87ed7","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"caption-visual-conditioning","title":"图像怎样约束一句新生成的描述？"},"page":{"id":"caption-composition-gap","title":"找出物体与词语之间尚未学会的使用关系","number":1},"sectionTitle":"DCC：分别认识物体和词语，为什么还不会组合描述","currentQuestion":"从训练目标到自由生成，怎样检查模型实际使用了图像？","sourceSlides":["l3-r2-p52"],"knowledgeBlock":"caption-compositional-gap","formulas":[],"previousBlock":"b-cfdfc4336b0d8f6fddc6672f","nextBlock":"b-7a3f26390c0510d0285059cb","prerequisites":[{"id":"b-41722f663a5b34afb28833bf","url":"#b-41722f663a5b34afb28833bf","role":"section-title"},{"id":"b-57450bcdb34f112e54acdb2f","url":"#b-57450bcdb34f112e54acdb2f","role":"problem-context"}],"relatedBlocks":[{"id":"b-d927cfb52c3f924a500f5628","url":"#b-d927cfb52c3f924a500f5628","relation":"figure-and-explanation"},{"id":"b-7a3f26390c0510d0285059cb","url":"#b-7a3f26390c0510d0285059cb","relation":"same-reasoning"},{"id":"b-72d1b0f41080c867bfe36520","url":"#b-72d1b0f41080c867bfe36520","relation":"same-reasoning"},{"id":"b-97c26f9371af40c489cea434","url":"#b-97c26f9371af40c489cea434","relation":"same-reasoning"},{"id":"b-3ca345e56ff79e20723b5831","url":"#b-3ca345e56ff79e20723b5831","relation":"same-reasoning"}]},{"id":"b-7a3f26390c0510d0285059cb","type":"p","scope":"caption-compositional-gap","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p52.html#b-7a3f26390c0510d0285059cb","content":"想象视觉分类器已会识别一种零件 K，文字语料也出现过“零件 K”，但图像描述训练集从没有 K 的配对句子。两个模块分别有知识，不等于联合系统已经知道在什么视觉条件下、怎样把 K 放进一条描述。","sha256":"2eb20432e38bdbf76b853ac229b1a484296a9f4e03bfc13f04eec1f6b166b5c0","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"caption-visual-conditioning","title":"图像怎样约束一句新生成的描述？"},"page":{"id":"caption-composition-gap","title":"找出物体与词语之间尚未学会的使用关系","number":1},"sectionTitle":"DCC：分别认识物体和词语，为什么还不会组合描述","currentQuestion":"从训练目标到自由生成，怎样检查模型实际使用了图像？","sourceSlides":["l3-r2-p52"],"knowledgeBlock":"caption-compositional-gap","formulas":[],"previousBlock":"b-a1ed0bc94e3cf20cea2f656a","nextBlock":"b-72d1b0f41080c867bfe36520","prerequisites":[{"id":"b-a1ed0bc94e3cf20cea2f656a","url":"#b-a1ed0bc94e3cf20cea2f656a","role":"section-title"},{"id":"b-57450bcdb34f112e54acdb2f","url":"#b-57450bcdb34f112e54acdb2f","role":"problem-context"}],"relatedBlocks":[{"id":"b-d927cfb52c3f924a500f5628","url":"#b-d927cfb52c3f924a500f5628","relation":"figure-and-explanation"},{"id":"b-a1ed0bc94e3cf20cea2f656a","url":"#b-a1ed0bc94e3cf20cea2f656a","relation":"same-reasoning"},{"id":"b-72d1b0f41080c867bfe36520","url":"#b-72d1b0f41080c867bfe36520","relation":"same-reasoning"},{"id":"b-97c26f9371af40c489cea434","url":"#b-97c26f9371af40c489cea434","relation":"same-reasoning"},{"id":"b-3ca345e56ff79e20723b5831","url":"#b-3ca345e56ff79e20723b5831","relation":"same-reasoning"}]},{"id":"b-72d1b0f41080c867bfe36520","type":"p","scope":"caption-compositional-gap","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p52.html#b-72d1b0f41080c867bfe36520","content":"DCC 将这个缺口明确为研究设定：某对象不在配对图像—句子训练资料里，却可以出现在独立类别图像和未配对文本中。方法利用这些外部知识并在语义相近概念间迁移，使描述模型能够提及配对训练中缺席的对象。论文在 MSCOCO 上做经验评价，也展示无配对描述的 ImageNet 类别例子。这是“组合与迁移需要额外机制”的正面案例，而不是宣称从所有数据中都没见过对象。","sha256":"3cd30c1374864eca90570c520a8358763c29f4142490f2d6090db4bfe1fb213f","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"caption-visual-conditioning","title":"图像怎样约束一句新生成的描述？"},"page":{"id":"caption-composition-gap","title":"找出物体与词语之间尚未学会的使用关系","number":1},"sectionTitle":"DCC：分别认识物体和词语，为什么还不会组合描述","currentQuestion":"从训练目标到自由生成，怎样检查模型实际使用了图像？","sourceSlides":["l3-r2-p52"],"knowledgeBlock":"caption-compositional-gap","formulas":[],"previousBlock":"b-7a3f26390c0510d0285059cb","nextBlock":"b-97c26f9371af40c489cea434","prerequisites":[{"id":"b-a1ed0bc94e3cf20cea2f656a","url":"#b-a1ed0bc94e3cf20cea2f656a","role":"section-title"},{"id":"b-57450bcdb34f112e54acdb2f","url":"#b-57450bcdb34f112e54acdb2f","role":"problem-context"}],"relatedBlocks":[{"id":"b-d927cfb52c3f924a500f5628","url":"#b-d927cfb52c3f924a500f5628","relation":"figure-and-explanation"},{"id":"b-a1ed0bc94e3cf20cea2f656a","url":"#b-a1ed0bc94e3cf20cea2f656a","relation":"same-reasoning"},{"id":"b-7a3f26390c0510d0285059cb","url":"#b-7a3f26390c0510d0285059cb","relation":"same-reasoning"},{"id":"b-97c26f9371af40c489cea434","url":"#b-97c26f9371af40c489cea434","relation":"same-reasoning"},{"id":"b-3ca345e56ff79e20723b5831","url":"#b-3ca345e56ff79e20723b5831","relation":"same-reasoning"}]},{"id":"b-97c26f9371af40c489cea434","type":"p","scope":"caption-compositional-gap","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p52.html#b-97c26f9371af40c489cea434","content":"网页零件 K 是课程自定例子，不是原论文类别或逐字输出。它帮助拆开三个来源：视觉证据教“它长什么样”，文字语料教“这个词怎样使用”，配对数据或迁移机制教“在什么画面里应该说它”。若只在两端各测一次准确率，就没有直接检验第三项。","sha256":"75ba8dac172d55602c23a0bebea1b1f45763290930c94cc0256fc6a64e90714c","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"caption-visual-conditioning","title":"图像怎样约束一句新生成的描述？"},"page":{"id":"caption-composition-gap","title":"找出物体与词语之间尚未学会的使用关系","number":1},"sectionTitle":"DCC：分别认识物体和词语，为什么还不会组合描述","currentQuestion":"从训练目标到自由生成，怎样检查模型实际使用了图像？","sourceSlides":["l3-r2-p52"],"knowledgeBlock":"caption-compositional-gap","formulas":[],"previousBlock":"b-72d1b0f41080c867bfe36520","nextBlock":"b-3ca345e56ff79e20723b5831","prerequisites":[{"id":"b-a1ed0bc94e3cf20cea2f656a","url":"#b-a1ed0bc94e3cf20cea2f656a","role":"section-title"},{"id":"b-57450bcdb34f112e54acdb2f","url":"#b-57450bcdb34f112e54acdb2f","role":"problem-context"}],"relatedBlocks":[{"id":"b-d927cfb52c3f924a500f5628","url":"#b-d927cfb52c3f924a500f5628","relation":"figure-and-explanation"},{"id":"b-a1ed0bc94e3cf20cea2f656a","url":"#b-a1ed0bc94e3cf20cea2f656a","relation":"same-reasoning"},{"id":"b-7a3f26390c0510d0285059cb","url":"#b-7a3f26390c0510d0285059cb","relation":"same-reasoning"},{"id":"b-72d1b0f41080c867bfe36520","url":"#b-72d1b0f41080c867bfe36520","relation":"same-reasoning"},{"id":"b-3ca345e56ff79e20723b5831","url":"#b-3ca345e56ff79e20723b5831","relation":"same-reasoning"}]},{"id":"b-3ca345e56ff79e20723b5831","type":"p","scope":"caption-compositional-gap","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p52.html#b-3ca345e56ff79e20723b5831","content":"这也预告 P64 的问题。投影层让视觉向量在维度上接得上语言模型，只解决接口的形状；根据视觉条件正确使用已有词语，还需要训练建立联合使用关系。","sha256":"d157be2058eab4edd7ef1474fdf09f94e9d9fb6bf033fbdd8aafc6d22e6713b8","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"caption-visual-conditioning","title":"图像怎样约束一句新生成的描述？"},"page":{"id":"caption-composition-gap","title":"找出物体与词语之间尚未学会的使用关系","number":1},"sectionTitle":"DCC：分别认识物体和词语，为什么还不会组合描述","currentQuestion":"从训练目标到自由生成，怎样检查模型实际使用了图像？","sourceSlides":["l3-r2-p52"],"knowledgeBlock":"caption-compositional-gap","formulas":[],"previousBlock":"b-97c26f9371af40c489cea434","nextBlock":"b-2aee9349597bc5b1add6b6d8","prerequisites":[{"id":"b-a1ed0bc94e3cf20cea2f656a","url":"#b-a1ed0bc94e3cf20cea2f656a","role":"section-title"},{"id":"b-57450bcdb34f112e54acdb2f","url":"#b-57450bcdb34f112e54acdb2f","role":"problem-context"}],"relatedBlocks":[{"id":"b-d927cfb52c3f924a500f5628","url":"#b-d927cfb52c3f924a500f5628","relation":"figure-and-explanation"},{"id":"b-a1ed0bc94e3cf20cea2f656a","url":"#b-a1ed0bc94e3cf20cea2f656a","relation":"same-reasoning"},{"id":"b-7a3f26390c0510d0285059cb","url":"#b-7a3f26390c0510d0285059cb","relation":"same-reasoning"},{"id":"b-72d1b0f41080c867bfe36520","url":"#b-72d1b0f41080c867bfe36520","relation":"same-reasoning"},{"id":"b-97c26f9371af40c489cea434","url":"#b-97c26f9371af40c489cea434","relation":"same-reasoning"}]},{"id":"b-2aee9349597bc5b1add6b6d8","type":"h3","scope":"dialogue","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p52.html#b-2aee9349597bc5b1add6b6d8","content":"### 沿着图解追问","sha256":"f16be0424762a44344338696f06ef266c4571d15367ae2cbbd28fac98ac7e9cb","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"caption-visual-conditioning","title":"图像怎样约束一句新生成的描述？"},"page":{"id":"caption-composition-gap","title":"找出物体与词语之间尚未学会的使用关系","number":1},"sectionTitle":"找出物体与词语之间尚未学会的使用关系","currentQuestion":"从训练目标到自由生成，怎样检查模型实际使用了图像？","sourceSlides":["l3-r2-p52"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-3ca345e56ff79e20723b5831","nextBlock":"b-df6d6c4e8c827dcf011f2307","prerequisites":[{"id":"b-a1ed0bc94e3cf20cea2f656a","url":"#b-a1ed0bc94e3cf20cea2f656a","role":"section-title"},{"id":"b-57450bcdb34f112e54acdb2f","url":"#b-57450bcdb34f112e54acdb2f","role":"problem-context"}],"relatedBlocks":[{"id":"b-d927cfb52c3f924a500f5628","url":"#b-d927cfb52c3f924a500f5628","relation":"figure-and-explanation"}]},{"id":"b-df6d6c4e8c827dcf011f2307","type":"p","scope":"dialogue","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p52.html#b-df6d6c4e8c827dcf011f2307","content":"以下是备课用的模拟师生对话，不是课堂实录。","sha256":"b25685f04cd1b885f3a5999e6a912460c7424021cdc5b2190584242086a82681","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"caption-visual-conditioning","title":"图像怎样约束一句新生成的描述？"},"page":{"id":"caption-composition-gap","title":"找出物体与词语之间尚未学会的使用关系","number":1},"sectionTitle":"找出物体与词语之间尚未学会的使用关系","currentQuestion":"从训练目标到自由生成，怎样检查模型实际使用了图像？","sourceSlides":["l3-r2-p52"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-2aee9349597bc5b1add6b6d8","nextBlock":"b-71599dc8e1a484c3aa0873cd","prerequisites":[{"id":"b-2aee9349597bc5b1add6b6d8","url":"#b-2aee9349597bc5b1add6b6d8","role":"section-title"},{"id":"b-57450bcdb34f112e54acdb2f","url":"#b-57450bcdb34f112e54acdb2f","role":"problem-context"}],"relatedBlocks":[{"id":"b-d927cfb52c3f924a500f5628","url":"#b-d927cfb52c3f924a500f5628","relation":"figure-and-explanation"}]},{"id":"b-71599dc8e1a484c3aa0873cd","type":"p","scope":"dialogue","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p52.html#b-71599dc8e1a484c3aa0873cd","content":"**学生：**既然分类器认识 K，把分类结果接进句子就行吗？","sha256":"833db77ab8059985a97f56f7da522d6c3a3035acb7ae45bbc6c1e3d6b4eb516b","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"caption-visual-conditioning","title":"图像怎样约束一句新生成的描述？"},"page":{"id":"caption-composition-gap","title":"找出物体与词语之间尚未学会的使用关系","number":1},"sectionTitle":"找出物体与词语之间尚未学会的使用关系","currentQuestion":"从训练目标到自由生成，怎样检查模型实际使用了图像？","sourceSlides":["l3-r2-p52"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-df6d6c4e8c827dcf011f2307","nextBlock":"b-00a3b42ff2acd44cbb156686","prerequisites":[{"id":"b-2aee9349597bc5b1add6b6d8","url":"#b-2aee9349597bc5b1add6b6d8","role":"section-title"},{"id":"b-57450bcdb34f112e54acdb2f","url":"#b-57450bcdb34f112e54acdb2f","role":"problem-context"}],"relatedBlocks":[{"id":"b-d927cfb52c3f924a500f5628","url":"#b-d927cfb52c3f924a500f5628","relation":"figure-and-explanation"}]},{"id":"b-00a3b42ff2acd44cbb156686","type":"p","scope":"dialogue","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p52.html#b-00a3b42ff2acd44cbb156686","content":"**教师：**这是可用基线，但还要决定词形、位置和与其他对象的关系。DCC 的问题是怎样利用独立视觉和文字知识填补配对数据缺口。","sha256":"252e0d78c2f8254c4ea6c68570f356138a76148e11618a452c7c8058143cfc35","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"caption-visual-conditioning","title":"图像怎样约束一句新生成的描述？"},"page":{"id":"caption-composition-gap","title":"找出物体与词语之间尚未学会的使用关系","number":1},"sectionTitle":"找出物体与词语之间尚未学会的使用关系","currentQuestion":"从训练目标到自由生成，怎样检查模型实际使用了图像？","sourceSlides":["l3-r2-p52"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-71599dc8e1a484c3aa0873cd","nextBlock":"b-b18ea5ef57c414f26698e65f","prerequisites":[{"id":"b-2aee9349597bc5b1add6b6d8","url":"#b-2aee9349597bc5b1add6b6d8","role":"section-title"},{"id":"b-57450bcdb34f112e54acdb2f","url":"#b-57450bcdb34f112e54acdb2f","role":"problem-context"}],"relatedBlocks":[{"id":"b-d927cfb52c3f924a500f5628","url":"#b-d927cfb52c3f924a500f5628","relation":"figure-and-explanation"}]},{"id":"b-b18ea5ef57c414f26698e65f","type":"h2","scope":"practice","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p52.html#b-b18ea5ef57c414f26698e65f","content":"## 沿这个问题继续","sha256":"cf44fcc62782fb4f6bdc42c16c5187d57556a03cb7ce6065e7bd9c36d8e7d733","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"caption-visual-conditioning","title":"图像怎样约束一句新生成的描述？"},"page":{"id":"caption-composition-gap","title":"找出物体与词语之间尚未学会的使用关系","number":1},"sectionTitle":"找出物体与词语之间尚未学会的使用关系","currentQuestion":"从训练目标到自由生成，怎样检查模型实际使用了图像？","sourceSlides":["l3-r2-p52"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-00a3b42ff2acd44cbb156686","nextBlock":"b-f73de35e43955b7758df2036","prerequisites":[{"id":"b-2aee9349597bc5b1add6b6d8","url":"#b-2aee9349597bc5b1add6b6d8","role":"section-title"},{"id":"b-57450bcdb34f112e54acdb2f","url":"#b-57450bcdb34f112e54acdb2f","role":"problem-context"}],"relatedBlocks":[{"id":"b-d927cfb52c3f924a500f5628","url":"#b-d927cfb52c3f924a500f5628","relation":"figure-and-explanation"}]},{"id":"b-f73de35e43955b7758df2036","type":"p","scope":"practice","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p52.html#b-f73de35e43955b7758df2036","content":"先区分“认识物体”“有对应词语”和“能组合成描述”三件事。读 DCC 对新颖对象的限定，再与 P64 的视觉—语言接口学习比较。","sha256":"3f12361d24fb375c4fa9212b387c4035be03d494433b2fe50e44ceac7c08a7e5","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"caption-visual-conditioning","title":"图像怎样约束一句新生成的描述？"},"page":{"id":"caption-composition-gap","title":"找出物体与词语之间尚未学会的使用关系","number":1},"sectionTitle":"找出物体与词语之间尚未学会的使用关系","currentQuestion":"从训练目标到自由生成，怎样检查模型实际使用了图像？","sourceSlides":["l3-r2-p52"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-b18ea5ef57c414f26698e65f","nextBlock":"b-431fdfa8f438f471dfca81a6","prerequisites":[{"id":"b-b18ea5ef57c414f26698e65f","url":"#b-b18ea5ef57c414f26698e65f","role":"section-title"},{"id":"b-57450bcdb34f112e54acdb2f","url":"#b-57450bcdb34f112e54acdb2f","role":"problem-context"}],"relatedBlocks":[{"id":"b-d927cfb52c3f924a500f5628","url":"#b-d927cfb52c3f924a500f5628","relation":"figure-and-explanation"}]},{"id":"b-431fdfa8f438f471dfca81a6","type":"h4","scope":"caption-composition-gap","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p52.html#b-431fdfa8f438f471dfca81a6","content":"#### 图示条件与公式约定","sha256":"ef7f915c1ae63ed67bd23fecf050249273ba0efa8965d2c8d1c1c6b2d3178122","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"caption-visual-conditioning","title":"图像怎样约束一句新生成的描述？"},"page":{"id":"caption-composition-gap","title":"找出物体与词语之间尚未学会的使用关系","number":1},"sectionTitle":"找出物体与词语之间尚未学会的使用关系","currentQuestion":"从训练目标到自由生成，怎样检查模型实际使用了图像？","sourceSlides":["l3-r2-p52"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-f73de35e43955b7758df2036","nextBlock":"b-7f20f3d9480171df79c8f19c","prerequisites":[{"id":"b-b18ea5ef57c414f26698e65f","url":"#b-b18ea5ef57c414f26698e65f","role":"section-title"},{"id":"b-57450bcdb34f112e54acdb2f","url":"#b-57450bcdb34f112e54acdb2f","role":"problem-context"}],"relatedBlocks":[{"id":"b-d927cfb52c3f924a500f5628","url":"#b-d927cfb52c3f924a500f5628","relation":"figure-and-explanation"}]},{"id":"b-7f20f3d9480171df79c8f19c","type":"p","scope":"document","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p52.html#b-7f20f3d9480171df79c8f19c","content":"图中的零件 K 是课程自己的概念例子；不是 DCC 原始实验类别，也不是原论文逐字生成结果。DCC 处理相关物体不在配对图文训练数据中的设定，但可以使用未配对的类别图像和文字语料。","sha256":"da8fefb58c7f9e5e864411adbd618eef5d3c41c618a32ba1af2e8d7d7cf2b4fc","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"caption-visual-conditioning","title":"图像怎样约束一句新生成的描述？"},"page":{"id":"caption-composition-gap","title":"找出物体与词语之间尚未学会的使用关系","number":1},"sectionTitle":"找出物体与词语之间尚未学会的使用关系","currentQuestion":"从训练目标到自由生成，怎样检查模型实际使用了图像？","sourceSlides":["l3-r2-p52"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-431fdfa8f438f471dfca81a6","nextBlock":null,"prerequisites":[{"id":"b-431fdfa8f438f471dfca81a6","url":"#b-431fdfa8f438f471dfca81a6","role":"section-title"},{"id":"b-57450bcdb34f112e54acdb2f","url":"#b-57450bcdb34f112e54acdb2f","role":"problem-context"}],"relatedBlocks":[{"id":"b-d927cfb52c3f924a500f5628","url":"#b-d927cfb52c3f924a500f5628","relation":"figure-and-explanation"}]}]}