{"version":2,"relationshipURLBase":"https://codingai-lec03.pages.dev/course/blocks/caption-compositional-gap.html","relationshipResolution":"Resolve relative relationship URLs against relationshipURLBase; complete records and absolute URLs are in blocks.","title":"DCC：分别认识物体和词语，为什么还不会组合描述","url":"https://codingai-lec03.pages.dev/course/blocks/caption-compositional-gap.html","markdown":"https://codingai-lec03.pages.dev/course/blocks/caption-compositional-gap.md","pages":[{"id":"reading-opening","title":"DCC：分别认识物体和词语，为什么还不会组合描述","unit":"图像怎样约束一句新生成的描述？","unitId":"caption-visual-conditioning","chapter":"III","number":1}],"aliases":[{"oldId":"b-bdb2e8dcefb3cd2cd7099c9b","targetId":"b-bdb2e8dcefb3cd2cd7099c9b","page":"reading-opening","operation":"move","beforeSHA256":"976ff9ac0992c1d1a0d067d838f472bf339ea670c472ccf4d99e86dea42236cd","afterSHA256":"976ff9ac0992c1d1a0d067d838f472bf339ea670c472ccf4d99e86dea42236cd","reason":"Same source body in authored semantic page"},{"oldId":"b-d9a9b660d44d1330914f15fe","targetId":"b-d9a9b660d44d1330914f15fe","page":"reading-opening","operation":"move","beforeSHA256":"7f63704d805edecaecdf1d96b13560ed33ec21e13105d30a9da9fe6493863d3d","afterSHA256":"7f63704d805edecaecdf1d96b13560ed33ec21e13105d30a9da9fe6493863d3d","reason":"Same source body in authored semantic page"},{"oldId":"b-afe3678289f8ed65911ac847","targetId":"b-afe3678289f8ed65911ac847","page":"reading-opening","operation":"move","beforeSHA256":"2eb20432e38bdbf76b853ac229b1a484296a9f4e03bfc13f04eec1f6b166b5c0","afterSHA256":"2eb20432e38bdbf76b853ac229b1a484296a9f4e03bfc13f04eec1f6b166b5c0","reason":"Same source body in authored semantic page"},{"oldId":"b-025c843d8da1be9fb73c2622","targetId":"b-025c843d8da1be9fb73c2622","page":"reading-opening","operation":"move","beforeSHA256":"3cd30c1374864eca90570c520a8358763c29f4142490f2d6090db4bfe1fb213f","afterSHA256":"3cd30c1374864eca90570c520a8358763c29f4142490f2d6090db4bfe1fb213f","reason":"Same source body in authored semantic page"},{"oldId":"b-47b4f18ab6072bfa43ffe0f4","targetId":"b-47b4f18ab6072bfa43ffe0f4","page":"reading-opening","operation":"move","beforeSHA256":"75ba8dac172d55602c23a0bebea1b1f45763290930c94cc0256fc6a64e90714c","afterSHA256":"75ba8dac172d55602c23a0bebea1b1f45763290930c94cc0256fc6a64e90714c","reason":"Same source body in authored semantic page"},{"oldId":"b-fc4b2f81e9f2fba511226ca9","targetId":"b-fc4b2f81e9f2fba511226ca9","page":"reading-opening","operation":"move","beforeSHA256":"d157be2058eab4edd7ef1474fdf09f94e9d9fb6bf033fbdd8aafc6d22e6713b8","afterSHA256":"d157be2058eab4edd7ef1474fdf09f94e9d9fb6bf033fbdd8aafc6d22e6713b8","reason":"Same source body in authored semantic page"},{"oldId":"b-fc9833053642779ceb549769","targetId":"b-fc9833053642779ceb549769","page":"reading-opening","operation":"rewrite","beforeSHA256":"6d86a62cccf887e9ac52ce44477fd36d7ce55ad750a521969c4dd12f6a80db8c","afterSHA256":"961dcdb67dc12b61e98c0dbe42090857fff98da0179ebe4b54ea497d21b6ffd7","reason":"Same source identity with figure semantics or reading context enriched"},{"oldId":"b-c7f0a8c14f4ea81cb868fc2e","targetId":"b-c7f0a8c14f4ea81cb868fc2e","page":"reading-opening","operation":"move","beforeSHA256":"2b9ccef7e1220e2250cceff1abd8bdd736b42f8da5a223f26b029f44b962e807","afterSHA256":"2b9ccef7e1220e2250cceff1abd8bdd736b42f8da5a223f26b029f44b962e807","reason":"Same source body in authored semantic page"}],"blocks":[{"id":"b-bdb2e8dcefb3cd2cd7099c9b","type":"h1","scope":"caption-compositional-gap","url":"https://codingai-lec03.pages.dev/course/blocks/caption-compositional-gap.html#b-bdb2e8dcefb3cd2cd7099c9b","content":"# DCC：分别认识物体和词语，为什么还不会组合描述","sha256":"976ff9ac0992c1d1a0d067d838f472bf339ea670c472ccf4d99e86dea42236cd","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"caption-visual-conditioning","title":"图像怎样约束一句新生成的描述？"},"page":{"id":"reading-opening","title":"DCC：分别认识物体和词语，为什么还不会组合描述","number":1},"sectionTitle":"DCC：分别认识物体和词语，为什么还不会组合描述","currentQuestion":"DCC：分别认识物体和词语，为什么还不会组合描述","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":null,"nextBlock":"b-d9a9b660d44d1330914f15fe","prerequisites":[{"id":"b-d9a9b660d44d1330914f15fe","url":"#b-d9a9b660d44d1330914f15fe","role":"problem-context"}],"relatedBlocks":[{"id":"b-fc9833053642779ceb549769","url":"#b-fc9833053642779ceb549769","relation":"figure-and-explanation"}]},{"id":"b-d9a9b660d44d1330914f15fe","type":"p","scope":"caption-compositional-gap","url":"https://codingai-lec03.pages.dev/course/blocks/caption-compositional-gap.html#b-d9a9b660d44d1330914f15fe","content":"本页的缺口发生在“知识如何共同使用”：视觉认识某物体、文字存在对应词，并不自动提供描述时的连接规则。DCC 的新颖对象限定在配对描述资料，而非所有数据。课程零件 K 是解释用例，理解这一限定后再回看 LLaVA 的接口学习。","sha256":"7f63704d805edecaecdf1d96b13560ed33ec21e13105d30a9da9fe6493863d3d","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"caption-visual-conditioning","title":"图像怎样约束一句新生成的描述？"},"page":{"id":"reading-opening","title":"DCC：分别认识物体和词语，为什么还不会组合描述","number":1},"sectionTitle":"DCC：分别认识物体和词语，为什么还不会组合描述","currentQuestion":"DCC：分别认识物体和词语，为什么还不会组合描述","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-bdb2e8dcefb3cd2cd7099c9b","nextBlock":"b-afe3678289f8ed65911ac847","prerequisites":[{"id":"b-bdb2e8dcefb3cd2cd7099c9b","url":"#b-bdb2e8dcefb3cd2cd7099c9b","role":"section-title"}],"relatedBlocks":[{"id":"b-fc9833053642779ceb549769","url":"#b-fc9833053642779ceb549769","relation":"figure-and-explanation"}]},{"id":"b-afe3678289f8ed65911ac847","type":"p","scope":"caption-compositional-gap","url":"https://codingai-lec03.pages.dev/course/blocks/caption-compositional-gap.html#b-afe3678289f8ed65911ac847","content":"想象视觉分类器已会识别一种零件 K，文字语料也出现过“零件 K”，但图像描述训练集从没有 K 的配对句子。两个模块分别有知识，不等于联合系统已经知道在什么视觉条件下、怎样把 K 放进一条描述。","sha256":"2eb20432e38bdbf76b853ac229b1a484296a9f4e03bfc13f04eec1f6b166b5c0","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"caption-visual-conditioning","title":"图像怎样约束一句新生成的描述？"},"page":{"id":"reading-opening","title":"DCC：分别认识物体和词语，为什么还不会组合描述","number":1},"sectionTitle":"DCC：分别认识物体和词语，为什么还不会组合描述","currentQuestion":"DCC：分别认识物体和词语，为什么还不会组合描述","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-d9a9b660d44d1330914f15fe","nextBlock":"b-025c843d8da1be9fb73c2622","prerequisites":[{"id":"b-bdb2e8dcefb3cd2cd7099c9b","url":"#b-bdb2e8dcefb3cd2cd7099c9b","role":"section-title"},{"id":"b-d9a9b660d44d1330914f15fe","url":"#b-d9a9b660d44d1330914f15fe","role":"problem-context"}],"relatedBlocks":[{"id":"b-fc9833053642779ceb549769","url":"#b-fc9833053642779ceb549769","relation":"figure-and-explanation"}]},{"id":"b-025c843d8da1be9fb73c2622","type":"p","scope":"caption-compositional-gap","url":"https://codingai-lec03.pages.dev/course/blocks/caption-compositional-gap.html#b-025c843d8da1be9fb73c2622","content":"DCC 将这个缺口明确为研究设定：某对象不在配对图像—句子训练资料里，却可以出现在独立类别图像和未配对文本中。方法利用这些外部知识并在语义相近概念间迁移，使描述模型能够提及配对训练中缺席的对象。论文在 MSCOCO 上做经验评价，也展示无配对描述的 ImageNet 类别例子。这是“组合与迁移需要额外机制”的正面案例，而不是宣称从所有数据中都没见过对象。","sha256":"3cd30c1374864eca90570c520a8358763c29f4142490f2d6090db4bfe1fb213f","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"caption-visual-conditioning","title":"图像怎样约束一句新生成的描述？"},"page":{"id":"reading-opening","title":"DCC：分别认识物体和词语，为什么还不会组合描述","number":1},"sectionTitle":"DCC：分别认识物体和词语，为什么还不会组合描述","currentQuestion":"DCC：分别认识物体和词语，为什么还不会组合描述","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-afe3678289f8ed65911ac847","nextBlock":"b-47b4f18ab6072bfa43ffe0f4","prerequisites":[{"id":"b-bdb2e8dcefb3cd2cd7099c9b","url":"#b-bdb2e8dcefb3cd2cd7099c9b","role":"section-title"},{"id":"b-d9a9b660d44d1330914f15fe","url":"#b-d9a9b660d44d1330914f15fe","role":"problem-context"}],"relatedBlocks":[{"id":"b-fc9833053642779ceb549769","url":"#b-fc9833053642779ceb549769","relation":"figure-and-explanation"}]},{"id":"b-47b4f18ab6072bfa43ffe0f4","type":"p","scope":"caption-compositional-gap","url":"https://codingai-lec03.pages.dev/course/blocks/caption-compositional-gap.html#b-47b4f18ab6072bfa43ffe0f4","content":"网页零件 K 是课程自定例子，不是原论文类别或逐字输出。它帮助拆开三个来源：视觉证据教“它长什么样”，文字语料教“这个词怎样使用”，配对数据或迁移机制教“在什么画面里应该说它”。若只在两端各测一次准确率，就没有直接检验第三项。","sha256":"75ba8dac172d55602c23a0bebea1b1f45763290930c94cc0256fc6a64e90714c","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"caption-visual-conditioning","title":"图像怎样约束一句新生成的描述？"},"page":{"id":"reading-opening","title":"DCC：分别认识物体和词语，为什么还不会组合描述","number":1},"sectionTitle":"DCC：分别认识物体和词语，为什么还不会组合描述","currentQuestion":"DCC：分别认识物体和词语，为什么还不会组合描述","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-025c843d8da1be9fb73c2622","nextBlock":"b-fc4b2f81e9f2fba511226ca9","prerequisites":[{"id":"b-bdb2e8dcefb3cd2cd7099c9b","url":"#b-bdb2e8dcefb3cd2cd7099c9b","role":"section-title"},{"id":"b-d9a9b660d44d1330914f15fe","url":"#b-d9a9b660d44d1330914f15fe","role":"problem-context"}],"relatedBlocks":[{"id":"b-fc9833053642779ceb549769","url":"#b-fc9833053642779ceb549769","relation":"figure-and-explanation"}]},{"id":"b-fc4b2f81e9f2fba511226ca9","type":"p","scope":"caption-compositional-gap","url":"https://codingai-lec03.pages.dev/course/blocks/caption-compositional-gap.html#b-fc4b2f81e9f2fba511226ca9","content":"这也预告 P64 的问题。投影层让视觉向量在维度上接得上语言模型，只解决接口的形状；根据视觉条件正确使用已有词语，还需要训练建立联合使用关系。","sha256":"d157be2058eab4edd7ef1474fdf09f94e9d9fb6bf033fbdd8aafc6d22e6713b8","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"caption-visual-conditioning","title":"图像怎样约束一句新生成的描述？"},"page":{"id":"reading-opening","title":"DCC：分别认识物体和词语，为什么还不会组合描述","number":1},"sectionTitle":"DCC：分别认识物体和词语，为什么还不会组合描述","currentQuestion":"DCC：分别认识物体和词语，为什么还不会组合描述","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-47b4f18ab6072bfa43ffe0f4","nextBlock":"b-fc9833053642779ceb549769","prerequisites":[{"id":"b-bdb2e8dcefb3cd2cd7099c9b","url":"#b-bdb2e8dcefb3cd2cd7099c9b","role":"section-title"},{"id":"b-d9a9b660d44d1330914f15fe","url":"#b-d9a9b660d44d1330914f15fe","role":"problem-context"}],"relatedBlocks":[{"id":"b-fc9833053642779ceb549769","url":"#b-fc9833053642779ceb549769","relation":"figure-and-explanation"}]},{"id":"b-fc9833053642779ceb549769","type":"figure","scope":"caption-compositional-gap","url":"https://codingai-lec03.pages.dev/course/blocks/caption-compositional-gap.html#b-fc9833053642779ceb549769","content":"图内标签与数值：课程构造：视觉类别“零件 K”已见，配对描述中未见；类别：零件 K；文字材料；“零件 K 安装在支架上。”；“工人检查了零件 K。”；这个词已出现在文字里；配对描述训练；图像 ↔ 句子；已有其他对象的配对；缺少零件 K 的配对；把两边的知识组合成描述；DCC 的问题：在配对描述中未见，仍能否说出？；DCC（CVPR 2016）研究设定的教学转写；零件 K 不是原论文样本或原论文输出。\n\n三份材料分别提供视觉知识、词语知识与配对缺口。 查看对应图解状态\n\n- 课程构造：视觉类别“零件 K”已见，配对描述中未见\n- 类别：零件 K\n- 文字材料\n- “零件 K 安装在支架上。”\n- “工人检查了零件 K。”\n- 这个词已出现在文字里\n- 配对描述训练\n- 图像 ↔ 句子\n- 已有其他对象的配对\n- 缺少零件 K 的配对\n- 把两边的知识组合成描述\n- DCC 的问题：在配对描述中未见，仍能否说出？\n- DCC（CVPR 2016）研究设定的教学转写；零件 K 不是原论文样本或原论文输出。三份材料分别提供视觉知识、词语知识与配对缺口。 [查看对应图解状态](/#/l3-r2-p52/0)[查看完整图解](/#/l3-r2-p52/0)","sha256":"961dcdb67dc12b61e98c0dbe42090857fff98da0179ebe4b54ea497d21b6ffd7","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"caption-visual-conditioning","title":"图像怎样约束一句新生成的描述？"},"page":{"id":"reading-opening","title":"DCC：分别认识物体和词语，为什么还不会组合描述","number":1},"sectionTitle":"DCC：分别认识物体和词语，为什么还不会组合描述","currentQuestion":"DCC：分别认识物体和词语，为什么还不会组合描述","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"figure":{"caption":"三份材料分别提供视觉知识、词语知识与配对缺口。 查看对应图解状态","labels":["课程构造：视觉类别“零件 K”已见，配对描述中未见","类别：零件 K","文字材料","“零件 K 安装在支架上。”","“工人检查了零件 K。”","这个词已出现在文字里","配对描述训练","图像 ↔ 句子","已有其他对象的配对","缺少零件 K 的配对","把两边的知识组合成描述","DCC 的问题：在配对描述中未见，仍能否说出？","DCC（CVPR 2016）研究设定的教学转写；零件 K 不是原论文样本或原论文输出。"],"description":"三份材料分别提供视觉知识、词语知识与配对缺口。 查看对应图解状态课程构造：视觉类别“零件 K”已见，配对描述中未见类别：零件 K文字材料“零件 K 安装在支架上。”“工人检查了零件 K。”这个词已出现在文字里配对描述训练图像 ↔ 句子已有其他对象的配对缺少零件 K 的配对把两边的知识组合成描述DCC 的问题：在配对描述中未见，仍能否说出？DCC（CVPR 2016）研究设定的教学转写；零件 K 不是原论文样本或原论文输出。","evidenceIdentity":"三份材料分别提供视觉知识、词语知识与配对缺口。 查看对应图解状态","images":[{"src":"/assets/display/8757616f3f92bbccd2d8db9c.webp","alt":""}]},"previousBlock":"b-fc4b2f81e9f2fba511226ca9","nextBlock":"b-c7f0a8c14f4ea81cb868fc2e","prerequisites":[{"id":"b-bdb2e8dcefb3cd2cd7099c9b","url":"#b-bdb2e8dcefb3cd2cd7099c9b","role":"section-title"},{"id":"b-d9a9b660d44d1330914f15fe","url":"#b-d9a9b660d44d1330914f15fe","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-c7f0a8c14f4ea81cb868fc2e","type":"h2","scope":"caption-compositional-gap","url":"https://codingai-lec03.pages.dev/course/blocks/caption-compositional-gap.html#b-c7f0a8c14f4ea81cb868fc2e","content":"## 继续阅读与实验","sha256":"2b9ccef7e1220e2250cceff1abd8bdd736b42f8da5a223f26b029f44b962e807","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"caption-visual-conditioning","title":"图像怎样约束一句新生成的描述？"},"page":{"id":"reading-opening","title":"DCC：分别认识物体和词语，为什么还不会组合描述","number":1},"sectionTitle":"DCC：分别认识物体和词语，为什么还不会组合描述","currentQuestion":"DCC：分别认识物体和词语，为什么还不会组合描述","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-fc9833053642779ceb549769","nextBlock":null,"prerequisites":[{"id":"b-bdb2e8dcefb3cd2cd7099c9b","url":"#b-bdb2e8dcefb3cd2cd7099c9b","role":"section-title"},{"id":"b-d9a9b660d44d1330914f15fe","url":"#b-d9a9b660d44d1330914f15fe","role":"problem-context"}],"relatedBlocks":[{"id":"b-fc9833053642779ceb549769","url":"#b-fc9833053642779ceb549769","relation":"figure-and-explanation"}]}]}