{"version":2,"relationshipURLBase":"https://codingai-lec03.pages.dev/course/blocks/vqa-question-selects-information.html","relationshipResolution":"Resolve relative relationship URLs against relationshipURLBase; complete records and absolute URLs are in blocks.","title":"VQA：同图换问题，同问换图","url":"https://codingai-lec03.pages.dev/course/blocks/vqa-question-selects-information.html","markdown":"https://codingai-lec03.pages.dev/course/blocks/vqa-question-selects-information.md","pages":[{"id":"reading-opening","title":"VQA：同图换问题，同问换图","unit":"语言怎样选择当前需要的视觉信息？","unitId":"language-local-and-question","chapter":"III","number":1}],"aliases":[{"oldId":"b-d7f89bba2d01ac8f82f39e61","targetId":"b-d7f89bba2d01ac8f82f39e61","page":"reading-opening","operation":"move","beforeSHA256":"ef10821e77de4b7a277eca39804ee5478a7728953ca2a4891cc00d73e6c79300","afterSHA256":"ef10821e77de4b7a277eca39804ee5478a7728953ca2a4891cc00d73e6c79300","reason":"Same source body in authored semantic page"},{"oldId":"b-0039ac87154da6bbeb3d70d0","targetId":"b-0039ac87154da6bbeb3d70d0","page":"reading-opening","operation":"move","beforeSHA256":"e694c4320e7af6f266b80de8b9d0453b3a6861ef55dfd0b426416c93d755292d","afterSHA256":"e694c4320e7af6f266b80de8b9d0453b3a6861ef55dfd0b426416c93d755292d","reason":"Same source body in authored semantic page"},{"oldId":"b-c41e2a9bd257953c10501001","targetId":"b-c41e2a9bd257953c10501001","page":"reading-opening","operation":"move","beforeSHA256":"aa3f2b08247ec91b010acd6820af5b45f343506a9021cf7091bb5eb6f7be29b5","afterSHA256":"aa3f2b08247ec91b010acd6820af5b45f343506a9021cf7091bb5eb6f7be29b5","reason":"Same source body in authored semantic page"},{"oldId":"b-e658d9a6c6df534456182fcd","targetId":"b-e658d9a6c6df534456182fcd","page":"reading-opening","operation":"move","beforeSHA256":"e6378866c99bf94b5778b07cba5df20b45c14b1e0d97b8c758ff5faaa9e61b5c","afterSHA256":"e6378866c99bf94b5778b07cba5df20b45c14b1e0d97b8c758ff5faaa9e61b5c","reason":"Same source body in authored semantic page"},{"oldId":"b-1cd49f606a10d82d5133db6c","targetId":"b-1cd49f606a10d82d5133db6c","page":"reading-opening","operation":"move","beforeSHA256":"d42e952fa996be83354241968ebec134716cca4b6f254ef0ba36d741e2c9d52a","afterSHA256":"d42e952fa996be83354241968ebec134716cca4b6f254ef0ba36d741e2c9d52a","reason":"Same source body in authored semantic page"},{"oldId":"b-27e6055f92617ad9da0ad37c","targetId":"b-27e6055f92617ad9da0ad37c","page":"reading-opening","operation":"move","beforeSHA256":"ad298559aecc4fbbef5ea5a5aa049c02bbcc3d6bc01e8ac57c78fb39a2e64c0d","afterSHA256":"ad298559aecc4fbbef5ea5a5aa049c02bbcc3d6bc01e8ac57c78fb39a2e64c0d","reason":"Same source body in authored semantic page"},{"oldId":"b-990d2e82cfedfc703cb62039","targetId":"b-990d2e82cfedfc703cb62039","page":"reading-opening","operation":"rewrite","beforeSHA256":"1aea10041688fa2754c7a39996cfce0e972cc024b23db1618326e873a922ffc5","afterSHA256":"8df8766a54a19164d44b519fb9b7518e8718c63cb15d1a926cc2db7113f18637","reason":"Same source identity with figure semantics or reading context enriched"},{"oldId":"b-61df4aedd62b456cb6f1e9c6","targetId":"b-61df4aedd62b456cb6f1e9c6","page":"reading-opening","operation":"move","beforeSHA256":"2b9ccef7e1220e2250cceff1abd8bdd736b42f8da5a223f26b029f44b962e807","afterSHA256":"2b9ccef7e1220e2250cceff1abd8bdd736b42f8da5a223f26b029f44b962e807","reason":"Same source body in authored semantic page"}],"blocks":[{"id":"b-d7f89bba2d01ac8f82f39e61","type":"h1","scope":"vqa-question-selects-information","url":"https://codingai-lec03.pages.dev/course/blocks/vqa-question-selects-information.html#b-d7f89bba2d01ac8f82f39e61","content":"# VQA：同图换问题，同问换图","sha256":"ef10821e77de4b7a277eca39804ee5478a7728953ca2a4891cc00d73e6c79300","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"language-local-and-question","title":"语言怎样选择当前需要的视觉信息？"},"page":{"id":"reading-opening","title":"VQA：同图换问题，同问换图","number":1},"sectionTitle":"VQA：同图换问题，同问换图","currentQuestion":"VQA：同图换问题，同问换图","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":null,"nextBlock":"b-0039ac87154da6bbeb3d70d0","prerequisites":[{"id":"b-0039ac87154da6bbeb3d70d0","url":"#b-0039ac87154da6bbeb3d70d0","role":"problem-context"}],"relatedBlocks":[{"id":"b-990d2e82cfedfc703cb62039","url":"#b-990d2e82cfedfc703cb62039","relation":"figure-and-explanation"}]},{"id":"b-0039ac87154da6bbeb3d70d0","type":"p","scope":"vqa-question-selects-information","url":"https://codingai-lec03.pages.dev/course/blocks/vqa-question-selects-information.html#b-0039ac87154da6bbeb3d70d0","content":"问题在这里选择要读的信息，同一图可以问数量、属性和关系。本页说明沿两个方向组织检查：同图换问、同问换图。LAB14 当前已经有读取像素的可学习分支和单独训练的 question-only，对象属性字典保留为 oracle 热身；三种身份应分别阅读。","sha256":"e694c4320e7af6f266b80de8b9d0453b3a6861ef55dfd0b426416c93d755292d","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"language-local-and-question","title":"语言怎样选择当前需要的视觉信息？"},"page":{"id":"reading-opening","title":"VQA：同图换问题，同问换图","number":1},"sectionTitle":"VQA：同图换问题，同问换图","currentQuestion":"VQA：同图换问题，同问换图","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-d7f89bba2d01ac8f82f39e61","nextBlock":"b-c41e2a9bd257953c10501001","prerequisites":[{"id":"b-d7f89bba2d01ac8f82f39e61","url":"#b-d7f89bba2d01ac8f82f39e61","role":"section-title"}],"relatedBlocks":[{"id":"b-990d2e82cfedfc703cb62039","url":"#b-990d2e82cfedfc703cb62039","relation":"figure-and-explanation"}]},{"id":"b-c41e2a9bd257953c10501001","type":"p","scope":"vqa-question-selects-information","url":"https://codingai-lec03.pages.dev/course/blocks/vqa-question-selects-information.html#b-c41e2a9bd257953c10501001","content":"VQA 的目标是 $p(a\\mid I,q)$。问题 $q$ 不只是输出旁的装饰，而是决定应该从图像中读取什么：问“有几个人”需要数量，问“谁拿球”需要关系，问“球是什么颜色”需要属性。保持图像固定，改变问题，正确答案就可能改变；保持问题固定，改变相关图像，正确答案也应变化。","sha256":"aa3f2b08247ec91b010acd6820af5b45f343506a9021cf7091bb5eb6f7be29b5","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"language-local-and-question","title":"语言怎样选择当前需要的视觉信息？"},"page":{"id":"reading-opening","title":"VQA：同图换问题，同问换图","number":1},"sectionTitle":"VQA：同图换问题，同问换图","currentQuestion":"VQA：同图换问题，同问换图","sourceSlides":[],"knowledgeBlock":null,"formulas":[{"latex":"p(a\\mid I,q)","definitionContext":"VQA：同图换问题，同问换图"},{"latex":"q","definitionContext":"VQA：同图换问题，同问换图"}],"previousBlock":"b-0039ac87154da6bbeb3d70d0","nextBlock":"b-e658d9a6c6df534456182fcd","prerequisites":[{"id":"b-d7f89bba2d01ac8f82f39e61","url":"#b-d7f89bba2d01ac8f82f39e61","role":"section-title"},{"id":"b-0039ac87154da6bbeb3d70d0","url":"#b-0039ac87154da6bbeb3d70d0","role":"problem-context"}],"relatedBlocks":[{"id":"b-990d2e82cfedfc703cb62039","url":"#b-990d2e82cfedfc703cb62039","relation":"figure-and-explanation"}]},{"id":"b-e658d9a6c6df534456182fcd","type":"p","scope":"vqa-question-selects-information","url":"https://codingai-lec03.pages.dev/course/blocks/vqa-question-selects-information.html#b-e658d9a6c6df534456182fcd","content":"一个封闭答案模型可以在预设词表里分类，一个生成式模型可以逐词回答。前者易于定义准确率，但答案范围受词表限制；后者表达更灵活，却需要处理同义表达、冗长回答和证据对应。不能只把分类头输出的一个标签换成长句，就称为具有新的视觉推理能力。","sha256":"e6378866c99bf94b5778b07cba5df20b45c14b1e0d97b8c758ff5faaa9e61b5c","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"language-local-and-question","title":"语言怎样选择当前需要的视觉信息？"},"page":{"id":"reading-opening","title":"VQA：同图换问题，同问换图","number":1},"sectionTitle":"VQA：同图换问题，同问换图","currentQuestion":"VQA：同图换问题，同问换图","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-c41e2a9bd257953c10501001","nextBlock":"b-1cd49f606a10d82d5133db6c","prerequisites":[{"id":"b-d7f89bba2d01ac8f82f39e61","url":"#b-d7f89bba2d01ac8f82f39e61","role":"section-title"},{"id":"b-0039ac87154da6bbeb3d70d0","url":"#b-0039ac87154da6bbeb3d70d0","role":"problem-context"}],"relatedBlocks":[{"id":"b-990d2e82cfedfc703cb62039","url":"#b-990d2e82cfedfc703cb62039","relation":"figure-and-explanation"}]},{"id":"b-1cd49f606a10d82d5133db6c","type":"p","scope":"vqa-question-selects-information","url":"https://codingai-lec03.pages.dev/course/blocks/vqa-question-selects-information.html#b-1cd49f606a10d82d5133db6c","content":"语言先验是有用的知识，也可能成为捷径。如果训练里“左边物体什么颜色”通常答红色，question-only 模型能取得不错成绩，却无法应对同问题下的蓝物体。VQA v2 的研究用相同问题、不同正确答案的成对图像，加强视觉证据在评价中的作用。这种设计把“答对”进一步变成“答对所需的信息确实来自图像”。","sha256":"d42e952fa996be83354241968ebec134716cca4b6f254ef0ba36d741e2c9d52a","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"language-local-and-question","title":"语言怎样选择当前需要的视觉信息？"},"page":{"id":"reading-opening","title":"VQA：同图换问题，同问换图","number":1},"sectionTitle":"VQA：同图换问题，同问换图","currentQuestion":"VQA：同图换问题，同问换图","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-e658d9a6c6df534456182fcd","nextBlock":"b-27e6055f92617ad9da0ad37c","prerequisites":[{"id":"b-d7f89bba2d01ac8f82f39e61","url":"#b-d7f89bba2d01ac8f82f39e61","role":"section-title"},{"id":"b-0039ac87154da6bbeb3d70d0","url":"#b-0039ac87154da6bbeb3d70d0","role":"problem-context"}],"relatedBlocks":[{"id":"b-990d2e82cfedfc703cb62039","url":"#b-990d2e82cfedfc703cb62039","relation":"figure-and-explanation"}]},{"id":"b-27e6055f92617ad9da0ad37c","type":"p","scope":"vqa-question-selects-information","url":"https://codingai-lec03.pages.dev/course/blocks/vqa-question-selects-information.html#b-27e6055f92617ad9da0ad37c","content":"课堂上可先固定图 A 左红右蓝：问左颜色答红，问右颜色答蓝。再换成左蓝右红的图 B，保持左颜色问题，此时应答蓝。三次比较让图像条件和问题条件分别显示作用。读取场景字典中的 `left_color` 是已知属性 oracle，它能检验问答逻辑，但不能检验从像素感知颜色。","sha256":"ad298559aecc4fbbef5ea5a5aa049c02bbcc3d6bc01e8ac57c78fb39a2e64c0d","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"language-local-and-question","title":"语言怎样选择当前需要的视觉信息？"},"page":{"id":"reading-opening","title":"VQA：同图换问题，同问换图","number":1},"sectionTitle":"VQA：同图换问题，同问换图","currentQuestion":"VQA：同图换问题，同问换图","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-1cd49f606a10d82d5133db6c","nextBlock":"b-990d2e82cfedfc703cb62039","prerequisites":[{"id":"b-d7f89bba2d01ac8f82f39e61","url":"#b-d7f89bba2d01ac8f82f39e61","role":"section-title"},{"id":"b-0039ac87154da6bbeb3d70d0","url":"#b-0039ac87154da6bbeb3d70d0","role":"problem-context"}],"relatedBlocks":[{"id":"b-990d2e82cfedfc703cb62039","url":"#b-990d2e82cfedfc703cb62039","relation":"figure-and-explanation"}]},{"id":"b-990d2e82cfedfc703cb62039","type":"figure","scope":"vqa-question-selects-information","url":"https://codingai-lec03.pages.dev/course/blocks/vqa-question-selects-information.html#b-990d2e82cfedfc703cb62039","content":"图内标签与数值：不给图片；有几个人？；需依据图像确定；谁拿着球？；需依据图像确定；球是什么颜色？；需依据图像确定；同一问题，换掉相关图像，正确答案应怎样改变？；答案为人工核对的场景属性；“不给图片”展示评价条件，不是模型问答实验。\n\n图像隐藏后，数量、颜色和持球者都失去当前场景证据；页面没有伪造无图模型回答。 查看对应图解状态\n\n- 不给图片\n- 有几个人？\n- 需依据图像确定\n- 谁拿着球？\n- 需依据图像确定\n- 球是什么颜色？\n- 需依据图像确定\n- 同一问题，换掉相关图像，正确答案应怎样改变？\n- 答案为人工核对的场景属性；“不给图片”展示评价条件，不是模型问答实验。图像隐藏后，数量、颜色和持球者都失去当前场景证据；页面没有伪造无图模型回答。 [查看对应图解状态](/#/l3-r2-p54/1)[查看完整图解](/#/l3-r2-p54/1)","sha256":"8df8766a54a19164d44b519fb9b7518e8718c63cb15d1a926cc2db7113f18637","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"language-local-and-question","title":"语言怎样选择当前需要的视觉信息？"},"page":{"id":"reading-opening","title":"VQA：同图换问题，同问换图","number":1},"sectionTitle":"VQA：同图换问题，同问换图","currentQuestion":"VQA：同图换问题，同问换图","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"figure":{"caption":"图像隐藏后，数量、颜色和持球者都失去当前场景证据；页面没有伪造无图模型回答。 查看对应图解状态","labels":["不给图片","有几个人？","需依据图像确定","谁拿着球？","需依据图像确定","球是什么颜色？","需依据图像确定","同一问题，换掉相关图像，正确答案应怎样改变？","答案为人工核对的场景属性；“不给图片”展示评价条件，不是模型问答实验。"],"description":"图像隐藏后，数量、颜色和持球者都失去当前场景证据；页面没有伪造无图模型回答。 查看对应图解状态不给图片有几个人？需依据图像确定谁拿着球？需依据图像确定球是什么颜色？需依据图像确定同一问题，换掉相关图像，正确答案应怎样改变？答案为人工核对的场景属性；“不给图片”展示评价条件，不是模型问答实验。","evidenceIdentity":"图像隐藏后，数量、颜色和持球者都失去当前场景证据；页面没有伪造无图模型回答。 查看对应图解状态","images":[]},"previousBlock":"b-27e6055f92617ad9da0ad37c","nextBlock":"b-61df4aedd62b456cb6f1e9c6","prerequisites":[{"id":"b-d7f89bba2d01ac8f82f39e61","url":"#b-d7f89bba2d01ac8f82f39e61","role":"section-title"},{"id":"b-0039ac87154da6bbeb3d70d0","url":"#b-0039ac87154da6bbeb3d70d0","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-61df4aedd62b456cb6f1e9c6","type":"h2","scope":"vqa-question-selects-information","url":"https://codingai-lec03.pages.dev/course/blocks/vqa-question-selects-information.html#b-61df4aedd62b456cb6f1e9c6","content":"## 继续阅读与实验","sha256":"2b9ccef7e1220e2250cceff1abd8bdd736b42f8da5a223f26b029f44b962e807","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"language-local-and-question","title":"语言怎样选择当前需要的视觉信息？"},"page":{"id":"reading-opening","title":"VQA：同图换问题，同问换图","number":1},"sectionTitle":"VQA：同图换问题，同问换图","currentQuestion":"VQA：同图换问题，同问换图","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-990d2e82cfedfc703cb62039","nextBlock":null,"prerequisites":[{"id":"b-d7f89bba2d01ac8f82f39e61","url":"#b-d7f89bba2d01ac8f82f39e61","role":"section-title"},{"id":"b-0039ac87154da6bbeb3d70d0","url":"#b-0039ac87154da6bbeb3d70d0","role":"problem-context"}],"relatedBlocks":[{"id":"b-990d2e82cfedfc703cb62039","url":"#b-990d2e82cfedfc703cb62039","relation":"figure-and-explanation"}]}]}