{"version":2,"relationshipURLBase":"https://codingai-lec03.pages.dev/course/blocks/llava-instruction-source.html","relationshipResolution":"Resolve relative relationship URLs against relationshipURLBase; complete records and absolute URLs are in blocks.","title":"视觉指令数据：问题和答案中的事实从哪里来","url":"https://codingai-lec03.pages.dev/course/blocks/llava-instruction-source.html","markdown":"https://codingai-lec03.pages.dev/course/blocks/llava-instruction-source.md","pages":[{"id":"reading-opening","title":"视觉指令数据：问题和答案中的事实从哪里来","unit":"可复用视觉与语言组件怎样学会共同回答？","unitId":"llava-interface-and-training","chapter":"III","number":1}],"aliases":[{"oldId":"b-51471fa68d201b1599c5b6fb","targetId":"b-51471fa68d201b1599c5b6fb","page":"reading-opening","operation":"move","beforeSHA256":"9ca0389f8d578f31ed92264aa577bf2587258bb762e5e1b4dfca191eca0bec5a","afterSHA256":"9ca0389f8d578f31ed92264aa577bf2587258bb762e5e1b4dfca191eca0bec5a","reason":"Same source body in authored semantic page"},{"oldId":"b-bd0a0203315d320e96adf86b","targetId":"b-bd0a0203315d320e96adf86b","page":"reading-opening","operation":"move","beforeSHA256":"8a6a2a5319b9b8ab3a47b691fdbcd9ed861d802806a729e54755137d694c7761","afterSHA256":"8a6a2a5319b9b8ab3a47b691fdbcd9ed861d802806a729e54755137d694c7761","reason":"Same source body in authored semantic page"},{"oldId":"b-46d17551bb95fcf144c23004","targetId":"b-46d17551bb95fcf144c23004","page":"reading-opening","operation":"move","beforeSHA256":"4ce8535bd05ed62fc249519a252f16d353f88d71908287e3503be3b94df1b3ce","afterSHA256":"4ce8535bd05ed62fc249519a252f16d353f88d71908287e3503be3b94df1b3ce","reason":"Same source body in authored semantic page"},{"oldId":"b-eed6847d7a95199c8931fd92","targetId":"b-eed6847d7a95199c8931fd92","page":"reading-opening","operation":"move","beforeSHA256":"7621634e9af518b6512f680ae33d2ae56bb958cfe8455e7351064837bd6ebecd","afterSHA256":"7621634e9af518b6512f680ae33d2ae56bb958cfe8455e7351064837bd6ebecd","reason":"Same source body in authored semantic page"},{"oldId":"b-99c4e229882ede0468b3573e","targetId":"b-99c4e229882ede0468b3573e","page":"reading-opening","operation":"move","beforeSHA256":"89177feb914484f1328ffec9fc598ecd16f0720fe4a1e54e98cf3d537ad64582","afterSHA256":"89177feb914484f1328ffec9fc598ecd16f0720fe4a1e54e98cf3d537ad64582","reason":"Same source body in authored semantic page"},{"oldId":"b-3d74cc4bbfdf48f536470de6","targetId":"b-3d74cc4bbfdf48f536470de6","page":"reading-opening","operation":"move","beforeSHA256":"d8b80895551b3f9ccca4bc88b4c6aff4b122e58ce7c07d080abad2159d251c8d","afterSHA256":"d8b80895551b3f9ccca4bc88b4c6aff4b122e58ce7c07d080abad2159d251c8d","reason":"Same source body in authored semantic page"},{"oldId":"b-a94d3ebf1bf22139b70f669f","targetId":"b-a94d3ebf1bf22139b70f669f","page":"reading-opening","operation":"move","beforeSHA256":"2b9ccef7e1220e2250cceff1abd8bdd736b42f8da5a223f26b029f44b962e807","afterSHA256":"2b9ccef7e1220e2250cceff1abd8bdd736b42f8da5a223f26b029f44b962e807","reason":"Same source body in authored semantic page"}],"blocks":[{"id":"b-51471fa68d201b1599c5b6fb","type":"h1","scope":"llava-instruction-source","url":"https://codingai-lec03.pages.dev/course/blocks/llava-instruction-source.html#b-51471fa68d201b1599c5b6fb","content":"# 视觉指令数据：问题和答案中的事实从哪里来","sha256":"9ca0389f8d578f31ed92264aa577bf2587258bb762e5e1b4dfca191eca0bec5a","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"reading-opening","title":"视觉指令数据：问题和答案中的事实从哪里来","number":1},"sectionTitle":"视觉指令数据：问题和答案中的事实从哪里来","currentQuestion":"视觉指令数据：问题和答案中的事实从哪里来","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":null,"nextBlock":"b-bd0a0203315d320e96adf86b","prerequisites":[{"id":"b-bd0a0203315d320e96adf86b","url":"#b-bd0a0203315d320e96adf86b","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-bd0a0203315d320e96adf86b","type":"p","scope":"llava-instruction-source","url":"https://codingai-lec03.pages.dev/course/blocks/llava-instruction-source.html#b-bd0a0203315d320e96adf86b","content":"本页两个状态只改变训练范围与数据任务：先冻结两大模型训练投影，再冻结视觉、更新投影和语言模型。把每个模块是否更新说清，避免用“端到端”概括掉细节。再读指令数据的来源，训练目标的视觉事实必须能追溯到相应输入。","sha256":"8a6a2a5319b9b8ab3a47b691fdbcd9ed861d802806a729e54755137d694c7761","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"reading-opening","title":"视觉指令数据：问题和答案中的事实从哪里来","number":1},"sectionTitle":"视觉指令数据：问题和答案中的事实从哪里来","currentQuestion":"视觉指令数据：问题和答案中的事实从哪里来","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-51471fa68d201b1599c5b6fb","nextBlock":"b-46d17551bb95fcf144c23004","prerequisites":[{"id":"b-51471fa68d201b1599c5b6fb","url":"#b-51471fa68d201b1599c5b6fb","role":"section-title"}],"relatedBlocks":[]},{"id":"b-46d17551bb95fcf144c23004","type":"p","scope":"llava-instruction-source","url":"https://codingai-lec03.pages.dev/course/blocks/llava-instruction-source.html#b-46d17551bb95fcf144c23004","content":"原始 LLaVA 的一部分指令资料由仅接收文本的 GPT-4 协助制作。输入是已有 caption 和对象类别、框坐标等文字化描述；这条数据制作路径没有让 GPT-4 直接查看原图。随后训练 LLaVA 时，原图才作为视觉输入与这些指令配对。","sha256":"4ce8535bd05ed62fc249519a252f16d353f88d71908287e3503be3b94df1b3ce","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"reading-opening","title":"视觉指令数据：问题和答案中的事实从哪里来","number":1},"sectionTitle":"视觉指令数据：问题和答案中的事实从哪里来","currentQuestion":"视觉指令数据：问题和答案中的事实从哪里来","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-bd0a0203315d320e96adf86b","nextBlock":"b-eed6847d7a95199c8931fd92","prerequisites":[{"id":"b-51471fa68d201b1599c5b6fb","url":"#b-51471fa68d201b1599c5b6fb","role":"section-title"},{"id":"b-bd0a0203315d320e96adf86b","url":"#b-bd0a0203315d320e96adf86b","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-eed6847d7a95199c8931fd92","type":"p","scope":"llava-instruction-source","url":"https://codingai-lec03.pages.dev/course/blocks/llava-instruction-source.html#b-eed6847d7a95199c8931fd92","content":"把一份场景记录想成三层。caption 可能说“两人在球场”；对象框提供人物和球的位置；若记录没有谁的手接触球、哪一人持球，数据制作模型就缺乏可靠依据回答持球关系。把回答写得更长不能补上丢失事实。需要追加标注、改问题或保留不确定性。","sha256":"7621634e9af518b6512f680ae33d2ae56bb958cfe8455e7351064837bd6ebecd","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"reading-opening","title":"视觉指令数据：问题和答案中的事实从哪里来","number":1},"sectionTitle":"视觉指令数据：问题和答案中的事实从哪里来","currentQuestion":"视觉指令数据：问题和答案中的事实从哪里来","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-46d17551bb95fcf144c23004","nextBlock":"b-99c4e229882ede0468b3573e","prerequisites":[{"id":"b-51471fa68d201b1599c5b6fb","url":"#b-51471fa68d201b1599c5b6fb","role":"section-title"},{"id":"b-bd0a0203315d320e96adf86b","url":"#b-bd0a0203315d320e96adf86b","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-99c4e229882ede0468b3573e","type":"p","scope":"llava-instruction-source","url":"https://codingai-lec03.pages.dev/course/blocks/llava-instruction-source.html#b-99c4e229882ede0468b3573e","content":"这种流程的积极价值是把已有图文资料转为更丰富的任务形式，让模型学习同图换问、描述与解释等交互。数据生产因此不只是在扩大样本数量，还在定义系统练习哪些条件使用方式。","sha256":"89177feb914484f1328ffec9fc598ecd16f0720fe4a1e54e98cf3d537ad64582","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"reading-opening","title":"视觉指令数据：问题和答案中的事实从哪里来","number":1},"sectionTitle":"视觉指令数据：问题和答案中的事实从哪里来","currentQuestion":"视觉指令数据：问题和答案中的事实从哪里来","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-eed6847d7a95199c8931fd92","nextBlock":"b-3d74cc4bbfdf48f536470de6","prerequisites":[{"id":"b-51471fa68d201b1599c5b6fb","url":"#b-51471fa68d201b1599c5b6fb","role":"section-title"},{"id":"b-bd0a0203315d320e96adf86b","url":"#b-bd0a0203315d320e96adf86b","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-3d74cc4bbfdf48f536470de6","type":"p","scope":"llava-instruction-source","url":"https://codingai-lec03.pages.dev/course/blocks/llava-instruction-source.html#b-3d74cc4bbfdf48f536470de6","content":"学生可以对一张课程图自己做一次检查：只给对象名和框，哪些问题已能确定，哪些要看像素才知道？“左右各是谁”可能由框和类别支持；球面的小字、遮挡细节或动作接触通常需要更细证据。这与本讲开头“训练答案怎样产生”是同一个问题。","sha256":"d8b80895551b3f9ccca4bc88b4c6aff4b122e58ce7c07d080abad2159d251c8d","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"reading-opening","title":"视觉指令数据：问题和答案中的事实从哪里来","number":1},"sectionTitle":"视觉指令数据：问题和答案中的事实从哪里来","currentQuestion":"视觉指令数据：问题和答案中的事实从哪里来","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-99c4e229882ede0468b3573e","nextBlock":"b-a94d3ebf1bf22139b70f669f","prerequisites":[{"id":"b-51471fa68d201b1599c5b6fb","url":"#b-51471fa68d201b1599c5b6fb","role":"section-title"},{"id":"b-bd0a0203315d320e96adf86b","url":"#b-bd0a0203315d320e96adf86b","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-a94d3ebf1bf22139b70f669f","type":"h2","scope":"llava-instruction-source","url":"https://codingai-lec03.pages.dev/course/blocks/llava-instruction-source.html#b-a94d3ebf1bf22139b70f669f","content":"## 继续阅读与实验","sha256":"2b9ccef7e1220e2250cceff1abd8bdd736b42f8da5a223f26b029f44b962e807","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"reading-opening","title":"视觉指令数据：问题和答案中的事实从哪里来","number":1},"sectionTitle":"视觉指令数据：问题和答案中的事实从哪里来","currentQuestion":"视觉指令数据：问题和答案中的事实从哪里来","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-3d74cc4bbfdf48f536470de6","nextBlock":null,"prerequisites":[{"id":"b-51471fa68d201b1599c5b6fb","url":"#b-51471fa68d201b1599c5b6fb","role":"section-title"},{"id":"b-bd0a0203315d320e96adf86b","url":"#b-bd0a0203315d320e96adf86b","role":"problem-context"}],"relatedBlocks":[]}]}