{"version":2,"relationshipURLBase":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p64.html","relationshipResolution":"Resolve relative relationship URLs against relationshipURLBase; complete records and absolute URLs are in blocks.","title":"把视觉表示接入一个已经会说话的模型","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p64.html","markdown":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p64.md","pages":[{"id":"llava-visual-input-space","title":"把视觉向量接进语言模型的输入空间","unit":"可复用视觉与语言组件怎样学会共同回答？","unitId":"llava-interface-and-training","chapter":"III","number":1}],"aliases":[{"oldId":"explanation","targetId":"llava-visual-input-space","operation":"split-explanation-range"},{"oldId":"b-64d42ba7b1d128e6ad1fee81","targetId":"b-64d42ba7b1d128e6ad1fee81","page":"llava-visual-input-space","operation":"move","beforeSHA256":"d71cf15e707c8a3674b4c90a599efee783cb84da5b8c8236a1d01c3add7b4958","afterSHA256":"d71cf15e707c8a3674b4c90a599efee783cb84da5b8c8236a1d01c3add7b4958","reason":"Same source body in authored semantic page"},{"oldId":"b-3216fb14b5c7ef5f644507a1","targetId":"b-3216fb14b5c7ef5f644507a1","page":"llava-visual-input-space","operation":"move","beforeSHA256":"34a5aafff638810df558395242f28c04ea74208cca2f033a62ae9143b5186e17","afterSHA256":"34a5aafff638810df558395242f28c04ea74208cca2f033a62ae9143b5186e17","reason":"Same source body in authored semantic page"},{"oldId":"b-467c4116d3628bc8d44e181c","targetId":"b-467c4116d3628bc8d44e181c","page":"llava-visual-input-space","operation":"move","beforeSHA256":"552d872bb02f5cdd79082b78c8e710e84ab1fa3d0b3c18fc2d2bb2a3f7a39796","afterSHA256":"552d872bb02f5cdd79082b78c8e710e84ab1fa3d0b3c18fc2d2bb2a3f7a39796","reason":"Same source body in authored semantic page"},{"oldId":"b-3091ca8953fdbb5607f75e0e","targetId":"b-3091ca8953fdbb5607f75e0e","page":"llava-visual-input-space","operation":"rewrite","beforeSHA256":"89facd95236b1d3012225c81a7fbdbd4ee12b61202a75f245fb7d24f7077cced","afterSHA256":"98a8e02e443e0b1e637aa920bf74e45447c3e3fddf6aed5251304fe2356c41b7","reason":"Same source identity with figure semantics or reading context enriched"},{"oldId":"b-004e07342853103d5b6fb1ad","targetId":"b-004e07342853103d5b6fb1ad","page":"llava-visual-input-space","operation":"move","beforeSHA256":"786ce10a6b32a492259a3dccd0a54e61cd41d28a34382fb7b5386c5f03b7ad69","afterSHA256":"786ce10a6b32a492259a3dccd0a54e61cd41d28a34382fb7b5386c5f03b7ad69","reason":"Same source body in authored semantic page"},{"oldId":"b-681249dc9d3b2c9063b4a23f","targetId":"b-681249dc9d3b2c9063b4a23f","page":"llava-visual-input-space","operation":"move","beforeSHA256":"be34c1fa3743f31ab28eca11b501b037ca00654cc62c41553e11de82c0d627a0","afterSHA256":"be34c1fa3743f31ab28eca11b501b037ca00654cc62c41553e11de82c0d627a0","reason":"Same source body in authored semantic page"},{"oldId":"b-5d98c63c33bad5dd3139029e","targetId":"b-5d98c63c33bad5dd3139029e","page":"llava-visual-input-space","operation":"move","beforeSHA256":"a807160b70a9bc7c59c8b86af1c74818348d1292fcbb8126cff471d4a6fd9905","afterSHA256":"a807160b70a9bc7c59c8b86af1c74818348d1292fcbb8126cff471d4a6fd9905","reason":"Same source body in authored semantic page"},{"oldId":"b-e0d54b750613f7f80e96847f","targetId":"b-e0d54b750613f7f80e96847f","page":"llava-visual-input-space","operation":"move","beforeSHA256":"301ef52026ba9ef8f1676a239e86372ff81653d9c1842aaf0dd14f01815b352d","afterSHA256":"301ef52026ba9ef8f1676a239e86372ff81653d9c1842aaf0dd14f01815b352d","reason":"Same source body in authored semantic page"},{"oldId":"b-f71342b8bd5c2f77541969a1","targetId":"b-f71342b8bd5c2f77541969a1","page":"llava-visual-input-space","operation":"move","beforeSHA256":"2ae95b830ed5e768a85b88c8740a6bf11f614e3ee1ca7afdb7806bc33dfcfd26","afterSHA256":"2ae95b830ed5e768a85b88c8740a6bf11f614e3ee1ca7afdb7806bc33dfcfd26","reason":"Same source body in authored semantic page"},{"oldId":"b-16fadf7e0f211f780dac81c1","targetId":"b-16fadf7e0f211f780dac81c1","page":"llava-visual-input-space","operation":"move","beforeSHA256":"65181fc527d112fe12214386cf7e5d56ced11acbdbf8c551a01100af82575c3e","afterSHA256":"65181fc527d112fe12214386cf7e5d56ced11acbdbf8c551a01100af82575c3e","reason":"Same source body in authored semantic page"},{"oldId":"b-d38d603f7d85be1f3e32811d","targetId":"b-d38d603f7d85be1f3e32811d","page":"llava-visual-input-space","operation":"move","beforeSHA256":"7b9949dc915b90d1aedf1cb20cd82efb100faee64c980c73912c0b809a3fdb0a","afterSHA256":"7b9949dc915b90d1aedf1cb20cd82efb100faee64c980c73912c0b809a3fdb0a","reason":"Same source body in authored semantic page"},{"oldId":"b-ea32dffe856067aeadfb5467","targetId":"b-ea32dffe856067aeadfb5467","page":"llava-visual-input-space","operation":"move","beforeSHA256":"f16be0424762a44344338696f06ef266c4571d15367ae2cbbd28fac98ac7e9cb","afterSHA256":"f16be0424762a44344338696f06ef266c4571d15367ae2cbbd28fac98ac7e9cb","reason":"Same source body in authored semantic page"},{"oldId":"b-5745b27abbeede2588a307ce","targetId":"b-5745b27abbeede2588a307ce","page":"llava-visual-input-space","operation":"move","beforeSHA256":"b25685f04cd1b885f3a5999e6a912460c7424021cdc5b2190584242086a82681","afterSHA256":"b25685f04cd1b885f3a5999e6a912460c7424021cdc5b2190584242086a82681","reason":"Same source body in authored semantic page"},{"oldId":"b-f8d1304f1fc33cb150d72aeb","targetId":"b-f8d1304f1fc33cb150d72aeb","page":"llava-visual-input-space","operation":"move","beforeSHA256":"acc9219b1ee4d733d601fd38a128e3812bea258ce14e524e4ee8e10a844073b8","afterSHA256":"acc9219b1ee4d733d601fd38a128e3812bea258ce14e524e4ee8e10a844073b8","reason":"Same source body in authored semantic page"},{"oldId":"b-3915726fa3cebb0b8345c869","targetId":"b-3915726fa3cebb0b8345c869","page":"llava-visual-input-space","operation":"move","beforeSHA256":"b716e0bb51bbe9c6192e94908863a1da81fb538e4d0587397110a0e70a5d2c18","afterSHA256":"b716e0bb51bbe9c6192e94908863a1da81fb538e4d0587397110a0e70a5d2c18","reason":"Same source body in authored semantic page"},{"oldId":"b-96a6bc64988fc14fc1f83162","targetId":"b-96a6bc64988fc14fc1f83162","page":"llava-visual-input-space","operation":"move","beforeSHA256":"cf44fcc62782fb4f6bdc42c16c5187d57556a03cb7ce6065e7bd9c36d8e7d733","afterSHA256":"cf44fcc62782fb4f6bdc42c16c5187d57556a03cb7ce6065e7bd9c36d8e7d733","reason":"Same source body in authored semantic page"},{"oldId":"b-5b2a37ac93dbb4c154b0a8d8","targetId":"b-5b2a37ac93dbb4c154b0a8d8","page":"llava-visual-input-space","operation":"move","beforeSHA256":"f088f2fefbfd65eef49998b5eae7aba924db9bbc324feed9a67dafd030c9fd29","afterSHA256":"f088f2fefbfd65eef49998b5eae7aba924db9bbc324feed9a67dafd030c9fd29","reason":"Same source body in authored semantic page"},{"oldId":"b-2d0ac43161dbb263b8abe4f3","targetId":"b-2d0ac43161dbb263b8abe4f3","page":"llava-visual-input-space","operation":"move","beforeSHA256":"213fa2e9822c365027e4a475af9b2fa2b0021ec509f7b2582a5b6207ad19e2aa","afterSHA256":"213fa2e9822c365027e4a475af9b2fa2b0021ec509f7b2582a5b6207ad19e2aa","reason":"Same source body in authored semantic page"},{"oldId":"b-2c05320afb108d4312c318f1","targetId":"b-2c05320afb108d4312c318f1","page":"llava-visual-input-space","operation":"move","beforeSHA256":"411d351539fcfdcacb86e80edc3bab67ee079b470b76e821db3ebc1f5c47de7d","afterSHA256":"411d351539fcfdcacb86e80edc3bab67ee079b470b76e821db3ebc1f5c47de7d","reason":"Same source body in authored semantic page"},{"oldId":"b-3e144a15a1e28abeb1aab175","targetId":"b-3e144a15a1e28abeb1aab175","page":"llava-visual-input-space","operation":"move","beforeSHA256":"a759cb0c908e0148eb93db1ea192238384f1661f7cb102e234ea86e003512bb8","afterSHA256":"a759cb0c908e0148eb93db1ea192238384f1661f7cb102e234ea86e003512bb8","reason":"Same source body in authored semantic page"},{"oldId":"b-b60682ab95420e8a879aabe8","targetId":"b-b60682ab95420e8a879aabe8","page":"llava-visual-input-space","operation":"move","beforeSHA256":"f85b46274ba2ee756bb7f4aeb8de78fd29998022ed14b38fdf994540150b2a34","afterSHA256":"f85b46274ba2ee756bb7f4aeb8de78fd29998022ed14b38fdf994540150b2a34","reason":"Same source body in authored semantic page"},{"oldId":"b-1f5729cb182fab620997ff45","targetId":"b-1f5729cb182fab620997ff45","page":"llava-visual-input-space","operation":"move","beforeSHA256":"12c2a6de3684559337c91c8a21e079797f93c3c3a68e2c752cbab0b92e56fb46","afterSHA256":"12c2a6de3684559337c91c8a21e079797f93c3c3a68e2c752cbab0b92e56fb46","reason":"Same source body in authored semantic page"},{"oldId":"b-ad86d5c0e999571083e9eadc","targetId":"b-ad86d5c0e999571083e9eadc","page":"llava-visual-input-space","operation":"move","beforeSHA256":"d8689dd3bd00bbd1e157ef9f2b0bcb0f2086e4c22ec90935d9fc6b77c72cc077","afterSHA256":"d8689dd3bd00bbd1e157ef9f2b0bcb0f2086e4c22ec90935d9fc6b77c72cc077","reason":"Same source body in authored semantic page"},{"oldId":"llava-visual-input-space-title","targetId":"b-a2850e6fb502a02125169a27","operation":"retain-source-identity"}],"blocks":[{"id":"b-a2850e6fb502a02125169a27","type":"h2","scope":"llava-visual-input-space","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p64.html#b-a2850e6fb502a02125169a27","content":"## 把视觉向量接进语言模型的输入空间","sha256":"7b4a7a347a7ed64ad44a7a884418624c00f7cd82ff2329543bcbb3755515e80a","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"llava-visual-input-space","title":"把视觉向量接进语言模型的输入空间","number":1},"sectionTitle":"把视觉向量接进语言模型的输入空间","currentQuestion":"接口、更新参数和指令事实各自需要怎样检查？","sourceSlides":["l3-r2-p64"],"knowledgeBlock":null,"formulas":[],"previousBlock":null,"nextBlock":"b-64d42ba7b1d128e6ad1fee81","prerequisites":[{"id":"b-467c4116d3628bc8d44e181c","url":"#b-467c4116d3628bc8d44e181c","role":"problem-context"}],"relatedBlocks":[{"id":"b-3091ca8953fdbb5607f75e0e","url":"#b-3091ca8953fdbb5607f75e0e","relation":"figure-and-explanation"}]},{"id":"b-64d42ba7b1d128e6ad1fee81","type":"h2","scope":"explanation","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p64.html#b-64d42ba7b1d128e6ad1fee81","content":"## 解释、算例与证据","sha256":"d71cf15e707c8a3674b4c90a599efee783cb84da5b8c8236a1d01c3add7b4958","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"llava-visual-input-space","title":"把视觉向量接进语言模型的输入空间","number":1},"sectionTitle":"把视觉向量接进语言模型的输入空间","currentQuestion":"接口、更新参数和指令事实各自需要怎样检查？","sourceSlides":["l3-r2-p64"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-a2850e6fb502a02125169a27","nextBlock":"b-3216fb14b5c7ef5f644507a1","prerequisites":[{"id":"b-a2850e6fb502a02125169a27","url":"#b-a2850e6fb502a02125169a27","role":"section-title"},{"id":"b-467c4116d3628bc8d44e181c","url":"#b-467c4116d3628bc8d44e181c","role":"problem-context"}],"relatedBlocks":[{"id":"b-3091ca8953fdbb5607f75e0e","url":"#b-3091ca8953fdbb5607f75e0e","relation":"figure-and-explanation"}]},{"id":"b-3216fb14b5c7ef5f644507a1","type":"h1","scope":"document","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p64.html#b-3216fb14b5c7ef5f644507a1","content":"# P64 · 把视觉表示接入一个已经会说话的模型","sha256":"34a5aafff638810df558395242f28c04ea74208cca2f033a62ae9143b5186e17","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"llava-visual-input-space","title":"把视觉向量接进语言模型的输入空间","number":1},"sectionTitle":"把视觉向量接进语言模型的输入空间","currentQuestion":"接口、更新参数和指令事实各自需要怎样检查？","sourceSlides":["l3-r2-p64"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-64d42ba7b1d128e6ad1fee81","nextBlock":"b-467c4116d3628bc8d44e181c","prerequisites":[{"id":"b-64d42ba7b1d128e6ad1fee81","url":"#b-64d42ba7b1d128e6ad1fee81","role":"section-title"},{"id":"b-467c4116d3628bc8d44e181c","url":"#b-467c4116d3628bc8d44e181c","role":"problem-context"}],"relatedBlocks":[{"id":"b-3091ca8953fdbb5607f75e0e","url":"#b-3091ca8953fdbb5607f75e0e","relation":"figure-and-explanation"}]},{"id":"b-467c4116d3628bc8d44e181c","type":"p","scope":"document","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p64.html#b-467c4116d3628bc8d44e181c","content":"原始 LLaVA 输入的是视觉网格向量，经线性投影接入已有语言模型，不是一个类别标签或相似度。尺寸兼容使计算可进行，任务训练才使视觉信息可被使用。先辨认这条信息流，内部序列架构留第五讲。","sha256":"552d872bb02f5cdd79082b78c8e710e84ab1fa3d0b3c18fc2d2bb2a3f7a39796","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"llava-visual-input-space","title":"把视觉向量接进语言模型的输入空间","number":1},"sectionTitle":"把视觉向量接进语言模型的输入空间","currentQuestion":"接口、更新参数和指令事实各自需要怎样检查？","sourceSlides":["l3-r2-p64"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-3216fb14b5c7ef5f644507a1","nextBlock":"b-3091ca8953fdbb5607f75e0e","prerequisites":[{"id":"b-3216fb14b5c7ef5f644507a1","url":"#b-3216fb14b5c7ef5f644507a1","role":"section-title"}],"relatedBlocks":[{"id":"b-3091ca8953fdbb5607f75e0e","url":"#b-3091ca8953fdbb5607f75e0e","relation":"figure-and-explanation"}]},{"id":"b-3091ca8953fdbb5607f75e0e","type":"figure","scope":"document","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p64.html#b-3091ca8953fdbb5607f75e0e","content":"图内标签与数值：CLIP 视觉编码器；原始 LLaVA：ViT-L/14；网格特征；线性投影 W；视觉向量 → 语言维度；文字指令：；“谁拿着球？”；语言模型 Vicuna；视觉向量 + 文字指令；回答；“红衣人”；接入的是一组视觉特征，而非一个 CLIP 匹配分数；原始 LLaVA §4.1；网格数量为示意。参考回答用于解释接口，未作本轮视觉指令推理。\n\n图像经CLIP视觉网格特征和线性投影进入Vicuna，与文字指令共同产生参考回答。\n\n- CLIP 视觉编码器\n- 原始 LLaVA：ViT-L/14\n- 网格特征\n- 线性投影 W\n- 视觉向量 → 语言维度\n- 文字指令：\n- “谁拿着球？”\n- 语言模型 Vicuna\n- 视觉向量 + 文字指令\n- 回答\n- “红衣人”\n- 接入的是一组视觉特征，而非一个 CLIP 匹配分数\n- 原始 LLaVA §4.1；网格数量为示意。参考回答用于解释接口，未作本轮视觉指令推理。图像经CLIP视觉网格特征和线性投影进入Vicuna，与文字指令共同产生参考回答。 [查看对应图解状态](/#/l3-r2-p64/0)[查看完整图解](/course/figures/l3-r2-p64-0.html)","sha256":"98a8e02e443e0b1e637aa920bf74e45447c3e3fddf6aed5251304fe2356c41b7","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"llava-visual-input-space","title":"把视觉向量接进语言模型的输入空间","number":1},"sectionTitle":"把视觉向量接进语言模型的输入空间","currentQuestion":"接口、更新参数和指令事实各自需要怎样检查？","sourceSlides":["l3-r2-p64"],"knowledgeBlock":null,"formulas":[],"figure":{"caption":"图像经CLIP视觉网格特征和线性投影进入Vicuna，与文字指令共同产生参考回答。 查看对应图解状态","labels":["CLIP 视觉编码器","原始 LLaVA：ViT-L/14","网格特征","线性投影 W","视觉向量 → 语言维度","文字指令：","“谁拿着球？”","语言模型 Vicuna","视觉向量 + 文字指令","回答","“红衣人”","接入的是一组视觉特征，而非一个 CLIP 匹配分数","原始 LLaVA §4.1；网格数量为示意。参考回答用于解释接口，未作本轮视觉指令推理。"],"description":"图像经CLIP视觉网格特征和线性投影进入Vicuna，与文字指令共同产生参考回答。CLIP 视觉编码器原始 LLaVA：ViT-L/14网格特征线性投影 W视觉向量 → 语言维度文字指令：“谁拿着球？”语言模型 Vicuna视觉向量 + 文字指令回答“红衣人”接入的是一组视觉特征，而非一个 CLIP 匹配分数原始 LLaVA §4.1；网格数量为示意。参考回答用于解释接口，未作本轮视觉指令推理。","evidenceIdentity":"图像经CLIP视觉网格特征和线性投影进入Vicuna，与文字指令共同产生参考回答。 查看对应图解状态","images":[{"src":"/assets/display/512610a019c560ff0384e86a.webp","alt":""}]},"previousBlock":"b-467c4116d3628bc8d44e181c","nextBlock":"b-004e07342853103d5b6fb1ad","prerequisites":[{"id":"b-3216fb14b5c7ef5f644507a1","url":"#b-3216fb14b5c7ef5f644507a1","role":"section-title"},{"id":"b-467c4116d3628bc8d44e181c","url":"#b-467c4116d3628bc8d44e181c","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-004e07342853103d5b6fb1ad","type":"h2","scope":"narrative","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p64.html#b-004e07342853103d5b6fb1ad","content":"## 课堂讲述","sha256":"786ce10a6b32a492259a3dccd0a54e61cd41d28a34382fb7b5386c5f03b7ad69","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"llava-visual-input-space","title":"把视觉向量接进语言模型的输入空间","number":1},"sectionTitle":"把视觉向量接进语言模型的输入空间","currentQuestion":"接口、更新参数和指令事实各自需要怎样检查？","sourceSlides":["l3-r2-p64"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-3091ca8953fdbb5607f75e0e","nextBlock":"b-681249dc9d3b2c9063b4a23f","prerequisites":[{"id":"b-3216fb14b5c7ef5f644507a1","url":"#b-3216fb14b5c7ef5f644507a1","role":"section-title"},{"id":"b-467c4116d3628bc8d44e181c","url":"#b-467c4116d3628bc8d44e181c","role":"problem-context"}],"relatedBlocks":[{"id":"b-3091ca8953fdbb5607f75e0e","url":"#b-3091ca8953fdbb5607f75e0e","relation":"figure-and-explanation"}]},{"id":"b-681249dc9d3b2c9063b4a23f","type":"p","scope":"narrative","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p64.html#b-681249dc9d3b2c9063b4a23f","content":"以原始 LLaVA 为例，视觉编码器提供一组图像特征，经可训练的投影后，与文字输入共同进入语言模型。这里利用的是已有视觉模型与语言模型的能力，再建立一个新的模态接口。\n请看图像信息走到了哪里。输入不是一个“图里大概有猫”的标签，也不是一项 CLIP 相似度分数，而是一组能够被后续模型使用的视觉向量。投影调整了表示接口，但维度对齐并不自动等于任务已经学会。\n今天需要理解这条模块关系，语言模型内部如何组织上下文留到第五讲。眼前更关键的问题是：这几个模块接起来以后，究竟训练了谁，用什么数据让它根据图像和指令作答？","sha256":"be34c1fa3743f31ab28eca11b501b037ca00654cc62c41553e11de82c0d627a0","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"llava-visual-input-space","title":"把视觉向量接进语言模型的输入空间","number":1},"sectionTitle":"把视觉向量接进语言模型的输入空间","currentQuestion":"接口、更新参数和指令事实各自需要怎样检查？","sourceSlides":["l3-r2-p64"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-004e07342853103d5b6fb1ad","nextBlock":"b-5d98c63c33bad5dd3139029e","prerequisites":[{"id":"b-004e07342853103d5b6fb1ad","url":"#b-004e07342853103d5b6fb1ad","role":"section-title"},{"id":"b-467c4116d3628bc8d44e181c","url":"#b-467c4116d3628bc8d44e181c","role":"problem-context"}],"relatedBlocks":[{"id":"b-3091ca8953fdbb5607f75e0e","url":"#b-3091ca8953fdbb5607f75e0e","relation":"figure-and-explanation"}]},{"id":"b-5d98c63c33bad5dd3139029e","type":"h3","scope":"llava-visual-vector-interface","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p64.html#b-5d98c63c33bad5dd3139029e","content":"### LLaVA 的连接层：把视觉向量接入已有语言空间","sha256":"a807160b70a9bc7c59c8b86af1c74818348d1292fcbb8126cff471d4a6fd9905","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"llava-visual-input-space","title":"把视觉向量接进语言模型的输入空间","number":1},"sectionTitle":"LLaVA 的连接层：把视觉向量接入已有语言空间","currentQuestion":"接口、更新参数和指令事实各自需要怎样检查？","sourceSlides":["l3-r2-p64"],"knowledgeBlock":"llava-visual-vector-interface","formulas":[],"previousBlock":"b-681249dc9d3b2c9063b4a23f","nextBlock":"b-e0d54b750613f7f80e96847f","prerequisites":[{"id":"b-004e07342853103d5b6fb1ad","url":"#b-004e07342853103d5b6fb1ad","role":"section-title"},{"id":"b-467c4116d3628bc8d44e181c","url":"#b-467c4116d3628bc8d44e181c","role":"problem-context"}],"relatedBlocks":[{"id":"b-3091ca8953fdbb5607f75e0e","url":"#b-3091ca8953fdbb5607f75e0e","relation":"figure-and-explanation"},{"id":"b-e0d54b750613f7f80e96847f","url":"#b-e0d54b750613f7f80e96847f","relation":"same-reasoning"},{"id":"b-f71342b8bd5c2f77541969a1","url":"#b-f71342b8bd5c2f77541969a1","relation":"same-reasoning"},{"id":"b-16fadf7e0f211f780dac81c1","url":"#b-16fadf7e0f211f780dac81c1","relation":"same-reasoning"},{"id":"b-d38d603f7d85be1f3e32811d","url":"#b-d38d603f7d85be1f3e32811d","relation":"same-reasoning"}]},{"id":"b-e0d54b750613f7f80e96847f","type":"p","scope":"llava-visual-vector-interface","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p64.html#b-e0d54b750613f7f80e96847f","content":"原始 LLaVA 采用 CLIP ViT-L/14 的网格视觉特征，经一个可训练线性投影接入 Vicuna。若视觉特征矩阵按行写为 $Z_v\\in\\mathbb R^{n\\times d_v}$，连接权重为 $W\\in\\mathbb R^{d_v\\times d_l}$，则 $H_v=Z_vW$ 有语言输入维度 $d_l$。不同论文的行列约定可以转置，但信息流相同。","sha256":"301ef52026ba9ef8f1676a239e86372ff81653d9c1842aaf0dd14f01815b352d","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"llava-visual-input-space","title":"把视觉向量接进语言模型的输入空间","number":1},"sectionTitle":"LLaVA 的连接层：把视觉向量接入已有语言空间","currentQuestion":"接口、更新参数和指令事实各自需要怎样检查？","sourceSlides":["l3-r2-p64"],"knowledgeBlock":"llava-visual-vector-interface","formulas":[{"latex":"Z_v\\in\\mathbb R^{n\\times d_v}","definitionContext":"原始 LLaVA 采用 CLIP ViT-L/14 的网格视觉特征，经一个可训练线性投影接入 Vicuna。若视觉特征矩阵按行写为 $Z_v\\in\\mathbb R^{n\\times d_v}$，连接权重为 $W\\in\\mathbb R^{d_v\\times d_l}$，则 $H_v=Z_vW$ 有语言输入维度 $d_l$。不同论文的行列约定可以转置，但信息流相同。\n传进去的是一组视觉向量，不是“图中有猫”的一个标签，也不是 CLIP 的一项整图余弦。保留多个位置的特征，使后续模块有机会读取不同内容。线性投影调整接口，没有凭空增加图像中已被前处理丢失的细节。\nCLIP 已有图文比较空间，为什么还要训练连接？因为 Vicuna 的词向量坐标和任务过程不是 CLIP 文本编码器的空间。向量维度对上只能保证输入合法；模型还要学会把这些输入用于按图回答。一个数学类比是两个测量仪器都输出三维数值，但坐标系、单位与使用规则不同，不能直接逐项相加。\n本页只要求辨认视觉编码、投影、文字条件和回答输出四个位置。语言模型如何在序列内部处理上下文、视觉 token 怎样参与 attention，留第五讲。当前页面的格子数量是示意，不代表真实网格 token 数；示范回答也不是本轮 LLaVA 推理。"},{"latex":"W\\in\\mathbb R^{d_v\\times d_l}","definitionContext":"原始 LLaVA 采用 CLIP ViT-L/14 的网格视觉特征，经一个可训练线性投影接入 Vicuna。若视觉特征矩阵按行写为 $Z_v\\in\\mathbb R^{n\\times d_v}$，连接权重为 $W\\in\\mathbb R^{d_v\\times d_l}$，则 $H_v=Z_vW$ 有语言输入维度 $d_l$。不同论文的行列约定可以转置，但信息流相同。\n传进去的是一组视觉向量，不是“图中有猫”的一个标签，也不是 CLIP 的一项整图余弦。保留多个位置的特征，使后续模块有机会读取不同内容。线性投影调整接口，没有凭空增加图像中已被前处理丢失的细节。\nCLIP 已有图文比较空间，为什么还要训练连接？因为 Vicuna 的词向量坐标和任务过程不是 CLIP 文本编码器的空间。向量维度对上只能保证输入合法；模型还要学会把这些输入用于按图回答。一个数学类比是两个测量仪器都输出三维数值，但坐标系、单位与使用规则不同，不能直接逐项相加。\n本页只要求辨认视觉编码、投影、文字条件和回答输出四个位置。语言模型如何在序列内部处理上下文、视觉 token 怎样参与 attention，留第五讲。当前页面的格子数量是示意，不代表真实网格 token 数；示范回答也不是本轮 LLaVA 推理。"},{"latex":"H_v=Z_vW","definitionContext":"原始 LLaVA 采用 CLIP ViT-L/14 的网格视觉特征，经一个可训练线性投影接入 Vicuna。若视觉特征矩阵按行写为 $Z_v\\in\\mathbb R^{n\\times d_v}$，连接权重为 $W\\in\\mathbb R^{d_v\\times d_l}$，则 $H_v=Z_vW$ 有语言输入维度 $d_l$。不同论文的行列约定可以转置，但信息流相同。\n传进去的是一组视觉向量，不是“图中有猫”的一个标签，也不是 CLIP 的一项整图余弦。保留多个位置的特征，使后续模块有机会读取不同内容。线性投影调整接口，没有凭空增加图像中已被前处理丢失的细节。\nCLIP 已有图文比较空间，为什么还要训练连接？因为 Vicuna 的词向量坐标和任务过程不是 CLIP 文本编码器的空间。向量维度对上只能保证输入合法；模型还要学会把这些输入用于按图回答。一个数学类比是两个测量仪器都输出三维数值，但坐标系、单位与使用规则不同，不能直接逐项相加。\n本页只要求辨认视觉编码、投影、文字条件和回答输出四个位置。语言模型如何在序列内部处理上下文、视觉 token 怎样参与 attention，留第五讲。当前页面的格子数量是示意，不代表真实网格 token 数；示范回答也不是本轮 LLaVA 推理。"},{"latex":"d_l","definitionContext":"原始 LLaVA 采用 CLIP ViT-L/14 的网格视觉特征，经一个可训练线性投影接入 Vicuna。若视觉特征矩阵按行写为 $Z_v\\in\\mathbb R^{n\\times d_v}$，连接权重为 $W\\in\\mathbb R^{d_v\\times d_l}$，则 $H_v=Z_vW$ 有语言输入维度 $d_l$。不同论文的行列约定可以转置，但信息流相同。\n传进去的是一组视觉向量，不是“图中有猫”的一个标签，也不是 CLIP 的一项整图余弦。保留多个位置的特征，使后续模块有机会读取不同内容。线性投影调整接口，没有凭空增加图像中已被前处理丢失的细节。\nCLIP 已有图文比较空间，为什么还要训练连接？因为 Vicuna 的词向量坐标和任务过程不是 CLIP 文本编码器的空间。向量维度对上只能保证输入合法；模型还要学会把这些输入用于按图回答。一个数学类比是两个测量仪器都输出三维数值，但坐标系、单位与使用规则不同，不能直接逐项相加。\n本页只要求辨认视觉编码、投影、文字条件和回答输出四个位置。语言模型如何在序列内部处理上下文、视觉 token 怎样参与 attention，留第五讲。当前页面的格子数量是示意，不代表真实网格 token 数；示范回答也不是本轮 LLaVA 推理。"}],"previousBlock":"b-5d98c63c33bad5dd3139029e","nextBlock":"b-f71342b8bd5c2f77541969a1","prerequisites":[{"id":"b-5d98c63c33bad5dd3139029e","url":"#b-5d98c63c33bad5dd3139029e","role":"section-title"},{"id":"b-467c4116d3628bc8d44e181c","url":"#b-467c4116d3628bc8d44e181c","role":"problem-context"}],"relatedBlocks":[{"id":"b-3091ca8953fdbb5607f75e0e","url":"#b-3091ca8953fdbb5607f75e0e","relation":"figure-and-explanation"},{"id":"b-5d98c63c33bad5dd3139029e","url":"#b-5d98c63c33bad5dd3139029e","relation":"same-reasoning"},{"id":"b-f71342b8bd5c2f77541969a1","url":"#b-f71342b8bd5c2f77541969a1","relation":"same-reasoning"},{"id":"b-16fadf7e0f211f780dac81c1","url":"#b-16fadf7e0f211f780dac81c1","relation":"same-reasoning"},{"id":"b-d38d603f7d85be1f3e32811d","url":"#b-d38d603f7d85be1f3e32811d","relation":"same-reasoning"}]},{"id":"b-f71342b8bd5c2f77541969a1","type":"p","scope":"llava-visual-vector-interface","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p64.html#b-f71342b8bd5c2f77541969a1","content":"传进去的是一组视觉向量，不是“图中有猫”的一个标签，也不是 CLIP 的一项整图余弦。保留多个位置的特征，使后续模块有机会读取不同内容。线性投影调整接口，没有凭空增加图像中已被前处理丢失的细节。","sha256":"2ae95b830ed5e768a85b88c8740a6bf11f614e3ee1ca7afdb7806bc33dfcfd26","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"llava-visual-input-space","title":"把视觉向量接进语言模型的输入空间","number":1},"sectionTitle":"LLaVA 的连接层：把视觉向量接入已有语言空间","currentQuestion":"接口、更新参数和指令事实各自需要怎样检查？","sourceSlides":["l3-r2-p64"],"knowledgeBlock":"llava-visual-vector-interface","formulas":[],"previousBlock":"b-e0d54b750613f7f80e96847f","nextBlock":"b-16fadf7e0f211f780dac81c1","prerequisites":[{"id":"b-5d98c63c33bad5dd3139029e","url":"#b-5d98c63c33bad5dd3139029e","role":"section-title"},{"id":"b-467c4116d3628bc8d44e181c","url":"#b-467c4116d3628bc8d44e181c","role":"problem-context"}],"relatedBlocks":[{"id":"b-3091ca8953fdbb5607f75e0e","url":"#b-3091ca8953fdbb5607f75e0e","relation":"figure-and-explanation"},{"id":"b-5d98c63c33bad5dd3139029e","url":"#b-5d98c63c33bad5dd3139029e","relation":"same-reasoning"},{"id":"b-e0d54b750613f7f80e96847f","url":"#b-e0d54b750613f7f80e96847f","relation":"same-reasoning"},{"id":"b-16fadf7e0f211f780dac81c1","url":"#b-16fadf7e0f211f780dac81c1","relation":"same-reasoning"},{"id":"b-d38d603f7d85be1f3e32811d","url":"#b-d38d603f7d85be1f3e32811d","relation":"same-reasoning"}]},{"id":"b-16fadf7e0f211f780dac81c1","type":"p","scope":"llava-visual-vector-interface","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p64.html#b-16fadf7e0f211f780dac81c1","content":"CLIP 已有图文比较空间，为什么还要训练连接？因为 Vicuna 的词向量坐标和任务过程不是 CLIP 文本编码器的空间。向量维度对上只能保证输入合法；模型还要学会把这些输入用于按图回答。一个数学类比是两个测量仪器都输出三维数值，但坐标系、单位与使用规则不同，不能直接逐项相加。","sha256":"65181fc527d112fe12214386cf7e5d56ced11acbdbf8c551a01100af82575c3e","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"llava-visual-input-space","title":"把视觉向量接进语言模型的输入空间","number":1},"sectionTitle":"LLaVA 的连接层：把视觉向量接入已有语言空间","currentQuestion":"接口、更新参数和指令事实各自需要怎样检查？","sourceSlides":["l3-r2-p64"],"knowledgeBlock":"llava-visual-vector-interface","formulas":[],"previousBlock":"b-f71342b8bd5c2f77541969a1","nextBlock":"b-d38d603f7d85be1f3e32811d","prerequisites":[{"id":"b-5d98c63c33bad5dd3139029e","url":"#b-5d98c63c33bad5dd3139029e","role":"section-title"},{"id":"b-467c4116d3628bc8d44e181c","url":"#b-467c4116d3628bc8d44e181c","role":"problem-context"}],"relatedBlocks":[{"id":"b-3091ca8953fdbb5607f75e0e","url":"#b-3091ca8953fdbb5607f75e0e","relation":"figure-and-explanation"},{"id":"b-5d98c63c33bad5dd3139029e","url":"#b-5d98c63c33bad5dd3139029e","relation":"same-reasoning"},{"id":"b-e0d54b750613f7f80e96847f","url":"#b-e0d54b750613f7f80e96847f","relation":"same-reasoning"},{"id":"b-f71342b8bd5c2f77541969a1","url":"#b-f71342b8bd5c2f77541969a1","relation":"same-reasoning"},{"id":"b-d38d603f7d85be1f3e32811d","url":"#b-d38d603f7d85be1f3e32811d","relation":"same-reasoning"}]},{"id":"b-d38d603f7d85be1f3e32811d","type":"p","scope":"llava-visual-vector-interface","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p64.html#b-d38d603f7d85be1f3e32811d","content":"本页只要求辨认视觉编码、投影、文字条件和回答输出四个位置。语言模型如何在序列内部处理上下文、视觉 token 怎样参与 attention，留第五讲。当前页面的格子数量是示意，不代表真实网格 token 数；示范回答也不是本轮 LLaVA 推理。","sha256":"7b9949dc915b90d1aedf1cb20cd82efb100faee64c980c73912c0b809a3fdb0a","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"llava-visual-input-space","title":"把视觉向量接进语言模型的输入空间","number":1},"sectionTitle":"LLaVA 的连接层：把视觉向量接入已有语言空间","currentQuestion":"接口、更新参数和指令事实各自需要怎样检查？","sourceSlides":["l3-r2-p64"],"knowledgeBlock":"llava-visual-vector-interface","formulas":[],"previousBlock":"b-16fadf7e0f211f780dac81c1","nextBlock":"b-ea32dffe856067aeadfb5467","prerequisites":[{"id":"b-5d98c63c33bad5dd3139029e","url":"#b-5d98c63c33bad5dd3139029e","role":"section-title"},{"id":"b-467c4116d3628bc8d44e181c","url":"#b-467c4116d3628bc8d44e181c","role":"problem-context"}],"relatedBlocks":[{"id":"b-3091ca8953fdbb5607f75e0e","url":"#b-3091ca8953fdbb5607f75e0e","relation":"figure-and-explanation"},{"id":"b-5d98c63c33bad5dd3139029e","url":"#b-5d98c63c33bad5dd3139029e","relation":"same-reasoning"},{"id":"b-e0d54b750613f7f80e96847f","url":"#b-e0d54b750613f7f80e96847f","relation":"same-reasoning"},{"id":"b-f71342b8bd5c2f77541969a1","url":"#b-f71342b8bd5c2f77541969a1","relation":"same-reasoning"},{"id":"b-16fadf7e0f211f780dac81c1","url":"#b-16fadf7e0f211f780dac81c1","relation":"same-reasoning"}]},{"id":"b-ea32dffe856067aeadfb5467","type":"h3","scope":"dialogue","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p64.html#b-ea32dffe856067aeadfb5467","content":"### 沿着图解追问","sha256":"f16be0424762a44344338696f06ef266c4571d15367ae2cbbd28fac98ac7e9cb","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"llava-visual-input-space","title":"把视觉向量接进语言模型的输入空间","number":1},"sectionTitle":"把视觉向量接进语言模型的输入空间","currentQuestion":"接口、更新参数和指令事实各自需要怎样检查？","sourceSlides":["l3-r2-p64"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-d38d603f7d85be1f3e32811d","nextBlock":"b-5745b27abbeede2588a307ce","prerequisites":[{"id":"b-5d98c63c33bad5dd3139029e","url":"#b-5d98c63c33bad5dd3139029e","role":"section-title"},{"id":"b-467c4116d3628bc8d44e181c","url":"#b-467c4116d3628bc8d44e181c","role":"problem-context"}],"relatedBlocks":[{"id":"b-3091ca8953fdbb5607f75e0e","url":"#b-3091ca8953fdbb5607f75e0e","relation":"figure-and-explanation"}]},{"id":"b-5745b27abbeede2588a307ce","type":"p","scope":"dialogue","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p64.html#b-5745b27abbeede2588a307ce","content":"以下是备课用的模拟师生对话，不是课堂实录。","sha256":"b25685f04cd1b885f3a5999e6a912460c7424021cdc5b2190584242086a82681","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"llava-visual-input-space","title":"把视觉向量接进语言模型的输入空间","number":1},"sectionTitle":"把视觉向量接进语言模型的输入空间","currentQuestion":"接口、更新参数和指令事实各自需要怎样检查？","sourceSlides":["l3-r2-p64"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-ea32dffe856067aeadfb5467","nextBlock":"b-f8d1304f1fc33cb150d72aeb","prerequisites":[{"id":"b-ea32dffe856067aeadfb5467","url":"#b-ea32dffe856067aeadfb5467","role":"section-title"},{"id":"b-467c4116d3628bc8d44e181c","url":"#b-467c4116d3628bc8d44e181c","role":"problem-context"}],"relatedBlocks":[{"id":"b-3091ca8953fdbb5607f75e0e","url":"#b-3091ca8953fdbb5607f75e0e","relation":"figure-and-explanation"}]},{"id":"b-f8d1304f1fc33cb150d72aeb","type":"p","scope":"dialogue","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p64.html#b-f8d1304f1fc33cb150d72aeb","content":"**学生：**CLIP 已经对齐图文，为什么再投影？","sha256":"acc9219b1ee4d733d601fd38a128e3812bea258ce14e524e4ee8e10a844073b8","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"llava-visual-input-space","title":"把视觉向量接进语言模型的输入空间","number":1},"sectionTitle":"把视觉向量接进语言模型的输入空间","currentQuestion":"接口、更新参数和指令事实各自需要怎样检查？","sourceSlides":["l3-r2-p64"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-5745b27abbeede2588a307ce","nextBlock":"b-3915726fa3cebb0b8345c869","prerequisites":[{"id":"b-ea32dffe856067aeadfb5467","url":"#b-ea32dffe856067aeadfb5467","role":"section-title"},{"id":"b-467c4116d3628bc8d44e181c","url":"#b-467c4116d3628bc8d44e181c","role":"problem-context"}],"relatedBlocks":[{"id":"b-3091ca8953fdbb5607f75e0e","url":"#b-3091ca8953fdbb5607f75e0e","relation":"figure-and-explanation"}]},{"id":"b-3915726fa3cebb0b8345c869","type":"p","scope":"dialogue","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p64.html#b-3915726fa3cebb0b8345c869","content":"**教师：**那是 CLIP 自己的比较空间。另一个语言模型有自己的向量坐标和使用任务，投影建立接口，后续训练教它按图像和指令使用信息。","sha256":"b716e0bb51bbe9c6192e94908863a1da81fb538e4d0587397110a0e70a5d2c18","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"llava-visual-input-space","title":"把视觉向量接进语言模型的输入空间","number":1},"sectionTitle":"把视觉向量接进语言模型的输入空间","currentQuestion":"接口、更新参数和指令事实各自需要怎样检查？","sourceSlides":["l3-r2-p64"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-f8d1304f1fc33cb150d72aeb","nextBlock":"b-96a6bc64988fc14fc1f83162","prerequisites":[{"id":"b-ea32dffe856067aeadfb5467","url":"#b-ea32dffe856067aeadfb5467","role":"section-title"},{"id":"b-467c4116d3628bc8d44e181c","url":"#b-467c4116d3628bc8d44e181c","role":"problem-context"}],"relatedBlocks":[{"id":"b-3091ca8953fdbb5607f75e0e","url":"#b-3091ca8953fdbb5607f75e0e","relation":"figure-and-explanation"}]},{"id":"b-96a6bc64988fc14fc1f83162","type":"h2","scope":"practice","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p64.html#b-96a6bc64988fc14fc1f83162","content":"## 沿这个问题继续","sha256":"cf44fcc62782fb4f6bdc42c16c5187d57556a03cb7ce6065e7bd9c36d8e7d733","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"llava-visual-input-space","title":"把视觉向量接进语言模型的输入空间","number":1},"sectionTitle":"把视觉向量接进语言模型的输入空间","currentQuestion":"接口、更新参数和指令事实各自需要怎样检查？","sourceSlides":["l3-r2-p64"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-3915726fa3cebb0b8345c869","nextBlock":"b-5b2a37ac93dbb4c154b0a8d8","prerequisites":[{"id":"b-ea32dffe856067aeadfb5467","url":"#b-ea32dffe856067aeadfb5467","role":"section-title"},{"id":"b-467c4116d3628bc8d44e181c","url":"#b-467c4116d3628bc8d44e181c","role":"problem-context"}],"relatedBlocks":[{"id":"b-3091ca8953fdbb5607f75e0e","url":"#b-3091ca8953fdbb5607f75e0e","relation":"figure-and-explanation"}]},{"id":"b-5b2a37ac93dbb4c154b0a8d8","type":"p","scope":"practice","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p64.html#b-5b2a37ac93dbb4c154b0a8d8","content":"沿视觉网格、线性投影和语言模型三个接口检查张量形状：图块数保持不变，向量宽度改变。然后到 P65 比较两个训练阶段分别更新哪些参数。","sha256":"f088f2fefbfd65eef49998b5eae7aba924db9bbc324feed9a67dafd030c9fd29","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"llava-visual-input-space","title":"把视觉向量接进语言模型的输入空间","number":1},"sectionTitle":"把视觉向量接进语言模型的输入空间","currentQuestion":"接口、更新参数和指令事实各自需要怎样检查？","sourceSlides":["l3-r2-p64"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-96a6bc64988fc14fc1f83162","nextBlock":"b-ebeed24dfa2706ee2f829ad6","prerequisites":[{"id":"b-96a6bc64988fc14fc1f83162","url":"#b-96a6bc64988fc14fc1f83162","role":"section-title"},{"id":"b-467c4116d3628bc8d44e181c","url":"#b-467c4116d3628bc8d44e181c","role":"problem-context"}],"relatedBlocks":[{"id":"b-3091ca8953fdbb5607f75e0e","url":"#b-3091ca8953fdbb5607f75e0e","relation":"figure-and-explanation"}]},{"id":"b-ebeed24dfa2706ee2f829ad6","type":"h4","scope":"llava-visual-input-space","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p64.html#b-ebeed24dfa2706ee2f829ad6","content":"#### 图示条件与公式约定","sha256":"ef7f915c1ae63ed67bd23fecf050249273ba0efa8965d2c8d1c1c6b2d3178122","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"llava-visual-input-space","title":"把视觉向量接进语言模型的输入空间","number":1},"sectionTitle":"把视觉向量接进语言模型的输入空间","currentQuestion":"接口、更新参数和指令事实各自需要怎样检查？","sourceSlides":["l3-r2-p64"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-5b2a37ac93dbb4c154b0a8d8","nextBlock":"b-2d0ac43161dbb263b8abe4f3","prerequisites":[{"id":"b-96a6bc64988fc14fc1f83162","url":"#b-96a6bc64988fc14fc1f83162","role":"section-title"},{"id":"b-467c4116d3628bc8d44e181c","url":"#b-467c4116d3628bc8d44e181c","role":"problem-context"}],"relatedBlocks":[{"id":"b-3091ca8953fdbb5607f75e0e","url":"#b-3091ca8953fdbb5607f75e0e","relation":"figure-and-explanation"}]},{"id":"b-2d0ac43161dbb263b8abe4f3","type":"p","scope":"document","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p64.html#b-2d0ac43161dbb263b8abe4f3","content":"原始 LLaVA 使用 CLIP ViT-L/14 的网格特征，线性投影接到 Vicuna 的文字输入维度。图中格数仅作表示，不对应实际 token 数量。示例回答为人工任务说明，不是本轮 LLaVA 运行。","sha256":"213fa2e9822c365027e4a475af9b2fa2b0021ec509f7b2582a5b6207ad19e2aa","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"llava-visual-input-space","title":"把视觉向量接进语言模型的输入空间","number":1},"sectionTitle":"把视觉向量接进语言模型的输入空间","currentQuestion":"接口、更新参数和指令事实各自需要怎样检查？","sourceSlides":["l3-r2-p64"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-ebeed24dfa2706ee2f829ad6","nextBlock":"b-d4d59b42f85f559a009488ea","prerequisites":[{"id":"b-ebeed24dfa2706ee2f829ad6","url":"#b-ebeed24dfa2706ee2f829ad6","role":"section-title"},{"id":"b-467c4116d3628bc8d44e181c","url":"#b-467c4116d3628bc8d44e181c","role":"problem-context"}],"relatedBlocks":[{"id":"b-3091ca8953fdbb5607f75e0e","url":"#b-3091ca8953fdbb5607f75e0e","relation":"figure-and-explanation"}]},{"id":"b-d4d59b42f85f559a009488ea","type":"h4","scope":"retained-reading","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p64.html#b-d4d59b42f85f559a009488ea","content":"#### 这条线索的展开与后续阅读","sha256":"48b8229f7559b36ed4d5628416e3991e5c73293fbce43374db67386c8736f3bc","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"llava-visual-input-space","title":"把视觉向量接进语言模型的输入空间","number":1},"sectionTitle":"把视觉向量接进语言模型的输入空间","currentQuestion":"接口、更新参数和指令事实各自需要怎样检查？","sourceSlides":["l3-r2-p64"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-2d0ac43161dbb263b8abe4f3","nextBlock":"b-2c05320afb108d4312c318f1","prerequisites":[{"id":"b-ebeed24dfa2706ee2f829ad6","url":"#b-ebeed24dfa2706ee2f829ad6","role":"section-title"},{"id":"b-467c4116d3628bc8d44e181c","url":"#b-467c4116d3628bc8d44e181c","role":"problem-context"}],"relatedBlocks":[{"id":"b-3091ca8953fdbb5607f75e0e","url":"#b-3091ca8953fdbb5607f75e0e","relation":"figure-and-explanation"}]},{"id":"b-2c05320afb108d4312c318f1","type":"ul","scope":"retained-reading","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p64.html#b-2c05320afb108d4312c318f1","content":"- [已有的语言模型，怎样接收一张图？](/supplement.html?return=l3-r2-p64/0#/l3-llava-interface/0) 本页说明\n- [视觉接入语言模型不是连一根线｜问题从哪里出现](/supplement.html?return=l3-r2-p64/0#/l3-multimodal-interface-problem/0) L03补充阅读\n- [视觉接入语言模型不是连一根线｜一起算一个完整例子](/supplement.html?return=l3-r2-p64/0#/l3-multimodal-interface-example/0) L03补充阅读\n- [视觉接入语言模型不是连一根线｜怎样检查这项解释](/supplement.html?return=l3-r2-p64/0#/l3-multimodal-interface-evidence/0) L03补充阅读","sha256":"411d351539fcfdcacb86e80edc3bab67ee079b470b76e821db3ebc1f5c47de7d","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"llava-visual-input-space","title":"把视觉向量接进语言模型的输入空间","number":1},"sectionTitle":"把视觉向量接进语言模型的输入空间","currentQuestion":"接口、更新参数和指令事实各自需要怎样检查？","sourceSlides":["l3-r2-p64"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-d4d59b42f85f559a009488ea","nextBlock":"b-3e144a15a1e28abeb1aab175","prerequisites":[{"id":"b-d4d59b42f85f559a009488ea","url":"#b-d4d59b42f85f559a009488ea","role":"section-title"},{"id":"b-467c4116d3628bc8d44e181c","url":"#b-467c4116d3628bc8d44e181c","role":"problem-context"}],"relatedBlocks":[{"id":"b-3091ca8953fdbb5607f75e0e","url":"#b-3091ca8953fdbb5607f75e0e","relation":"figure-and-explanation"}]},{"id":"b-3e144a15a1e28abeb1aab175","type":"li","scope":"retained-reading","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p64.html#b-3e144a15a1e28abeb1aab175","content":"- [已有的语言模型，怎样接收一张图？](/supplement.html?return=l3-r2-p64/0#/l3-llava-interface/0) 本页说明","sha256":"a759cb0c908e0148eb93db1ea192238384f1661f7cb102e234ea86e003512bb8","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"llava-visual-input-space","title":"把视觉向量接进语言模型的输入空间","number":1},"sectionTitle":"把视觉向量接进语言模型的输入空间","currentQuestion":"接口、更新参数和指令事实各自需要怎样检查？","sourceSlides":["l3-r2-p64"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-2c05320afb108d4312c318f1","nextBlock":"b-b60682ab95420e8a879aabe8","prerequisites":[{"id":"b-d4d59b42f85f559a009488ea","url":"#b-d4d59b42f85f559a009488ea","role":"section-title"},{"id":"b-467c4116d3628bc8d44e181c","url":"#b-467c4116d3628bc8d44e181c","role":"problem-context"}],"relatedBlocks":[{"id":"b-3091ca8953fdbb5607f75e0e","url":"#b-3091ca8953fdbb5607f75e0e","relation":"figure-and-explanation"}]},{"id":"b-b60682ab95420e8a879aabe8","type":"li","scope":"retained-reading","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p64.html#b-b60682ab95420e8a879aabe8","content":"- [视觉接入语言模型不是连一根线｜问题从哪里出现](/supplement.html?return=l3-r2-p64/0#/l3-multimodal-interface-problem/0) L03补充阅读","sha256":"f85b46274ba2ee756bb7f4aeb8de78fd29998022ed14b38fdf994540150b2a34","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"llava-visual-input-space","title":"把视觉向量接进语言模型的输入空间","number":1},"sectionTitle":"把视觉向量接进语言模型的输入空间","currentQuestion":"接口、更新参数和指令事实各自需要怎样检查？","sourceSlides":["l3-r2-p64"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-3e144a15a1e28abeb1aab175","nextBlock":"b-1f5729cb182fab620997ff45","prerequisites":[{"id":"b-d4d59b42f85f559a009488ea","url":"#b-d4d59b42f85f559a009488ea","role":"section-title"},{"id":"b-467c4116d3628bc8d44e181c","url":"#b-467c4116d3628bc8d44e181c","role":"problem-context"}],"relatedBlocks":[{"id":"b-3091ca8953fdbb5607f75e0e","url":"#b-3091ca8953fdbb5607f75e0e","relation":"figure-and-explanation"}]},{"id":"b-1f5729cb182fab620997ff45","type":"li","scope":"retained-reading","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p64.html#b-1f5729cb182fab620997ff45","content":"- [视觉接入语言模型不是连一根线｜一起算一个完整例子](/supplement.html?return=l3-r2-p64/0#/l3-multimodal-interface-example/0) L03补充阅读","sha256":"12c2a6de3684559337c91c8a21e079797f93c3c3a68e2c752cbab0b92e56fb46","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"llava-visual-input-space","title":"把视觉向量接进语言模型的输入空间","number":1},"sectionTitle":"把视觉向量接进语言模型的输入空间","currentQuestion":"接口、更新参数和指令事实各自需要怎样检查？","sourceSlides":["l3-r2-p64"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-b60682ab95420e8a879aabe8","nextBlock":"b-ad86d5c0e999571083e9eadc","prerequisites":[{"id":"b-d4d59b42f85f559a009488ea","url":"#b-d4d59b42f85f559a009488ea","role":"section-title"},{"id":"b-467c4116d3628bc8d44e181c","url":"#b-467c4116d3628bc8d44e181c","role":"problem-context"}],"relatedBlocks":[{"id":"b-3091ca8953fdbb5607f75e0e","url":"#b-3091ca8953fdbb5607f75e0e","relation":"figure-and-explanation"}]},{"id":"b-ad86d5c0e999571083e9eadc","type":"li","scope":"retained-reading","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p64.html#b-ad86d5c0e999571083e9eadc","content":"- [视觉接入语言模型不是连一根线｜怎样检查这项解释](/supplement.html?return=l3-r2-p64/0#/l3-multimodal-interface-evidence/0) L03补充阅读","sha256":"d8689dd3bd00bbd1e157ef9f2b0bcb0f2086e4c22ec90935d9fc6b77c72cc077","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"llava-visual-input-space","title":"把视觉向量接进语言模型的输入空间","number":1},"sectionTitle":"把视觉向量接进语言模型的输入空间","currentQuestion":"接口、更新参数和指令事实各自需要怎样检查？","sourceSlides":["l3-r2-p64"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-1f5729cb182fab620997ff45","nextBlock":null,"prerequisites":[{"id":"b-d4d59b42f85f559a009488ea","url":"#b-d4d59b42f85f559a009488ea","role":"section-title"},{"id":"b-467c4116d3628bc8d44e181c","url":"#b-467c4116d3628bc8d44e181c","role":"problem-context"}],"relatedBlocks":[{"id":"b-3091ca8953fdbb5607f75e0e","url":"#b-3091ca8953fdbb5607f75e0e","relation":"figure-and-explanation"}]}]}