{"version":2,"relationshipURLBase":"https://codingai-lec03.pages.dev/course/blocks/llava-visual-vector-interface.html","relationshipResolution":"Resolve relative relationship URLs against relationshipURLBase; complete records and absolute URLs are in blocks.","title":"LLaVA 的连接层：把视觉向量接入已有语言空间","url":"https://codingai-lec03.pages.dev/course/blocks/llava-visual-vector-interface.html","markdown":"https://codingai-lec03.pages.dev/course/blocks/llava-visual-vector-interface.md","pages":[{"id":"reading-opening","title":"LLaVA 的连接层：把视觉向量接入已有语言空间","unit":"可复用视觉与语言组件怎样学会共同回答？","unitId":"llava-interface-and-training","chapter":"III","number":1}],"aliases":[{"oldId":"b-f15cc400f416177ccaed477e","targetId":"b-f15cc400f416177ccaed477e","page":"reading-opening","operation":"move","beforeSHA256":"9bed69fca3dae74d26a8796a63140f42bba01d6780a7a37c7bda760e999b1065","afterSHA256":"9bed69fca3dae74d26a8796a63140f42bba01d6780a7a37c7bda760e999b1065","reason":"Same source body in authored semantic page"},{"oldId":"b-e80c77140932ec81b8f0e4ba","targetId":"b-e80c77140932ec81b8f0e4ba","page":"reading-opening","operation":"move","beforeSHA256":"552d872bb02f5cdd79082b78c8e710e84ab1fa3d0b3c18fc2d2bb2a3f7a39796","afterSHA256":"552d872bb02f5cdd79082b78c8e710e84ab1fa3d0b3c18fc2d2bb2a3f7a39796","reason":"Same source body in authored semantic page"},{"oldId":"b-837e3521c3004fabc2c59f2c","targetId":"b-837e3521c3004fabc2c59f2c","page":"reading-opening","operation":"move","beforeSHA256":"301ef52026ba9ef8f1676a239e86372ff81653d9c1842aaf0dd14f01815b352d","afterSHA256":"301ef52026ba9ef8f1676a239e86372ff81653d9c1842aaf0dd14f01815b352d","reason":"Same source body in authored semantic page"},{"oldId":"b-af3ba15a0602a379aca0022e","targetId":"b-af3ba15a0602a379aca0022e","page":"reading-opening","operation":"move","beforeSHA256":"2ae95b830ed5e768a85b88c8740a6bf11f614e3ee1ca7afdb7806bc33dfcfd26","afterSHA256":"2ae95b830ed5e768a85b88c8740a6bf11f614e3ee1ca7afdb7806bc33dfcfd26","reason":"Same source body in authored semantic page"},{"oldId":"b-ac6bae9cd4aa658d03e07760","targetId":"b-ac6bae9cd4aa658d03e07760","page":"reading-opening","operation":"move","beforeSHA256":"65181fc527d112fe12214386cf7e5d56ced11acbdbf8c551a01100af82575c3e","afterSHA256":"65181fc527d112fe12214386cf7e5d56ced11acbdbf8c551a01100af82575c3e","reason":"Same source body in authored semantic page"},{"oldId":"b-d6ba5ba246e109996814e86e","targetId":"b-d6ba5ba246e109996814e86e","page":"reading-opening","operation":"move","beforeSHA256":"7b9949dc915b90d1aedf1cb20cd82efb100faee64c980c73912c0b809a3fdb0a","afterSHA256":"7b9949dc915b90d1aedf1cb20cd82efb100faee64c980c73912c0b809a3fdb0a","reason":"Same source body in authored semantic page"},{"oldId":"b-d461d82c0c23d03a09aed338","targetId":"b-d461d82c0c23d03a09aed338","page":"reading-opening","operation":"rewrite","beforeSHA256":"2abc4815d1331400633e9b27cb5792f64001f1ee88af608a697ac38c28624490","afterSHA256":"85f80cce13ab159988b8e996e323b53a8b067d434699593f3760372b201c5d66","reason":"Same source identity with figure semantics or reading context enriched"},{"oldId":"b-9d6ee39a5691f0749b5eacb5","targetId":"b-9d6ee39a5691f0749b5eacb5","page":"reading-opening","operation":"move","beforeSHA256":"2b9ccef7e1220e2250cceff1abd8bdd736b42f8da5a223f26b029f44b962e807","afterSHA256":"2b9ccef7e1220e2250cceff1abd8bdd736b42f8da5a223f26b029f44b962e807","reason":"Same source body in authored semantic page"}],"blocks":[{"id":"b-f15cc400f416177ccaed477e","type":"h1","scope":"llava-visual-vector-interface","url":"https://codingai-lec03.pages.dev/course/blocks/llava-visual-vector-interface.html#b-f15cc400f416177ccaed477e","content":"# LLaVA 的连接层：把视觉向量接入已有语言空间","sha256":"9bed69fca3dae74d26a8796a63140f42bba01d6780a7a37c7bda760e999b1065","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"reading-opening","title":"LLaVA 的连接层：把视觉向量接入已有语言空间","number":1},"sectionTitle":"LLaVA 的连接层：把视觉向量接入已有语言空间","currentQuestion":"LLaVA 的连接层：把视觉向量接入已有语言空间","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":null,"nextBlock":"b-e80c77140932ec81b8f0e4ba","prerequisites":[{"id":"b-e80c77140932ec81b8f0e4ba","url":"#b-e80c77140932ec81b8f0e4ba","role":"problem-context"}],"relatedBlocks":[{"id":"b-d461d82c0c23d03a09aed338","url":"#b-d461d82c0c23d03a09aed338","relation":"figure-and-explanation"}]},{"id":"b-e80c77140932ec81b8f0e4ba","type":"p","scope":"llava-visual-vector-interface","url":"https://codingai-lec03.pages.dev/course/blocks/llava-visual-vector-interface.html#b-e80c77140932ec81b8f0e4ba","content":"原始 LLaVA 输入的是视觉网格向量，经线性投影接入已有语言模型，不是一个类别标签或相似度。尺寸兼容使计算可进行，任务训练才使视觉信息可被使用。先辨认这条信息流，内部序列架构留第五讲。","sha256":"552d872bb02f5cdd79082b78c8e710e84ab1fa3d0b3c18fc2d2bb2a3f7a39796","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"reading-opening","title":"LLaVA 的连接层：把视觉向量接入已有语言空间","number":1},"sectionTitle":"LLaVA 的连接层：把视觉向量接入已有语言空间","currentQuestion":"LLaVA 的连接层：把视觉向量接入已有语言空间","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-f15cc400f416177ccaed477e","nextBlock":"b-837e3521c3004fabc2c59f2c","prerequisites":[{"id":"b-f15cc400f416177ccaed477e","url":"#b-f15cc400f416177ccaed477e","role":"section-title"}],"relatedBlocks":[{"id":"b-d461d82c0c23d03a09aed338","url":"#b-d461d82c0c23d03a09aed338","relation":"figure-and-explanation"}]},{"id":"b-837e3521c3004fabc2c59f2c","type":"p","scope":"llava-visual-vector-interface","url":"https://codingai-lec03.pages.dev/course/blocks/llava-visual-vector-interface.html#b-837e3521c3004fabc2c59f2c","content":"原始 LLaVA 采用 CLIP ViT-L/14 的网格视觉特征，经一个可训练线性投影接入 Vicuna。若视觉特征矩阵按行写为 $Z_v\\in\\mathbb R^{n\\times d_v}$，连接权重为 $W\\in\\mathbb R^{d_v\\times d_l}$，则 $H_v=Z_vW$ 有语言输入维度 $d_l$。不同论文的行列约定可以转置，但信息流相同。","sha256":"301ef52026ba9ef8f1676a239e86372ff81653d9c1842aaf0dd14f01815b352d","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"reading-opening","title":"LLaVA 的连接层：把视觉向量接入已有语言空间","number":1},"sectionTitle":"LLaVA 的连接层：把视觉向量接入已有语言空间","currentQuestion":"LLaVA 的连接层：把视觉向量接入已有语言空间","sourceSlides":[],"knowledgeBlock":null,"formulas":[{"latex":"Z_v\\in\\mathbb R^{n\\times d_v}","definitionContext":"LLaVA 的连接层：把视觉向量接入已有语言空间"},{"latex":"W\\in\\mathbb R^{d_v\\times d_l}","definitionContext":"LLaVA 的连接层：把视觉向量接入已有语言空间"},{"latex":"H_v=Z_vW","definitionContext":"LLaVA 的连接层：把视觉向量接入已有语言空间"},{"latex":"d_l","definitionContext":"LLaVA 的连接层：把视觉向量接入已有语言空间"}],"previousBlock":"b-e80c77140932ec81b8f0e4ba","nextBlock":"b-af3ba15a0602a379aca0022e","prerequisites":[{"id":"b-f15cc400f416177ccaed477e","url":"#b-f15cc400f416177ccaed477e","role":"section-title"},{"id":"b-e80c77140932ec81b8f0e4ba","url":"#b-e80c77140932ec81b8f0e4ba","role":"problem-context"}],"relatedBlocks":[{"id":"b-d461d82c0c23d03a09aed338","url":"#b-d461d82c0c23d03a09aed338","relation":"figure-and-explanation"}]},{"id":"b-af3ba15a0602a379aca0022e","type":"p","scope":"llava-visual-vector-interface","url":"https://codingai-lec03.pages.dev/course/blocks/llava-visual-vector-interface.html#b-af3ba15a0602a379aca0022e","content":"传进去的是一组视觉向量，不是“图中有猫”的一个标签，也不是 CLIP 的一项整图余弦。保留多个位置的特征，使后续模块有机会读取不同内容。线性投影调整接口，没有凭空增加图像中已被前处理丢失的细节。","sha256":"2ae95b830ed5e768a85b88c8740a6bf11f614e3ee1ca7afdb7806bc33dfcfd26","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"reading-opening","title":"LLaVA 的连接层：把视觉向量接入已有语言空间","number":1},"sectionTitle":"LLaVA 的连接层：把视觉向量接入已有语言空间","currentQuestion":"LLaVA 的连接层：把视觉向量接入已有语言空间","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-837e3521c3004fabc2c59f2c","nextBlock":"b-ac6bae9cd4aa658d03e07760","prerequisites":[{"id":"b-f15cc400f416177ccaed477e","url":"#b-f15cc400f416177ccaed477e","role":"section-title"},{"id":"b-e80c77140932ec81b8f0e4ba","url":"#b-e80c77140932ec81b8f0e4ba","role":"problem-context"}],"relatedBlocks":[{"id":"b-d461d82c0c23d03a09aed338","url":"#b-d461d82c0c23d03a09aed338","relation":"figure-and-explanation"}]},{"id":"b-ac6bae9cd4aa658d03e07760","type":"p","scope":"llava-visual-vector-interface","url":"https://codingai-lec03.pages.dev/course/blocks/llava-visual-vector-interface.html#b-ac6bae9cd4aa658d03e07760","content":"CLIP 已有图文比较空间，为什么还要训练连接？因为 Vicuna 的词向量坐标和任务过程不是 CLIP 文本编码器的空间。向量维度对上只能保证输入合法；模型还要学会把这些输入用于按图回答。一个数学类比是两个测量仪器都输出三维数值，但坐标系、单位与使用规则不同，不能直接逐项相加。","sha256":"65181fc527d112fe12214386cf7e5d56ced11acbdbf8c551a01100af82575c3e","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"reading-opening","title":"LLaVA 的连接层：把视觉向量接入已有语言空间","number":1},"sectionTitle":"LLaVA 的连接层：把视觉向量接入已有语言空间","currentQuestion":"LLaVA 的连接层：把视觉向量接入已有语言空间","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-af3ba15a0602a379aca0022e","nextBlock":"b-d6ba5ba246e109996814e86e","prerequisites":[{"id":"b-f15cc400f416177ccaed477e","url":"#b-f15cc400f416177ccaed477e","role":"section-title"},{"id":"b-e80c77140932ec81b8f0e4ba","url":"#b-e80c77140932ec81b8f0e4ba","role":"problem-context"}],"relatedBlocks":[{"id":"b-d461d82c0c23d03a09aed338","url":"#b-d461d82c0c23d03a09aed338","relation":"figure-and-explanation"}]},{"id":"b-d6ba5ba246e109996814e86e","type":"p","scope":"llava-visual-vector-interface","url":"https://codingai-lec03.pages.dev/course/blocks/llava-visual-vector-interface.html#b-d6ba5ba246e109996814e86e","content":"本页只要求辨认视觉编码、投影、文字条件和回答输出四个位置。语言模型如何在序列内部处理上下文、视觉 token 怎样参与 attention，留第五讲。当前页面的格子数量是示意，不代表真实网格 token 数；示范回答也不是本轮 LLaVA 推理。","sha256":"7b9949dc915b90d1aedf1cb20cd82efb100faee64c980c73912c0b809a3fdb0a","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"reading-opening","title":"LLaVA 的连接层：把视觉向量接入已有语言空间","number":1},"sectionTitle":"LLaVA 的连接层：把视觉向量接入已有语言空间","currentQuestion":"LLaVA 的连接层：把视觉向量接入已有语言空间","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-ac6bae9cd4aa658d03e07760","nextBlock":"b-d461d82c0c23d03a09aed338","prerequisites":[{"id":"b-f15cc400f416177ccaed477e","url":"#b-f15cc400f416177ccaed477e","role":"section-title"},{"id":"b-e80c77140932ec81b8f0e4ba","url":"#b-e80c77140932ec81b8f0e4ba","role":"problem-context"}],"relatedBlocks":[{"id":"b-d461d82c0c23d03a09aed338","url":"#b-d461d82c0c23d03a09aed338","relation":"figure-and-explanation"}]},{"id":"b-d461d82c0c23d03a09aed338","type":"figure","scope":"llava-visual-vector-interface","url":"https://codingai-lec03.pages.dev/course/blocks/llava-visual-vector-interface.html#b-d461d82c0c23d03a09aed338","content":"图内标签与数值：CLIP 视觉编码器；原始 LLaVA：ViT-L/14；网格特征；线性投影 W；视觉向量 → 语言维度；文字指令：；“谁拿着球？”；语言模型 Vicuna；视觉向量 + 文字指令；回答；“红衣人”；接入的是一组视觉特征，而非一个 CLIP 匹配分数；原始 LLaVA §4.1；网格数量为示意。参考回答用于解释接口，未作本轮视觉指令推理。\n\n视觉路径进入语言模型前有独立、可训练的接口；信息身份并非文字标签。 查看对应图解状态\n\n- CLIP 视觉编码器\n- 原始 LLaVA：ViT-L/14\n- 网格特征\n- 线性投影 W\n- 视觉向量 → 语言维度\n- 文字指令：\n- “谁拿着球？”\n- 语言模型 Vicuna\n- 视觉向量 + 文字指令\n- 回答\n- “红衣人”\n- 接入的是一组视觉特征，而非一个 CLIP 匹配分数\n- 原始 LLaVA §4.1；网格数量为示意。参考回答用于解释接口，未作本轮视觉指令推理。视觉路径进入语言模型前有独立、可训练的接口；信息身份并非文字标签。 [查看对应图解状态](/#/l3-r2-p64/0)[查看完整图解](/#/l3-r2-p64/0)","sha256":"85f80cce13ab159988b8e996e323b53a8b067d434699593f3760372b201c5d66","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"reading-opening","title":"LLaVA 的连接层：把视觉向量接入已有语言空间","number":1},"sectionTitle":"LLaVA 的连接层：把视觉向量接入已有语言空间","currentQuestion":"LLaVA 的连接层：把视觉向量接入已有语言空间","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"figure":{"caption":"视觉路径进入语言模型前有独立、可训练的接口；信息身份并非文字标签。 查看对应图解状态","labels":["CLIP 视觉编码器","原始 LLaVA：ViT-L/14","网格特征","线性投影 W","视觉向量 → 语言维度","文字指令：","“谁拿着球？”","语言模型 Vicuna","视觉向量 + 文字指令","回答","“红衣人”","接入的是一组视觉特征，而非一个 CLIP 匹配分数","原始 LLaVA §4.1；网格数量为示意。参考回答用于解释接口，未作本轮视觉指令推理。"],"description":"视觉路径进入语言模型前有独立、可训练的接口；信息身份并非文字标签。 查看对应图解状态CLIP 视觉编码器原始 LLaVA：ViT-L/14网格特征线性投影 W视觉向量 → 语言维度文字指令：“谁拿着球？”语言模型 Vicuna视觉向量 + 文字指令回答“红衣人”接入的是一组视觉特征，而非一个 CLIP 匹配分数原始 LLaVA §4.1；网格数量为示意。参考回答用于解释接口，未作本轮视觉指令推理。","evidenceIdentity":"视觉路径进入语言模型前有独立、可训练的接口；信息身份并非文字标签。 查看对应图解状态","images":[{"src":"/assets/display/512610a019c560ff0384e86a.webp","alt":""}]},"previousBlock":"b-d6ba5ba246e109996814e86e","nextBlock":"b-9d6ee39a5691f0749b5eacb5","prerequisites":[{"id":"b-f15cc400f416177ccaed477e","url":"#b-f15cc400f416177ccaed477e","role":"section-title"},{"id":"b-e80c77140932ec81b8f0e4ba","url":"#b-e80c77140932ec81b8f0e4ba","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-9d6ee39a5691f0749b5eacb5","type":"h2","scope":"llava-visual-vector-interface","url":"https://codingai-lec03.pages.dev/course/blocks/llava-visual-vector-interface.html#b-9d6ee39a5691f0749b5eacb5","content":"## 继续阅读与实验","sha256":"2b9ccef7e1220e2250cceff1abd8bdd736b42f8da5a223f26b029f44b962e807","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"reading-opening","title":"LLaVA 的连接层：把视觉向量接入已有语言空间","number":1},"sectionTitle":"LLaVA 的连接层：把视觉向量接入已有语言空间","currentQuestion":"LLaVA 的连接层：把视觉向量接入已有语言空间","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-d461d82c0c23d03a09aed338","nextBlock":null,"prerequisites":[{"id":"b-f15cc400f416177ccaed477e","url":"#b-f15cc400f416177ccaed477e","role":"section-title"},{"id":"b-e80c77140932ec81b8f0e4ba","url":"#b-e80c77140932ec81b8f0e4ba","role":"problem-context"}],"relatedBlocks":[{"id":"b-d461d82c0c23d03a09aed338","url":"#b-d461d82c0c23d03a09aed338","relation":"figure-and-explanation"}]}]}