{"version":2,"relationshipURLBase":"https://codingai-lec03.pages.dev/course/notebook-blocks/ddbfd4bf2bf747b0ae9f1cba853e8b68.html","relationshipResolution":"Resolve relative relationship URLs against relationshipURLBase; complete records and absolute URLs are in blocks.","title":"原始 LLaVA 的接口是视觉编码器 → 线性投影 → 语言模型","url":"https://codingai-lec03.pages.dev/course/notebook-blocks/ddbfd4bf2bf747b0ae9f1cba853e8b68.html","markdown":"https://codingai-lec03.pages.dev/course/notebook-blocks/ddbfd4bf2bf747b0ae9f1cba853e8b68.md","pages":[{"id":"reading-opening","title":"原始 LLaVA 的接口是视觉编码器 → 线性投影 → 语言模型","unit":"原始 LLaVA 的接口是视觉编码器 → 线性投影 → 语言模型","unitId":"notebook-blocks/ddbfd4bf2bf747b0ae9f1cba853e8b68","chapter":"course","number":1}],"aliases":[{"oldId":"b-539095fc9a9e03372e950a98","targetId":"b-539095fc9a9e03372e950a98","page":"reading-opening","operation":"move","beforeSHA256":"343fb71807f077069b41c6495a8824ea4d353c740175b5775ccc5145b3bf19ca","afterSHA256":"343fb71807f077069b41c6495a8824ea4d353c740175b5775ccc5145b3bf19ca","reason":"Same source body in authored semantic page"},{"oldId":"b-c961ce98fce072ed964e8f37","targetId":"b-c961ce98fce072ed964e8f37","page":"reading-opening","operation":"move","beforeSHA256":"019e1322f310ae4464ffcf1f43367df922cd0ab05f16591e0d9611eddee12794","afterSHA256":"019e1322f310ae4464ffcf1f43367df922cd0ab05f16591e0d9611eddee12794","reason":"Same source body in authored semantic page"},{"oldId":"b-83827592b80bdcac37f59982","targetId":"b-83827592b80bdcac37f59982","page":"reading-opening","operation":"move","beforeSHA256":"42cba31deccba2a350b9b23bd0ca1dcaa4c4c5cb9d35c773e1fcb89124788ccd","afterSHA256":"42cba31deccba2a350b9b23bd0ca1dcaa4c4c5cb9d35c773e1fcb89124788ccd","reason":"Same source body in authored semantic page"},{"oldId":"b-cf4979e7fff2676ac45af2b1","targetId":"b-cf4979e7fff2676ac45af2b1","page":"reading-opening","operation":"move","beforeSHA256":"41c70c2c07b82678ff0a60f2d009b6afa750c5c691d109e754a69abf23868b50","afterSHA256":"41c70c2c07b82678ff0a60f2d009b6afa750c5c691d109e754a69abf23868b50","reason":"Same source body in authored semantic page"},{"oldId":"b-7139064d67ae3195db4bb775","targetId":"b-7139064d67ae3195db4bb775","page":"reading-opening","operation":"move","beforeSHA256":"617e419c317d8efed8d341ce4af6ad4d1caf49a51c7bae9d4d1605603b749536","afterSHA256":"617e419c317d8efed8d341ce4af6ad4d1caf49a51c7bae9d4d1605603b749536","reason":"Same source body in authored semantic page"},{"oldId":"b-6f7ea242698921b194063d45","targetId":"b-6f7ea242698921b194063d45","page":"reading-opening","operation":"move","beforeSHA256":"05a43b7dca3fcbe6656e1d5fe6c72c489f7746be571acf25013a8234f62cc4ca","afterSHA256":"05a43b7dca3fcbe6656e1d5fe6c72c489f7746be571acf25013a8234f62cc4ca","reason":"Same source body in authored semantic page"},{"oldId":"b-c1196106d5b7b22ca705865a","targetId":"b-c1196106d5b7b22ca705865a","page":"reading-opening","operation":"move","beforeSHA256":"53284538628970eb67c2983de4a891ba92836b5ea5f1c443f8064a010523ac09","afterSHA256":"53284538628970eb67c2983de4a891ba92836b5ea5f1c443f8064a010523ac09","reason":"Same source body in authored semantic page"}],"blocks":[{"id":"b-539095fc9a9e03372e950a98","type":"p","scope":"document","url":"https://codingai-lec03.pages.dev/course/notebook-blocks/ddbfd4bf2bf747b0ae9f1cba853e8b68.html#b-539095fc9a9e03372e950a98","content":"[CLS 03 · 视觉与语言：任务、训练与接口](/course/notebooks/680da245145447dd88983aabd136fd24.html)","sha256":"343fb71807f077069b41c6495a8824ea4d353c740175b5775ccc5145b3bf19ca","chapter":{"id":"course","title":"原始 LLaVA 的接口是视觉编码器 → 线性投影 → 语言模型"},"unit":{"id":"notebook-blocks/ddbfd4bf2bf747b0ae9f1cba853e8b68","title":"原始 LLaVA 的接口是视觉编码器 → 线性投影 → 语言模型"},"page":{"id":"reading-opening","title":"原始 LLaVA 的接口是视觉编码器 → 线性投影 → 语言模型","number":1},"sectionTitle":"原始 LLaVA 的接口是视觉编码器 → 线性投影 → 语言模型","currentQuestion":"原始 LLaVA 的接口是视觉编码器 → 线性投影 → 语言模型","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":null,"nextBlock":"b-c961ce98fce072ed964e8f37","prerequisites":[],"relatedBlocks":[]},{"id":"b-c961ce98fce072ed964e8f37","type":"h1","scope":"document","url":"https://codingai-lec03.pages.dev/course/notebook-blocks/ddbfd4bf2bf747b0ae9f1cba853e8b68.html#b-c961ce98fce072ed964e8f37","content":"# 原始 LLaVA 的接口是视觉编码器 → 线性投影 → 语言模型","sha256":"019e1322f310ae4464ffcf1f43367df922cd0ab05f16591e0d9611eddee12794","chapter":{"id":"course","title":"原始 LLaVA 的接口是视觉编码器 → 线性投影 → 语言模型"},"unit":{"id":"notebook-blocks/ddbfd4bf2bf747b0ae9f1cba853e8b68","title":"原始 LLaVA 的接口是视觉编码器 → 线性投影 → 语言模型"},"page":{"id":"reading-opening","title":"原始 LLaVA 的接口是视觉编码器 → 线性投影 → 语言模型","number":1},"sectionTitle":"原始 LLaVA 的接口是视觉编码器 → 线性投影 → 语言模型","currentQuestion":"原始 LLaVA 的接口是视觉编码器 → 线性投影 → 语言模型","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-539095fc9a9e03372e950a98","nextBlock":"b-83827592b80bdcac37f59982","prerequisites":[{"id":"b-539095fc9a9e03372e950a98","url":"#b-539095fc9a9e03372e950a98","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-83827592b80bdcac37f59982","type":"p","scope":"document","url":"https://codingai-lec03.pages.dev/course/notebook-blocks/ddbfd4bf2bf747b0ae9f1cba853e8b68.html#b-83827592b80bdcac37f59982","content":"主线阅读与优先实验","sha256":"42cba31deccba2a350b9b23bd0ca1dcaa4c4c5cb9d35c773e1fcb89124788ccd","chapter":{"id":"course","title":"原始 LLaVA 的接口是视觉编码器 → 线性投影 → 语言模型"},"unit":{"id":"notebook-blocks/ddbfd4bf2bf747b0ae9f1cba853e8b68","title":"原始 LLaVA 的接口是视觉编码器 → 线性投影 → 语言模型"},"page":{"id":"reading-opening","title":"原始 LLaVA 的接口是视觉编码器 → 线性投影 → 语言模型","number":1},"sectionTitle":"原始 LLaVA 的接口是视觉编码器 → 线性投影 → 语言模型","currentQuestion":"原始 LLaVA 的接口是视觉编码器 → 线性投影 → 语言模型","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-c961ce98fce072ed964e8f37","nextBlock":"b-cf4979e7fff2676ac45af2b1","prerequisites":[{"id":"b-c961ce98fce072ed964e8f37","url":"#b-c961ce98fce072ed964e8f37","role":"section-title"},{"id":"b-539095fc9a9e03372e950a98","url":"#b-539095fc9a9e03372e950a98","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-cf4979e7fff2676ac45af2b1","type":"p","scope":"document","url":"https://codingai-lec03.pages.dev/course/notebook-blocks/ddbfd4bf2bf747b0ae9f1cba853e8b68.html#b-cf4979e7fff2676ac45af2b1","content":"caption历史、VQA、CLIP候选、关系控制、GLIP/LLaVA与生成桥梁","sha256":"41c70c2c07b82678ff0a60f2d009b6afa750c5c691d109e754a69abf23868b50","chapter":{"id":"course","title":"原始 LLaVA 的接口是视觉编码器 → 线性投影 → 语言模型"},"unit":{"id":"notebook-blocks/ddbfd4bf2bf747b0ae9f1cba853e8b68","title":"原始 LLaVA 的接口是视觉编码器 → 线性投影 → 语言模型"},"page":{"id":"reading-opening","title":"原始 LLaVA 的接口是视觉编码器 → 线性投影 → 语言模型","number":1},"sectionTitle":"原始 LLaVA 的接口是视觉编码器 → 线性投影 → 语言模型","currentQuestion":"原始 LLaVA 的接口是视觉编码器 → 线性投影 → 语言模型","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-83827592b80bdcac37f59982","nextBlock":"b-7139064d67ae3195db4bb775","prerequisites":[{"id":"b-c961ce98fce072ed964e8f37","url":"#b-c961ce98fce072ed964e8f37","role":"section-title"},{"id":"b-539095fc9a9e03372e950a98","url":"#b-539095fc9a9e03372e950a98","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-7139064d67ae3195db4bb775","type":"p","scope":"document","url":"https://codingai-lec03.pages.dev/course/notebook-blocks/ddbfd4bf2bf747b0ae9f1cba853e8b68.html#b-7139064d67ae3195db4bb775","content":"原始 LLaVA 的接口是视觉编码器 → 线性投影 → 语言模型。送入的是视觉网格向量，不是一个类别标签。阶段一冻结视觉与语言模型，只训练投影；阶段二仍冻结视觉，更新投影与语言模型。","sha256":"617e419c317d8efed8d341ce4af6ad4d1caf49a51c7bae9d4d1605603b749536","chapter":{"id":"course","title":"原始 LLaVA 的接口是视觉编码器 → 线性投影 → 语言模型"},"unit":{"id":"notebook-blocks/ddbfd4bf2bf747b0ae9f1cba853e8b68","title":"原始 LLaVA 的接口是视觉编码器 → 线性投影 → 语言模型"},"page":{"id":"reading-opening","title":"原始 LLaVA 的接口是视觉编码器 → 线性投影 → 语言模型","number":1},"sectionTitle":"原始 LLaVA 的接口是视觉编码器 → 线性投影 → 语言模型","currentQuestion":"原始 LLaVA 的接口是视觉编码器 → 线性投影 → 语言模型","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-cf4979e7fff2676ac45af2b1","nextBlock":"b-6f7ea242698921b194063d45","prerequisites":[{"id":"b-c961ce98fce072ed964e8f37","url":"#b-c961ce98fce072ed964e8f37","role":"section-title"},{"id":"b-539095fc9a9e03372e950a98","url":"#b-539095fc9a9e03372e950a98","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-6f7ea242698921b194063d45","type":"p","scope":"document","url":"https://codingai-lec03.pages.dev/course/notebook-blocks/ddbfd4bf2bf747b0ae9f1cba853e8b68.html#b-6f7ea242698921b194063d45","content":"[P64](/course/pages/l3-r2-p64.html) 与 [P65](/course/pages/l3-r2-p65.html) 对照这两个阶段。完整解释集中于 [CLS 11 · 语言定位与视觉模型接口](https://deepnote.com/workspace/ise3309jingao-5ba66b0a-4283-4f5a-ae06-3f68d3201e99/project/38755e4e-23f6-4f5f-8c68-8a2e4114b1c1/notebook/95169c3e298c4c819373bb4534f3dff9?utm_source=openai&utm_medium=mcp&utm_campaign=openaimcp&utm_content=95169c3e298c4c819373bb4534f3dff9&utm_term=get_notebook)；注意后续版本的连接器不倒写为原始配置，RNN/attention 的内部结构留第五讲。","sha256":"05a43b7dca3fcbe6656e1d5fe6c72c489f7746be571acf25013a8234f62cc4ca","chapter":{"id":"course","title":"原始 LLaVA 的接口是视觉编码器 → 线性投影 → 语言模型"},"unit":{"id":"notebook-blocks/ddbfd4bf2bf747b0ae9f1cba853e8b68","title":"原始 LLaVA 的接口是视觉编码器 → 线性投影 → 语言模型"},"page":{"id":"reading-opening","title":"原始 LLaVA 的接口是视觉编码器 → 线性投影 → 语言模型","number":1},"sectionTitle":"原始 LLaVA 的接口是视觉编码器 → 线性投影 → 语言模型","currentQuestion":"原始 LLaVA 的接口是视觉编码器 → 线性投影 → 语言模型","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-7139064d67ae3195db4bb775","nextBlock":"b-c1196106d5b7b22ca705865a","prerequisites":[{"id":"b-c961ce98fce072ed964e8f37","url":"#b-c961ce98fce072ed964e8f37","role":"section-title"},{"id":"b-539095fc9a9e03372e950a98","url":"#b-539095fc9a9e03372e950a98","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-c1196106d5b7b22ca705865a","type":"p","scope":"document","url":"https://codingai-lec03.pages.dev/course/notebook-blocks/ddbfd4bf2bf747b0ae9f1cba853e8b68.html#b-c1196106d5b7b22ca705865a","content":"[为何 CLIP 对齐后仍需要连接层](/course/blocks/llava-visual-vector-interface.html) · [逐模块核对冻结范围](/course/blocks/llava-two-stage-freezing.html)。","sha256":"53284538628970eb67c2983de4a891ba92836b5ea5f1c443f8064a010523ac09","chapter":{"id":"course","title":"原始 LLaVA 的接口是视觉编码器 → 线性投影 → 语言模型"},"unit":{"id":"notebook-blocks/ddbfd4bf2bf747b0ae9f1cba853e8b68","title":"原始 LLaVA 的接口是视觉编码器 → 线性投影 → 语言模型"},"page":{"id":"reading-opening","title":"原始 LLaVA 的接口是视觉编码器 → 线性投影 → 语言模型","number":1},"sectionTitle":"原始 LLaVA 的接口是视觉编码器 → 线性投影 → 语言模型","currentQuestion":"原始 LLaVA 的接口是视觉编码器 → 线性投影 → 语言模型","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-6f7ea242698921b194063d45","nextBlock":null,"prerequisites":[{"id":"b-c961ce98fce072ed964e8f37","url":"#b-c961ce98fce072ed964e8f37","role":"section-title"},{"id":"b-539095fc9a9e03372e950a98","url":"#b-539095fc9a9e03372e950a98","role":"problem-context"}],"relatedBlocks":[]}]}