{"version":2,"relationshipURLBase":"https://codingai-lec03.pages.dev/course/blocks/llava-two-stage-freezing.html","relationshipResolution":"Resolve relative relationship URLs against relationshipURLBase; complete records and absolute URLs are in blocks.","title":"原始 LLaVA 两阶段：学习接口，再学习视觉指令","url":"https://codingai-lec03.pages.dev/course/blocks/llava-two-stage-freezing.html","markdown":"https://codingai-lec03.pages.dev/course/blocks/llava-two-stage-freezing.md","pages":[{"id":"reading-opening","title":"原始 LLaVA 两阶段：学习接口，再学习视觉指令","unit":"可复用视觉与语言组件怎样学会共同回答？","unitId":"llava-interface-and-training","chapter":"III","number":1}],"aliases":[{"oldId":"b-6ca7c1a651e355139e30c966","targetId":"b-6ca7c1a651e355139e30c966","page":"reading-opening","operation":"move","beforeSHA256":"f21e3ea2b230cac3baea313dacf95ba88df6846166e0733e0035960723422161","afterSHA256":"f21e3ea2b230cac3baea313dacf95ba88df6846166e0733e0035960723422161","reason":"Same source body in authored semantic page"},{"oldId":"b-ce2578b6e0f33c478a809ed3","targetId":"b-ce2578b6e0f33c478a809ed3","page":"reading-opening","operation":"move","beforeSHA256":"8a6a2a5319b9b8ab3a47b691fdbcd9ed861d802806a729e54755137d694c7761","afterSHA256":"8a6a2a5319b9b8ab3a47b691fdbcd9ed861d802806a729e54755137d694c7761","reason":"Same source body in authored semantic page"},{"oldId":"b-91408b33cf7151d2b301e5d4","targetId":"b-91408b33cf7151d2b301e5d4","page":"reading-opening","operation":"move","beforeSHA256":"87a6785a721b7c4f15e50082fc0085dd997913f6a03a73a379a7a1107ed64174","afterSHA256":"87a6785a721b7c4f15e50082fc0085dd997913f6a03a73a379a7a1107ed64174","reason":"Same source body in authored semantic page"},{"oldId":"b-f0946d1af67fb9ee3156b4ff","targetId":"b-f0946d1af67fb9ee3156b4ff","page":"reading-opening","operation":"rewrite","beforeSHA256":"30ffc21dc1b8b08cc65212c19d6afe1ccef2565d0b8013deb366b63f58f3f910","afterSHA256":"d29e0eaabd68eea2110a49c8c7a9e29cf3ecc4cec131d6744dac2d33dc456091","reason":"Same source identity with figure semantics or reading context enriched"},{"oldId":"b-5ec07600f1eb63317cfe8699","targetId":"b-5ec07600f1eb63317cfe8699","page":"reading-opening","operation":"move","beforeSHA256":"5427ca1e3160be948135875e1b7ca9da3169818b912f9d963e8754786d7157a5","afterSHA256":"5427ca1e3160be948135875e1b7ca9da3169818b912f9d963e8754786d7157a5","reason":"Same source body in authored semantic page"},{"oldId":"b-f0a479fd0f7fd5947cc31306","targetId":"b-f0a479fd0f7fd5947cc31306","page":"reading-opening","operation":"move","beforeSHA256":"648327083320ae884440e22820ac2d90fa5606a4e6a8401b51445aeab27abd9f","afterSHA256":"648327083320ae884440e22820ac2d90fa5606a4e6a8401b51445aeab27abd9f","reason":"Same source body in authored semantic page"},{"oldId":"b-299c1fe88a3744974994781f","targetId":"b-299c1fe88a3744974994781f","page":"reading-opening","operation":"move","beforeSHA256":"fb8950c5997f55cc4a6d23292ef2faaae6ca0ceed6ddb10b3e2a54c54d6d9195","afterSHA256":"fb8950c5997f55cc4a6d23292ef2faaae6ca0ceed6ddb10b3e2a54c54d6d9195","reason":"Same source body in authored semantic page"},{"oldId":"b-da96d66e4b00890d4a727d25","targetId":"b-da96d66e4b00890d4a727d25","page":"reading-opening","operation":"rewrite","beforeSHA256":"cb0ef76ca3ba0857b4ef6d7455ac28c9396259cfe515135fb6729520bd10cd8a","afterSHA256":"ce45fc03aba7f2e165112cd283ee55f4501e48b21771831fa1b28cad02b02602","reason":"Same source identity with figure semantics or reading context enriched"},{"oldId":"b-1152a6814af08458e211a734","targetId":"b-1152a6814af08458e211a734","page":"reading-opening","operation":"move","beforeSHA256":"2b9ccef7e1220e2250cceff1abd8bdd736b42f8da5a223f26b029f44b962e807","afterSHA256":"2b9ccef7e1220e2250cceff1abd8bdd736b42f8da5a223f26b029f44b962e807","reason":"Same source body in authored semantic page"}],"blocks":[{"id":"b-6ca7c1a651e355139e30c966","type":"h1","scope":"llava-two-stage-freezing","url":"https://codingai-lec03.pages.dev/course/blocks/llava-two-stage-freezing.html#b-6ca7c1a651e355139e30c966","content":"# 原始 LLaVA 两阶段：学习接口，再学习视觉指令","sha256":"f21e3ea2b230cac3baea313dacf95ba88df6846166e0733e0035960723422161","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"reading-opening","title":"原始 LLaVA 两阶段：学习接口，再学习视觉指令","number":1},"sectionTitle":"原始 LLaVA 两阶段：学习接口，再学习视觉指令","currentQuestion":"原始 LLaVA 两阶段：学习接口，再学习视觉指令","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":null,"nextBlock":"b-ce2578b6e0f33c478a809ed3","prerequisites":[{"id":"b-ce2578b6e0f33c478a809ed3","url":"#b-ce2578b6e0f33c478a809ed3","role":"problem-context"}],"relatedBlocks":[{"id":"b-da96d66e4b00890d4a727d25","url":"#b-da96d66e4b00890d4a727d25","relation":"figure-and-explanation"}]},{"id":"b-ce2578b6e0f33c478a809ed3","type":"p","scope":"llava-two-stage-freezing","url":"https://codingai-lec03.pages.dev/course/blocks/llava-two-stage-freezing.html#b-ce2578b6e0f33c478a809ed3","content":"本页两个状态只改变训练范围与数据任务：先冻结两大模型训练投影，再冻结视觉、更新投影和语言模型。把每个模块是否更新说清，避免用“端到端”概括掉细节。再读指令数据的来源，训练目标的视觉事实必须能追溯到相应输入。","sha256":"8a6a2a5319b9b8ab3a47b691fdbcd9ed861d802806a729e54755137d694c7761","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"reading-opening","title":"原始 LLaVA 两阶段：学习接口，再学习视觉指令","number":1},"sectionTitle":"原始 LLaVA 两阶段：学习接口，再学习视觉指令","currentQuestion":"原始 LLaVA 两阶段：学习接口，再学习视觉指令","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-6ca7c1a651e355139e30c966","nextBlock":"b-91408b33cf7151d2b301e5d4","prerequisites":[{"id":"b-6ca7c1a651e355139e30c966","url":"#b-6ca7c1a651e355139e30c966","role":"section-title"}],"relatedBlocks":[{"id":"b-da96d66e4b00890d4a727d25","url":"#b-da96d66e4b00890d4a727d25","relation":"figure-and-explanation"}]},{"id":"b-91408b33cf7151d2b301e5d4","type":"p","scope":"llava-two-stage-freezing","url":"https://codingai-lec03.pages.dev/course/blocks/llava-two-stage-freezing.html#b-91408b33cf7151d2b301e5d4","content":"冻结参数表示训练时不更新这组权重，不表示其输出不参与后续计算。原始 LLaVA 的训练范围必须按模块读：","sha256":"87a6785a721b7c4f15e50082fc0085dd997913f6a03a73a379a7a1107ed64174","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"reading-opening","title":"原始 LLaVA 两阶段：学习接口，再学习视觉指令","number":1},"sectionTitle":"原始 LLaVA 两阶段：学习接口，再学习视觉指令","currentQuestion":"原始 LLaVA 两阶段：学习接口，再学习视觉指令","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-ce2578b6e0f33c478a809ed3","nextBlock":"b-f0946d1af67fb9ee3156b4ff","prerequisites":[{"id":"b-6ca7c1a651e355139e30c966","url":"#b-6ca7c1a651e355139e30c966","role":"section-title"},{"id":"b-ce2578b6e0f33c478a809ed3","url":"#b-ce2578b6e0f33c478a809ed3","role":"problem-context"}],"relatedBlocks":[{"id":"b-da96d66e4b00890d4a727d25","url":"#b-da96d66e4b00890d4a727d25","relation":"figure-and-explanation"}]},{"id":"b-f0946d1af67fb9ee3156b4ff","type":"table","scope":"llava-two-stage-freezing","url":"https://codingai-lec03.pages.dev/course/blocks/llava-two-stage-freezing.html#b-f0946d1af67fb9ee3156b4ff","content":"| 阶段 | 视觉编码器 | 线性投影 | 语言模型 | 主要任务 |\n| --- | --- | --- | --- | --- |\n| 特征对齐 | 冻结 | 更新 | 冻结 | 用图文配对建立可用视觉接口 |\n| 视觉指令微调 | 仍冻结 | 继续更新 | 更新 | 根据图片和指令生成回答 |","sha256":"d29e0eaabd68eea2110a49c8c7a9e29cf3ecc4cec131d6744dac2d33dc456091","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"reading-opening","title":"原始 LLaVA 两阶段：学习接口，再学习视觉指令","number":1},"sectionTitle":"原始 LLaVA 两阶段：学习接口，再学习视觉指令","currentQuestion":"原始 LLaVA 两阶段：学习接口，再学习视觉指令","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"table":{"headers":["阶段","视觉编码器","线性投影","语言模型","主要任务"],"rows":[["阶段","视觉编码器","线性投影","语言模型","主要任务"],["特征对齐","冻结","更新","冻结","用图文配对建立可用视觉接口"],["视觉指令微调","仍冻结","继续更新","更新","根据图片和指令生成回答"]]},"previousBlock":"b-91408b33cf7151d2b301e5d4","nextBlock":"b-5ec07600f1eb63317cfe8699","prerequisites":[{"id":"b-6ca7c1a651e355139e30c966","url":"#b-6ca7c1a651e355139e30c966","role":"section-title"},{"id":"b-ce2578b6e0f33c478a809ed3","url":"#b-ce2578b6e0f33c478a809ed3","role":"problem-context"}],"relatedBlocks":[{"id":"b-da96d66e4b00890d4a727d25","url":"#b-da96d66e4b00890d4a727d25","relation":"figure-and-explanation"}]},{"id":"b-5ec07600f1eb63317cfe8699","type":"p","scope":"llava-two-stage-freezing","url":"https://codingai-lec03.pages.dev/course/blocks/llava-two-stage-freezing.html#b-5ec07600f1eb63317cfe8699","content":"第一阶段让已有视觉表示进入语言接口，第二阶段进一步学习如何在不同问法下使用这些信息。两阶段的视觉参数都冻结；论文第二阶段标题虽含 End-to-End，不能据此把视觉分支也画成更新。","sha256":"5427ca1e3160be948135875e1b7ca9da3169818b912f9d963e8754786d7157a5","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"reading-opening","title":"原始 LLaVA 两阶段：学习接口，再学习视觉指令","number":1},"sectionTitle":"原始 LLaVA 两阶段：学习接口，再学习视觉指令","currentQuestion":"原始 LLaVA 两阶段：学习接口，再学习视觉指令","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-f0946d1af67fb9ee3156b4ff","nextBlock":"b-f0a479fd0f7fd5947cc31306","prerequisites":[{"id":"b-6ca7c1a651e355139e30c966","url":"#b-6ca7c1a651e355139e30c966","role":"section-title"},{"id":"b-ce2578b6e0f33c478a809ed3","url":"#b-ce2578b6e0f33c478a809ed3","role":"problem-context"}],"relatedBlocks":[{"id":"b-da96d66e4b00890d4a727d25","url":"#b-da96d66e4b00890d4a727d25","relation":"figure-and-explanation"}]},{"id":"b-f0a479fd0f7fd5947cc31306","type":"p","scope":"llava-two-stage-freezing","url":"https://codingai-lec03.pages.dev/course/blocks/llava-two-stage-freezing.html#b-f0a479fd0f7fd5947cc31306","content":"训练目标仍是条件下预测后续回答，只在应由助手生成的回答与停止位置计入目标。问题和图像属于条件，不是让模型原样抄回的答案。若只记录“用了指令数据”而不记录更新范围，就无法判断能力来自视觉编码变化、连接适配还是语言模型的任务行为变化。","sha256":"648327083320ae884440e22820ac2d90fa5606a4e6a8401b51445aeab27abd9f","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"reading-opening","title":"原始 LLaVA 两阶段：学习接口，再学习视觉指令","number":1},"sectionTitle":"原始 LLaVA 两阶段：学习接口，再学习视觉指令","currentQuestion":"原始 LLaVA 两阶段：学习接口，再学习视觉指令","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-5ec07600f1eb63317cfe8699","nextBlock":"b-299c1fe88a3744974994781f","prerequisites":[{"id":"b-6ca7c1a651e355139e30c966","url":"#b-6ca7c1a651e355139e30c966","role":"section-title"},{"id":"b-ce2578b6e0f33c478a809ed3","url":"#b-ce2578b6e0f33c478a809ed3","role":"problem-context"}],"relatedBlocks":[{"id":"b-da96d66e4b00890d4a727d25","url":"#b-da96d66e4b00890d4a727d25","relation":"figure-and-explanation"}]},{"id":"b-299c1fe88a3744974994781f","type":"p","scope":"llava-two-stage-freezing","url":"https://codingai-lec03.pages.dev/course/blocks/llava-two-stage-freezing.html#b-299c1fe88a3744974994781f","content":"这与第一部分冻结读出/微调的逻辑直接相连：不更新主干仍可以学习更有效的下游用法。其他多模态方案可以选择解冻视觉，这是一项新的训练设计，不能倒写成原始 LLaVA。","sha256":"fb8950c5997f55cc4a6d23292ef2faaae6ca0ceed6ddb10b3e2a54c54d6d9195","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"reading-opening","title":"原始 LLaVA 两阶段：学习接口，再学习视觉指令","number":1},"sectionTitle":"原始 LLaVA 两阶段：学习接口，再学习视觉指令","currentQuestion":"原始 LLaVA 两阶段：学习接口，再学习视觉指令","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-f0a479fd0f7fd5947cc31306","nextBlock":"b-da96d66e4b00890d4a727d25","prerequisites":[{"id":"b-6ca7c1a651e355139e30c966","url":"#b-6ca7c1a651e355139e30c966","role":"section-title"},{"id":"b-ce2578b6e0f33c478a809ed3","url":"#b-ce2578b6e0f33c478a809ed3","role":"problem-context"}],"relatedBlocks":[{"id":"b-da96d66e4b00890d4a727d25","url":"#b-da96d66e4b00890d4a727d25","relation":"figure-and-explanation"}]},{"id":"b-da96d66e4b00890d4a727d25","type":"figure","scope":"llava-two-stage-freezing","url":"https://codingai-lec03.pages.dev/course/blocks/llava-two-stage-freezing.html#b-da96d66e4b00890d4a727d25","content":"图内标签与数值：阶段二 · 图像指令训练任务使用；视觉编码器；冻结；保持参数；投影 W；可训练；更新参数；语言模型；可训练；更新参数；输入：图像 + 视觉指令；目标：指令对应的回答；可训练集合 { W, φ }；两个阶段都冻结视觉编码器；任务训练改变如何使用其特征。；限定原始 LLaVA §4.2。φ 为语言模型参数；不概括所有后续多模态模型的训练方案。\n\n阶段二更新投影与语言模型，视觉编码器仍冻结。 查看对应图解状态\n\n- 阶段二 · 图像指令训练任务使用\n- 视觉编码器\n- 冻结\n- 保持参数\n- 投影 W\n- 可训练\n- 更新参数\n- 语言模型\n- 可训练\n- 更新参数\n- 输入：图像 + 视觉指令\n- 目标：指令对应的回答\n- 可训练集合 { W, φ }\n- 两个阶段都冻结视觉编码器；任务训练改变如何使用其特征。\n- 限定原始 LLaVA §4.2。φ 为语言模型参数；不概括所有后续多模态模型的训练方案。阶段二更新投影与语言模型，视觉编码器仍冻结。 [查看对应图解状态](/#/l3-r2-p65/1)[查看完整图解](/#/l3-r2-p65/1)","sha256":"ce45fc03aba7f2e165112cd283ee55f4501e48b21771831fa1b28cad02b02602","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"reading-opening","title":"原始 LLaVA 两阶段：学习接口，再学习视觉指令","number":1},"sectionTitle":"原始 LLaVA 两阶段：学习接口，再学习视觉指令","currentQuestion":"原始 LLaVA 两阶段：学习接口，再学习视觉指令","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"figure":{"caption":"阶段二更新投影与语言模型，视觉编码器仍冻结。 查看对应图解状态","labels":["阶段二 · 图像指令训练任务使用","视觉编码器","冻结","保持参数","投影 W","可训练","更新参数","语言模型","可训练","更新参数","输入：图像 + 视觉指令","目标：指令对应的回答","可训练集合 { W, φ }","两个阶段都冻结视觉编码器；任务训练改变如何使用其特征。","限定原始 LLaVA §4.2。φ 为语言模型参数；不概括所有后续多模态模型的训练方案。"],"description":"阶段二更新投影与语言模型，视觉编码器仍冻结。 查看对应图解状态阶段二 · 图像指令训练任务使用视觉编码器冻结保持参数投影 W可训练更新参数语言模型可训练更新参数输入：图像 + 视觉指令目标：指令对应的回答可训练集合 { W, φ }两个阶段都冻结视觉编码器；任务训练改变如何使用其特征。限定原始 LLaVA §4.2。φ 为语言模型参数；不概括所有后续多模态模型的训练方案。","evidenceIdentity":"阶段二更新投影与语言模型，视觉编码器仍冻结。 查看对应图解状态","images":[]},"previousBlock":"b-299c1fe88a3744974994781f","nextBlock":"b-1152a6814af08458e211a734","prerequisites":[{"id":"b-6ca7c1a651e355139e30c966","url":"#b-6ca7c1a651e355139e30c966","role":"section-title"},{"id":"b-ce2578b6e0f33c478a809ed3","url":"#b-ce2578b6e0f33c478a809ed3","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-1152a6814af08458e211a734","type":"h2","scope":"llava-two-stage-freezing","url":"https://codingai-lec03.pages.dev/course/blocks/llava-two-stage-freezing.html#b-1152a6814af08458e211a734","content":"## 继续阅读与实验","sha256":"2b9ccef7e1220e2250cceff1abd8bdd736b42f8da5a223f26b029f44b962e807","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"llava-interface-and-training","title":"可复用视觉与语言组件怎样学会共同回答？"},"page":{"id":"reading-opening","title":"原始 LLaVA 两阶段：学习接口，再学习视觉指令","number":1},"sectionTitle":"原始 LLaVA 两阶段：学习接口，再学习视觉指令","currentQuestion":"原始 LLaVA 两阶段：学习接口，再学习视觉指令","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-da96d66e4b00890d4a727d25","nextBlock":null,"prerequisites":[{"id":"b-6ca7c1a651e355139e30c966","url":"#b-6ca7c1a651e355139e30c966","role":"section-title"},{"id":"b-ce2578b6e0f33c478a809ed3","url":"#b-ce2578b6e0f33c478a809ed3","role":"problem-context"}],"relatedBlocks":[{"id":"b-da96d66e4b00890d4a727d25","url":"#b-da96d66e4b00890d4a727d25","relation":"figure-and-explanation"}]}]}