{"version":2,"relationshipURLBase":"https://codingai-lec03.pages.dev/course/notebook-blocks/e7b880e8e242436992df27f02a3a2506.html","relationshipResolution":"Resolve relative relationship URLs against relationshipURLBase; complete records and absolute URLs are in blocks.","title":"3. 视觉特征怎样真正进入语言模型","url":"https://codingai-lec03.pages.dev/course/notebook-blocks/e7b880e8e242436992df27f02a3a2506.html","markdown":"https://codingai-lec03.pages.dev/course/notebook-blocks/e7b880e8e242436992df27f02a3a2506.md","pages":[{"id":"reading-opening","title":"3. 视觉特征怎样真正进入语言模型","unit":"3. 视觉特征怎样真正进入语言模型","unitId":"notebook-blocks/e7b880e8e242436992df27f02a3a2506","chapter":"course","number":1}],"aliases":[{"oldId":"b-824a9ce256ad3fe515425fa8","targetId":"b-824a9ce256ad3fe515425fa8","page":"reading-opening","operation":"move","beforeSHA256":"db08960319942d9119c9cad2cf7d0b78a68c914f8e19ab3bd375531d27c96204","afterSHA256":"db08960319942d9119c9cad2cf7d0b78a68c914f8e19ab3bd375531d27c96204","reason":"Same source body in authored semantic page"},{"oldId":"b-35d4e6d97ba74689623549b5","targetId":"b-35d4e6d97ba74689623549b5","page":"reading-opening","operation":"move","beforeSHA256":"233b63f87fa95e0f39172b075413abe5100a112ae83a1fa57a9297a635df727e","afterSHA256":"233b63f87fa95e0f39172b075413abe5100a112ae83a1fa57a9297a635df727e","reason":"Same source body in authored semantic page"},{"oldId":"b-4ebf9d5428e51cc3eb10f9a8","targetId":"b-4ebf9d5428e51cc3eb10f9a8","page":"reading-opening","operation":"move","beforeSHA256":"e91f24e4e4ca40e2099a0d81d30db6568f4bf2074b499eeee6e3959617bd97ac","afterSHA256":"e91f24e4e4ca40e2099a0d81d30db6568f4bf2074b499eeee6e3959617bd97ac","reason":"Same source body in authored semantic page"},{"oldId":"b-97ddbcf753f16f2127f2608c","targetId":"b-97ddbcf753f16f2127f2608c","page":"reading-opening","operation":"move","beforeSHA256":"6a611aef342a7535362f318ee0c33b63170ce4ab853fa44a05f76a04873fd320","afterSHA256":"6a611aef342a7535362f318ee0c33b63170ce4ab853fa44a05f76a04873fd320","reason":"Same source body in authored semantic page"},{"oldId":"b-36ad65ef571a1e240d86dade","targetId":"b-36ad65ef571a1e240d86dade","page":"reading-opening","operation":"move","beforeSHA256":"84a8b4f5b7259e5f7eac7b28526d4cdb36c75dd3753a2f41bc5d449c22fa73dd","afterSHA256":"84a8b4f5b7259e5f7eac7b28526d4cdb36c75dd3753a2f41bc5d449c22fa73dd","reason":"Same source body in authored semantic page"},{"oldId":"b-c5f53626705ee9c8b669c266","targetId":"b-c5f53626705ee9c8b669c266","page":"reading-opening","operation":"move","beforeSHA256":"5c9d510c1d7a3d21f74bf5a01052b1e1ee1bf8ef098c98e4cb1cc31e721b9e49","afterSHA256":"5c9d510c1d7a3d21f74bf5a01052b1e1ee1bf8ef098c98e4cb1cc31e721b9e49","reason":"Same source body in authored semantic page"},{"oldId":"b-65a4fc9b3437c1badc0432e7","targetId":"b-65a4fc9b3437c1badc0432e7","page":"reading-opening","operation":"move","beforeSHA256":"c41542200dc86ff049ee954783114853bc6d71abbc3bbc3e4dfa4ec098bcce9e","afterSHA256":"c41542200dc86ff049ee954783114853bc6d71abbc3bbc3e4dfa4ec098bcce9e","reason":"Same source body in authored semantic page"},{"oldId":"b-7bb9bce8c47a26127676b144","targetId":"b-7bb9bce8c47a26127676b144","page":"reading-opening","operation":"move","beforeSHA256":"a7a9d6c8a32680a641a952d4142bce66e88936bef853a8ee6e590ab5d5471792","afterSHA256":"a7a9d6c8a32680a641a952d4142bce66e88936bef853a8ee6e590ab5d5471792","reason":"Same source body in authored semantic page"},{"oldId":"b-981f47613e152c08db978134","targetId":"b-981f47613e152c08db978134","page":"reading-opening","operation":"move","beforeSHA256":"1464747dc70ad5f323ac232fd0cebf01dd19e3c36331a819b7d66e25fae7bc5c","afterSHA256":"1464747dc70ad5f323ac232fd0cebf01dd19e3c36331a819b7d66e25fae7bc5c","reason":"Same source body in authored semantic page"},{"oldId":"block-e7b880e8e242436992df27f02a3a2506--3-视觉特征怎样真正进入语言模型","targetId":"b-36ad65ef571a1e240d86dade","operation":"retain-source-identity"}],"blocks":[{"id":"b-824a9ce256ad3fe515425fa8","type":"p","scope":"document","url":"https://codingai-lec03.pages.dev/course/notebook-blocks/e7b880e8e242436992df27f02a3a2506.html#b-824a9ce256ad3fe515425fa8","content":"[LAB 13 · 看图生成一句话：训练、解码与消融](/course/notebooks/3da47f3ee3554ee0a36e0bb6a2fdc61f.html)","sha256":"db08960319942d9119c9cad2cf7d0b78a68c914f8e19ab3bd375531d27c96204","chapter":{"id":"course","title":"3. 视觉特征怎样真正进入语言模型"},"unit":{"id":"notebook-blocks/e7b880e8e242436992df27f02a3a2506","title":"3. 视觉特征怎样真正进入语言模型"},"page":{"id":"reading-opening","title":"3. 视觉特征怎样真正进入语言模型","number":1},"sectionTitle":"3. 视觉特征怎样真正进入语言模型","currentQuestion":"3. 视觉特征怎样真正进入语言模型","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":null,"nextBlock":"b-35d4e6d97ba74689623549b5","prerequisites":[],"relatedBlocks":[]},{"id":"b-35d4e6d97ba74689623549b5","type":"h1","scope":"document","url":"https://codingai-lec03.pages.dev/course/notebook-blocks/e7b880e8e242436992df27f02a3a2506.html#b-35d4e6d97ba74689623549b5","content":"# 3. 视觉特征怎样真正进入语言模型","sha256":"233b63f87fa95e0f39172b075413abe5100a112ae83a1fa57a9297a635df727e","chapter":{"id":"course","title":"3. 视觉特征怎样真正进入语言模型"},"unit":{"id":"notebook-blocks/e7b880e8e242436992df27f02a3a2506","title":"3. 视觉特征怎样真正进入语言模型"},"page":{"id":"reading-opening","title":"3. 视觉特征怎样真正进入语言模型","number":1},"sectionTitle":"3. 视觉特征怎样真正进入语言模型","currentQuestion":"3. 视觉特征怎样真正进入语言模型","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-824a9ce256ad3fe515425fa8","nextBlock":"b-4ebf9d5428e51cc3eb10f9a8","prerequisites":[{"id":"b-824a9ce256ad3fe515425fa8","url":"#b-824a9ce256ad3fe515425fa8","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-4ebf9d5428e51cc3eb10f9a8","type":"p","scope":"document","url":"https://codingai-lec03.pages.dev/course/notebook-blocks/e7b880e8e242436992df27f02a3a2506.html#b-4ebf9d5428e51cc3eb10f9a8","content":"按问题继续","sha256":"e91f24e4e4ca40e2099a0d81d30db6568f4bf2074b499eeee6e3959617bd97ac","chapter":{"id":"course","title":"3. 视觉特征怎样真正进入语言模型"},"unit":{"id":"notebook-blocks/e7b880e8e242436992df27f02a3a2506","title":"3. 视觉特征怎样真正进入语言模型"},"page":{"id":"reading-opening","title":"3. 视觉特征怎样真正进入语言模型","number":1},"sectionTitle":"3. 视觉特征怎样真正进入语言模型","currentQuestion":"3. 视觉特征怎样真正进入语言模型","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-35d4e6d97ba74689623549b5","nextBlock":"b-97ddbcf753f16f2127f2608c","prerequisites":[{"id":"b-35d4e6d97ba74689623549b5","url":"#b-35d4e6d97ba74689623549b5","role":"section-title"},{"id":"b-824a9ce256ad3fe515425fa8","url":"#b-824a9ce256ad3fe515425fa8","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-97ddbcf753f16f2127f2608c","type":"p","scope":"document","url":"https://codingai-lec03.pages.dev/course/notebook-blocks/e7b880e8e242436992df27f02a3a2506.html#b-97ddbcf753f16f2127f2608c","content":"真实CNN–GRU条件训练与greedy/beam生成、遮图/错图消融","sha256":"6a611aef342a7535362f318ee0c33b63170ce4ab853fa44a05f76a04873fd320","chapter":{"id":"course","title":"3. 视觉特征怎样真正进入语言模型"},"unit":{"id":"notebook-blocks/e7b880e8e242436992df27f02a3a2506","title":"3. 视觉特征怎样真正进入语言模型"},"page":{"id":"reading-opening","title":"3. 视觉特征怎样真正进入语言模型","number":1},"sectionTitle":"3. 视觉特征怎样真正进入语言模型","currentQuestion":"3. 视觉特征怎样真正进入语言模型","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-4ebf9d5428e51cc3eb10f9a8","nextBlock":"b-36ad65ef571a1e240d86dade","prerequisites":[{"id":"b-35d4e6d97ba74689623549b5","url":"#b-35d4e6d97ba74689623549b5","role":"section-title"},{"id":"b-824a9ce256ad3fe515425fa8","url":"#b-824a9ce256ad3fe515425fa8","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-36ad65ef571a1e240d86dade","type":"h2","scope":"document","url":"https://codingai-lec03.pages.dev/course/notebook-blocks/e7b880e8e242436992df27f02a3a2506.html#b-36ad65ef571a1e240d86dade","content":"## 3. 视觉特征怎样真正进入语言模型","sha256":"84a8b4f5b7259e5f7eac7b28526d4cdb36c75dd3753a2f41bc5d449c22fa73dd","chapter":{"id":"course","title":"3. 视觉特征怎样真正进入语言模型"},"unit":{"id":"notebook-blocks/e7b880e8e242436992df27f02a3a2506","title":"3. 视觉特征怎样真正进入语言模型"},"page":{"id":"reading-opening","title":"3. 视觉特征怎样真正进入语言模型","number":1},"sectionTitle":"3. 视觉特征怎样真正进入语言模型","currentQuestion":"3. 视觉特征怎样真正进入语言模型","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-97ddbcf753f16f2127f2608c","nextBlock":"b-c5f53626705ee9c8b669c266","prerequisites":[{"id":"b-35d4e6d97ba74689623549b5","url":"#b-35d4e6d97ba74689623549b5","role":"section-title"},{"id":"b-824a9ce256ad3fe515425fa8","url":"#b-824a9ce256ad3fe515425fa8","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-c5f53626705ee9c8b669c266","type":"p","scope":"document","url":"https://codingai-lec03.pages.dev/course/notebook-blocks/e7b880e8e242436992df27f02a3a2506.html#b-c5f53626705ee9c8b669c266","content":"输入是 `[B,3,32,32]`。两次卷积和池化产生空间特征，展平后映射为 48 维视觉向量；这里不用全局平均池化，因为“左/右”依赖空间布局。每个词嵌入为 24 维，与同一视觉向量拼接后送入共享的 GRU；视觉向量还初始化 64 维隐藏状态。输出 logits 为 `[B,T,12]`。","sha256":"5c9d510c1d7a3d21f74bf5a01052b1e1ee1bf8ef098c98e4cb1cc31e721b9e49","chapter":{"id":"course","title":"3. 视觉特征怎样真正进入语言模型"},"unit":{"id":"notebook-blocks/e7b880e8e242436992df27f02a3a2506","title":"3. 视觉特征怎样真正进入语言模型"},"page":{"id":"reading-opening","title":"3. 视觉特征怎样真正进入语言模型","number":1},"sectionTitle":"3. 视觉特征怎样真正进入语言模型","currentQuestion":"3. 视觉特征怎样真正进入语言模型","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-36ad65ef571a1e240d86dade","nextBlock":"b-65a4fc9b3437c1badc0432e7","prerequisites":[{"id":"b-36ad65ef571a1e240d86dade","url":"#b-36ad65ef571a1e240d86dade","role":"section-title"},{"id":"b-824a9ce256ad3fe515425fa8","url":"#b-824a9ce256ad3fe515425fa8","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-65a4fc9b3437c1badc0432e7","type":"p","scope":"document","url":"https://codingai-lec03.pages.dev/course/notebook-blocks/e7b880e8e242436992df27f02a3a2506.html#b-65a4fc9b3437c1badc0432e7","content":"$L=-\\frac{\\sum_{i,t}\\mathbf 1[y_{it}\\ne PAD]\\log p_\\theta(y_{it}\\mid I_i,y_{i,<t})}{\\sum_{i,t}\\mathbf 1[y_{it}\\ne PAD]}.$","sha256":"c41542200dc86ff049ee954783114853bc6d71abbc3bbc3e4dfa4ec098bcce9e","chapter":{"id":"course","title":"3. 视觉特征怎样真正进入语言模型"},"unit":{"id":"notebook-blocks/e7b880e8e242436992df27f02a3a2506","title":"3. 视觉特征怎样真正进入语言模型"},"page":{"id":"reading-opening","title":"3. 视觉特征怎样真正进入语言模型","number":1},"sectionTitle":"3. 视觉特征怎样真正进入语言模型","currentQuestion":"3. 视觉特征怎样真正进入语言模型","sourceSlides":[],"knowledgeBlock":null,"formulas":[{"latex":"L=-\\frac{\\sum_{i,t}\\mathbf 1[y_{it}\\ne PAD]\\log p_\\theta(y_{it}\\mid I_i,y_{i,<t})}{\\sum_{i,t}\\mathbf 1[y_{it}\\ne PAD]}.","definitionContext":"3. 视觉特征怎样真正进入语言模型"}],"previousBlock":"b-c5f53626705ee9c8b669c266","nextBlock":"b-7bb9bce8c47a26127676b144","prerequisites":[{"id":"b-36ad65ef571a1e240d86dade","url":"#b-36ad65ef571a1e240d86dade","role":"section-title"},{"id":"b-824a9ce256ad3fe515425fa8","url":"#b-824a9ce256ad3fe515425fa8","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-7bb9bce8c47a26127676b144","type":"p","scope":"document","url":"https://codingai-lec03.pages.dev/course/notebook-blocks/e7b880e8e242436992df27f02a3a2506.html#b-7bb9bce8c47a26127676b144","content":"代码中的 `targets=Y[:,1:]` 对齐 `prefix=Y[:,:-1]`，`ignore_index=PAD` 排除补齐位置。优化器同时更新 CNN、词嵌入、GRU 和输出头，没有把真实颜色或位置喂给 CNN。每个 epoch 的保留集交叉熵仍是 teacher-forced 量，因此不能替代下一节的自由生成评价。","sha256":"a7a9d6c8a32680a641a952d4142bce66e88936bef853a8ee6e590ab5d5471792","chapter":{"id":"course","title":"3. 视觉特征怎样真正进入语言模型"},"unit":{"id":"notebook-blocks/e7b880e8e242436992df27f02a3a2506","title":"3. 视觉特征怎样真正进入语言模型"},"page":{"id":"reading-opening","title":"3. 视觉特征怎样真正进入语言模型","number":1},"sectionTitle":"3. 视觉特征怎样真正进入语言模型","currentQuestion":"3. 视觉特征怎样真正进入语言模型","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-65a4fc9b3437c1badc0432e7","nextBlock":"b-981f47613e152c08db978134","prerequisites":[{"id":"b-36ad65ef571a1e240d86dade","url":"#b-36ad65ef571a1e240d86dade","role":"section-title"},{"id":"b-824a9ce256ad3fe515425fa8","url":"#b-824a9ce256ad3fe515425fa8","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-981f47613e152c08db978134","type":"p","scope":"document","url":"https://codingai-lec03.pages.dev/course/notebook-blocks/e7b880e8e242436992df27f02a3a2506.html#b-981f47613e152c08db978134","content":"**一次真实失败也有教学意义：** 首轮使用未中心化像素和 tanh 视觉投影时，模型几乎忽略图像，整句准确率为 5.56%，视觉置零结果相同；历史代码、权重和 JSON 均保留。最终结构改为输入中心化、LayerNorm/ReLU 投影并降低学习率，之后锁定新的保留集。结果支持“视觉依赖需要检验”，不支持只凭训练损失下降就宣布模型看图。","sha256":"1464747dc70ad5f323ac232fd0cebf01dd19e3c36331a819b7d66e25fae7bc5c","chapter":{"id":"course","title":"3. 视觉特征怎样真正进入语言模型"},"unit":{"id":"notebook-blocks/e7b880e8e242436992df27f02a3a2506","title":"3. 视觉特征怎样真正进入语言模型"},"page":{"id":"reading-opening","title":"3. 视觉特征怎样真正进入语言模型","number":1},"sectionTitle":"3. 视觉特征怎样真正进入语言模型","currentQuestion":"3. 视觉特征怎样真正进入语言模型","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-7bb9bce8c47a26127676b144","nextBlock":null,"prerequisites":[{"id":"b-36ad65ef571a1e240d86dade","url":"#b-36ad65ef571a1e240d86dade","role":"section-title"},{"id":"b-824a9ce256ad3fe515425fa8","url":"#b-824a9ce256ad3fe515425fa8","role":"problem-context"}],"relatedBlocks":[]}]}