{"version":2,"relationshipURLBase":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p16.html","relationshipResolution":"Resolve relative relationship URLs against relationshipURLBase; complete records and absolute URLs are in blocks.","title":"两个编码器，不同的更新方式","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p16.html","markdown":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p16.md","pages":[{"id":"h-i-ema-updates-parameters","title":"梯度和 EMA，更新的是不同对象","unit":"怎样增加候选，同时让历史表示仍可比较？","unitId":"i-dictionary-and-updates","chapter":"I","number":1}],"aliases":[{"oldId":"explanation","targetId":"h-i-ema-updates-parameters","operation":"split-explanation-range"},{"oldId":"b-bcfb45ef0e08d8b57516cf4a","targetId":"b-bcfb45ef0e08d8b57516cf4a","page":"h-i-ema-updates-parameters","operation":"move","beforeSHA256":"d71cf15e707c8a3674b4c90a599efee783cb84da5b8c8236a1d01c3add7b4958","afterSHA256":"d71cf15e707c8a3674b4c90a599efee783cb84da5b8c8236a1d01c3add7b4958","reason":"Same source body in authored semantic page"},{"oldId":"b-a4636d9714dec21fbe85545c","targetId":"b-a4636d9714dec21fbe85545c","page":"h-i-ema-updates-parameters","operation":"move","beforeSHA256":"fd2b33e3a5e638ff2f2f8ade607257ba36c67e57ea9a0f4e49171131f53ac2c0","afterSHA256":"fd2b33e3a5e638ff2f2f8ade607257ba36c67e57ea9a0f4e49171131f53ac2c0","reason":"Same source body in authored semantic page"},{"oldId":"b-13432c4cf8c708c91e3505e0","targetId":"b-13432c4cf8c708c91e3505e0","page":"h-i-ema-updates-parameters","operation":"move","beforeSHA256":"ee2944b46c15b76772509b769e97f697ec292a22473941d7beba2993b59b3ab2","afterSHA256":"ee2944b46c15b76772509b769e97f697ec292a22473941d7beba2993b59b3ab2","reason":"Same source body in authored semantic page"},{"oldId":"b-746e56cbc855023f26ebe3dc","targetId":"b-746e56cbc855023f26ebe3dc","page":"h-i-ema-updates-parameters","operation":"rewrite","beforeSHA256":"e25e685cf073058c178cb48b69d8cf0734cc5705a9f77a3c62e61326408fdd32","afterSHA256":"ac821d36ae0d1a48c2b5ff787a9030bdaee066fad7ed7094860509753f4d1615","reason":"Same source identity with figure semantics or reading context enriched"},{"oldId":"b-4c5ae6afd428670c72e8ac3c","targetId":"b-4c5ae6afd428670c72e8ac3c","page":"h-i-ema-updates-parameters","operation":"move","beforeSHA256":"786ce10a6b32a492259a3dccd0a54e61cd41d28a34382fb7b5386c5f03b7ad69","afterSHA256":"786ce10a6b32a492259a3dccd0a54e61cd41d28a34382fb7b5386c5f03b7ad69","reason":"Same source body in authored semantic page"},{"oldId":"b-ccf2d9efcc5e5a182493729e","targetId":"b-ccf2d9efcc5e5a182493729e","page":"h-i-ema-updates-parameters","operation":"move","beforeSHA256":"d542a0f5a7605ee40870ccd96ebb75ad96c51a2e2562acdf3f95b7a132f89e0e","afterSHA256":"d542a0f5a7605ee40870ccd96ebb75ad96c51a2e2562acdf3f95b7a132f89e0e","reason":"Same source body in authored semantic page"},{"oldId":"b-22b3bd62dd8bd813f134bcc5","targetId":"b-22b3bd62dd8bd813f134bcc5","page":"h-i-ema-updates-parameters","operation":"move","beforeSHA256":"1c66c594bc8faa85246c1d32ecfea70c38f4796b049761f8afec8ab29fa28442","afterSHA256":"1c66c594bc8faa85246c1d32ecfea70c38f4796b049761f8afec8ab29fa28442","reason":"Same source body in authored semantic page"},{"oldId":"b-d3bc02a89d932979235ea623","targetId":"b-d3bc02a89d932979235ea623","page":"h-i-ema-updates-parameters","operation":"move","beforeSHA256":"6d156c8990596f280690c01b6aebc4611b362c5f7b77a9004dd2912d3a872081","afterSHA256":"6d156c8990596f280690c01b6aebc4611b362c5f7b77a9004dd2912d3a872081","reason":"Same source body in authored semantic page"},{"oldId":"b-188b115bce26f0bb92100b67","targetId":"b-188b115bce26f0bb92100b67","page":"h-i-ema-updates-parameters","operation":"move","beforeSHA256":"06c66c996376d8fa1b89b4e84770ae5de44e3397a0f0e5c2c6514d25104db6ff","afterSHA256":"06c66c996376d8fa1b89b4e84770ae5de44e3397a0f0e5c2c6514d25104db6ff","reason":"Same source body in authored semantic page"},{"oldId":"b-649d98de0a91a63441d5cb8b","targetId":"b-649d98de0a91a63441d5cb8b","page":"h-i-ema-updates-parameters","operation":"move","beforeSHA256":"8b1c1dbb533ad09c59f791550563c5ff85e712ef6972bca73782b226906d9666","afterSHA256":"8b1c1dbb533ad09c59f791550563c5ff85e712ef6972bca73782b226906d9666","reason":"Same source body in authored semantic page"},{"oldId":"b-b5e1bc5d30f468694ded38d7","targetId":"b-b5e1bc5d30f468694ded38d7","page":"h-i-ema-updates-parameters","operation":"move","beforeSHA256":"cf44fcc62782fb4f6bdc42c16c5187d57556a03cb7ce6065e7bd9c36d8e7d733","afterSHA256":"cf44fcc62782fb4f6bdc42c16c5187d57556a03cb7ce6065e7bd9c36d8e7d733","reason":"Same source body in authored semantic page"},{"oldId":"b-1a914cfc536d9974e6fcc68b","targetId":"b-1a914cfc536d9974e6fcc68b","page":"h-i-ema-updates-parameters","operation":"move","beforeSHA256":"e7b20c638a8f288a4afd7f528ac97bf91e2fc3427174f2d59ba30e7ae465727e","afterSHA256":"e7b20c638a8f288a4afd7f528ac97bf91e2fc3427174f2d59ba30e7ae465727e","reason":"Same source body in authored semantic page"},{"oldId":"b-e2f508f9a75e52d88bf46308","targetId":"b-e2f508f9a75e52d88bf46308","page":"h-i-ema-updates-parameters","operation":"move","beforeSHA256":"93340b8a4ae7ee0e667bd9b1d1296c8592f75ba3147b125e14fabe858da33d08","afterSHA256":"93340b8a4ae7ee0e667bd9b1d1296c8592f75ba3147b125e14fabe858da33d08","reason":"Same source body in authored semantic page"},{"oldId":"b-3fce75b9972be04832faf0e9","targetId":"b-3fce75b9972be04832faf0e9","page":"h-i-ema-updates-parameters","operation":"move","beforeSHA256":"d7dce06e9c64fd8a5461eca52b666ed33324f7810c961f6364b7ea9952575617","afterSHA256":"d7dce06e9c64fd8a5461eca52b666ed33324f7810c961f6364b7ea9952575617","reason":"Same source body in authored semantic page"},{"oldId":"b-34070ac0605eb181f3c60458","targetId":"b-34070ac0605eb181f3c60458","page":"h-i-ema-updates-parameters","operation":"move","beforeSHA256":"d7dce06e9c64fd8a5461eca52b666ed33324f7810c961f6364b7ea9952575617","afterSHA256":"d7dce06e9c64fd8a5461eca52b666ed33324f7810c961f6364b7ea9952575617","reason":"Same source body in authored semantic page"},{"oldId":"h-i-ema-updates-parameters-title","targetId":"b-784a38c46b1dbd9b9da655a2","operation":"retain-source-identity"}],"blocks":[{"id":"b-784a38c46b1dbd9b9da655a2","type":"h2","scope":"h-i-ema-updates-parameters","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p16.html#b-784a38c46b1dbd9b9da655a2","content":"## 梯度和 EMA，更新的是不同对象","sha256":"42134f2baa010c1ab734e3ec4e8478184802362b75407d8db85336310f8749f6","chapter":{"id":"I","title":"学习信号与表示"},"unit":{"id":"i-dictionary-and-updates","title":"怎样增加候选，同时让历史表示仍可比较？"},"page":{"id":"h-i-ema-updates-parameters","title":"梯度和 EMA，更新的是不同对象","number":1},"sectionTitle":"梯度和 EMA，更新的是不同对象","currentQuestion":"怎样增加候选，同时让历史表示仍可比较？","sourceSlides":["l3-r2-p16"],"knowledgeBlock":null,"formulas":[],"previousBlock":null,"nextBlock":"b-bcfb45ef0e08d8b57516cf4a","prerequisites":[{"id":"b-13432c4cf8c708c91e3505e0","url":"#b-13432c4cf8c708c91e3505e0","role":"problem-context"}],"relatedBlocks":[{"id":"b-746e56cbc855023f26ebe3dc","url":"#b-746e56cbc855023f26ebe3dc","relation":"figure-and-explanation"}]},{"id":"b-bcfb45ef0e08d8b57516cf4a","type":"h2","scope":"explanation","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p16.html#b-bcfb45ef0e08d8b57516cf4a","content":"## 解释、算例与证据","sha256":"d71cf15e707c8a3674b4c90a599efee783cb84da5b8c8236a1d01c3add7b4958","chapter":{"id":"I","title":"学习信号与表示"},"unit":{"id":"i-dictionary-and-updates","title":"怎样增加候选，同时让历史表示仍可比较？"},"page":{"id":"h-i-ema-updates-parameters","title":"梯度和 EMA，更新的是不同对象","number":1},"sectionTitle":"梯度和 EMA，更新的是不同对象","currentQuestion":"怎样增加候选，同时让历史表示仍可比较？","sourceSlides":["l3-r2-p16"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-784a38c46b1dbd9b9da655a2","nextBlock":"b-a4636d9714dec21fbe85545c","prerequisites":[{"id":"b-784a38c46b1dbd9b9da655a2","url":"#b-784a38c46b1dbd9b9da655a2","role":"section-title"},{"id":"b-13432c4cf8c708c91e3505e0","url":"#b-13432c4cf8c708c91e3505e0","role":"problem-context"}],"relatedBlocks":[{"id":"b-746e56cbc855023f26ebe3dc","url":"#b-746e56cbc855023f26ebe3dc","relation":"figure-and-explanation"}]},{"id":"b-a4636d9714dec21fbe85545c","type":"h1","scope":"document","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p16.html#b-a4636d9714dec21fbe85545c","content":"# P16 · 两个编码器，不同的更新方式","sha256":"fd2b33e3a5e638ff2f2f8ade607257ba36c67e57ea9a0f4e49171131f53ac2c0","chapter":{"id":"I","title":"学习信号与表示"},"unit":{"id":"i-dictionary-and-updates","title":"怎样增加候选，同时让历史表示仍可比较？"},"page":{"id":"h-i-ema-updates-parameters","title":"梯度和 EMA，更新的是不同对象","number":1},"sectionTitle":"梯度和 EMA，更新的是不同对象","currentQuestion":"怎样增加候选，同时让历史表示仍可比较？","sourceSlides":["l3-r2-p16"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-bcfb45ef0e08d8b57516cf4a","nextBlock":"b-13432c4cf8c708c91e3505e0","prerequisites":[{"id":"b-bcfb45ef0e08d8b57516cf4a","url":"#b-bcfb45ef0e08d8b57516cf4a","role":"section-title"},{"id":"b-13432c4cf8c708c91e3505e0","url":"#b-13432c4cf8c708c91e3505e0","role":"problem-context"}],"relatedBlocks":[{"id":"b-746e56cbc855023f26ebe3dc","url":"#b-746e56cbc855023f26ebe3dc","relation":"figure-and-explanation"}]},{"id":"b-13432c4cf8c708c91e3505e0","type":"p","scope":"document","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p16.html#b-13432c4cf8c708c91e3505e0","content":"两种参数更新解决不同任务：query学当前选择，key维持较平滑的编码规则。","sha256":"ee2944b46c15b76772509b769e97f697ec292a22473941d7beba2993b59b3ab2","chapter":{"id":"I","title":"学习信号与表示"},"unit":{"id":"i-dictionary-and-updates","title":"怎样增加候选，同时让历史表示仍可比较？"},"page":{"id":"h-i-ema-updates-parameters","title":"梯度和 EMA，更新的是不同对象","number":1},"sectionTitle":"梯度和 EMA，更新的是不同对象","currentQuestion":"怎样增加候选，同时让历史表示仍可比较？","sourceSlides":["l3-r2-p16"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-a4636d9714dec21fbe85545c","nextBlock":"b-746e56cbc855023f26ebe3dc","prerequisites":[{"id":"b-a4636d9714dec21fbe85545c","url":"#b-a4636d9714dec21fbe85545c","role":"section-title"}],"relatedBlocks":[{"id":"b-746e56cbc855023f26ebe3dc","url":"#b-746e56cbc855023f26ebe3dc","relation":"figure-and-explanation"}]},{"id":"b-746e56cbc855023f26ebe3dc","type":"figure","scope":"document","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p16.html#b-746e56cbc855023f26ebe3dc","content":"图内标签与数值：P16 学习表示机制图，状态 2；查询视图；查询编码器 θq；q；对比损失；键视图；键编码器 θk；k⁺；梯度更新；键分支停止梯度；参数 EMA；θk ← m θk + (1−m) θq；0.9 × 1 + 0.1 × 2 = 1.1；这里的平均对象是网络参数；m = 0.9 仅用于教学算例。\n\nquery分支接受损失梯度，key分支停止梯度且参数由EMA跟随；算例0.9×1+0.1×2=1.1。\n\n- 查询视图\n- 查询编码器 θq\n- q\n- 对比损失\n- 键视图\n- 键编码器 θk\n- k⁺\n- 梯度更新\n- 键分支停止梯度\n- 参数 EMA\n- θk ← m θk + (1−m) θq\n- 0.9 × 1 + 0.1 × 2 = 1.1\n- 这里的平均对象是网络参数；m = 0.9 仅用于教学算例。query分支接受损失梯度，key分支停止梯度且参数由EMA跟随；算例0.9×1+0.1×2=1.1。 [查看对应图解状态](/#/l3-r2-p16/1)[查看完整图解](/course/figures/l3-r2-p16-1.html)","sha256":"ac821d36ae0d1a48c2b5ff787a9030bdaee066fad7ed7094860509753f4d1615","chapter":{"id":"I","title":"学习信号与表示"},"unit":{"id":"i-dictionary-and-updates","title":"怎样增加候选，同时让历史表示仍可比较？"},"page":{"id":"h-i-ema-updates-parameters","title":"梯度和 EMA，更新的是不同对象","number":1},"sectionTitle":"梯度和 EMA，更新的是不同对象","currentQuestion":"怎样增加候选，同时让历史表示仍可比较？","sourceSlides":["l3-r2-p16"],"knowledgeBlock":null,"formulas":[],"figure":{"caption":"query分支接受损失梯度，key分支停止梯度且参数由EMA跟随；算例0.9×1+0.1×2=1.1。 查看对应图解状态","labels":["查询视图","查询编码器 θq","q","对比损失","键视图","键编码器 θk","k⁺","梯度更新","键分支停止梯度","参数 EMA","θk ← m θk + (1−m) θq","0.9 × 1 + 0.1 × 2 = 1.1","这里的平均对象是网络参数；m = 0.9 仅用于教学算例。"],"description":"query分支接受损失梯度，key分支停止梯度且参数由EMA跟随；算例0.9×1+0.1×2=1.1。查询视图查询编码器 θqq对比损失键视图键编码器 θkk⁺梯度更新键分支停止梯度参数 EMAθk ← m θk + (1−m) θq0.9 × 1 + 0.1 × 2 = 1.1这里的平均对象是网络参数；m = 0.9 仅用于教学算例。","evidenceIdentity":"query分支接受损失梯度，key分支停止梯度且参数由EMA跟随；算例0.9×1+0.1×2=1.1。 查看对应图解状态","images":[]},"previousBlock":"b-13432c4cf8c708c91e3505e0","nextBlock":"b-4c5ae6afd428670c72e8ac3c","prerequisites":[{"id":"b-a4636d9714dec21fbe85545c","url":"#b-a4636d9714dec21fbe85545c","role":"section-title"},{"id":"b-13432c4cf8c708c91e3505e0","url":"#b-13432c4cf8c708c91e3505e0","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-4c5ae6afd428670c72e8ac3c","type":"h2","scope":"narrative","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p16.html#b-4c5ae6afd428670c72e8ac3c","content":"## 课堂讲述","sha256":"786ce10a6b32a492259a3dccd0a54e61cd41d28a34382fb7b5386c5f03b7ad69","chapter":{"id":"I","title":"学习信号与表示"},"unit":{"id":"i-dictionary-and-updates","title":"怎样增加候选，同时让历史表示仍可比较？"},"page":{"id":"h-i-ema-updates-parameters","title":"梯度和 EMA，更新的是不同对象","number":1},"sectionTitle":"梯度和 EMA，更新的是不同对象","currentQuestion":"怎样增加候选，同时让历史表示仍可比较？","sourceSlides":["l3-r2-p16"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-746e56cbc855023f26ebe3dc","nextBlock":"b-ccf2d9efcc5e5a182493729e","prerequisites":[{"id":"b-a4636d9714dec21fbe85545c","url":"#b-a4636d9714dec21fbe85545c","role":"section-title"},{"id":"b-13432c4cf8c708c91e3505e0","url":"#b-13432c4cf8c708c91e3505e0","role":"problem-context"}],"relatedBlocks":[{"id":"b-746e56cbc855023f26ebe3dc","url":"#b-746e56cbc855023f26ebe3dc","relation":"figure-and-explanation"}]},{"id":"b-ccf2d9efcc5e5a182493729e","type":"p","scope":"narrative","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p16.html#b-ccf2d9efcc5e5a182493729e","content":"查询编码器用梯度学习；键编码器不直接通过这条对比损失反向更新，而是用查询编码器参数的移动平均来跟随。\n例如只看一个参数分量，旧的键参数是 1，当前查询参数是 2，平均系数取 0.9，那么键参数会变成 1.1，而不是立刻跳到 2。真实网络是对所有参数作这样的更新，这个小数字只是帮助我们看清“缓慢跟随”是什么意思。\n这里的 momentum 平均的是网络参数，不是优化器里累积梯度的动量。较慢变化的键编码器，让跨批次保存的键更一致，但旧特征依然不是被神奇地刷新为今天的结果。队列和参数平均，分别回应“大”和“相对一致”两个要求。","sha256":"d542a0f5a7605ee40870ccd96ebb75ad96c51a2e2562acdf3f95b7a132f89e0e","chapter":{"id":"I","title":"学习信号与表示"},"unit":{"id":"i-dictionary-and-updates","title":"怎样增加候选，同时让历史表示仍可比较？"},"page":{"id":"h-i-ema-updates-parameters","title":"梯度和 EMA，更新的是不同对象","number":1},"sectionTitle":"梯度和 EMA，更新的是不同对象","currentQuestion":"怎样增加候选，同时让历史表示仍可比较？","sourceSlides":["l3-r2-p16"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-4c5ae6afd428670c72e8ac3c","nextBlock":"b-22b3bd62dd8bd813f134bcc5","prerequisites":[{"id":"b-4c5ae6afd428670c72e8ac3c","url":"#b-4c5ae6afd428670c72e8ac3c","role":"section-title"},{"id":"b-13432c4cf8c708c91e3505e0","url":"#b-13432c4cf8c708c91e3505e0","role":"problem-context"}],"relatedBlocks":[{"id":"b-746e56cbc855023f26ebe3dc","url":"#b-746e56cbc855023f26ebe3dc","relation":"figure-and-explanation"}]},{"id":"b-22b3bd62dd8bd813f134bcc5","type":"h3","scope":"moco-two-parameter-updates","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p16.html#b-22b3bd62dd8bd813f134bcc5","content":"### 梯度更新和 EMA 更新作用在不同地方","sha256":"1c66c594bc8faa85246c1d32ecfea70c38f4796b049761f8afec8ab29fa28442","chapter":{"id":"I","title":"学习信号与表示"},"unit":{"id":"i-dictionary-and-updates","title":"怎样增加候选，同时让历史表示仍可比较？"},"page":{"id":"h-i-ema-updates-parameters","title":"梯度和 EMA，更新的是不同对象","number":1},"sectionTitle":"梯度更新和 EMA 更新作用在不同地方","currentQuestion":"怎样增加候选，同时让历史表示仍可比较？","sourceSlides":["l3-r2-p16"],"knowledgeBlock":"moco-two-parameter-updates","formulas":[],"previousBlock":"b-ccf2d9efcc5e5a182493729e","nextBlock":"b-d3bc02a89d932979235ea623","prerequisites":[{"id":"b-4c5ae6afd428670c72e8ac3c","url":"#b-4c5ae6afd428670c72e8ac3c","role":"section-title"},{"id":"b-13432c4cf8c708c91e3505e0","url":"#b-13432c4cf8c708c91e3505e0","role":"problem-context"}],"relatedBlocks":[{"id":"b-746e56cbc855023f26ebe3dc","url":"#b-746e56cbc855023f26ebe3dc","relation":"figure-and-explanation"},{"id":"b-d3bc02a89d932979235ea623","url":"#b-d3bc02a89d932979235ea623","relation":"same-reasoning"},{"id":"b-188b115bce26f0bb92100b67","url":"#b-188b115bce26f0bb92100b67","relation":"same-reasoning"},{"id":"b-649d98de0a91a63441d5cb8b","url":"#b-649d98de0a91a63441d5cb8b","relation":"same-reasoning"}]},{"id":"b-d3bc02a89d932979235ea623","type":"p","scope":"moco-two-parameter-updates","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p16.html#b-d3bc02a89d932979235ea623","content":"query编码器从当前损失接收梯度，例如 $\\theta_q\\leftarrow\\theta_q-\\eta\\nabla_{\\theta_q}L$。key编码器不沿当前loss反传，而通过参数移动平均跟随query：$\\theta_k\\leftarrow m\\theta_k+(1-m)\\theta_q$。","sha256":"6d156c8990596f280690c01b6aebc4611b362c5f7b77a9004dd2912d3a872081","chapter":{"id":"I","title":"学习信号与表示"},"unit":{"id":"i-dictionary-and-updates","title":"怎样增加候选，同时让历史表示仍可比较？"},"page":{"id":"h-i-ema-updates-parameters","title":"梯度和 EMA，更新的是不同对象","number":1},"sectionTitle":"梯度更新和 EMA 更新作用在不同地方","currentQuestion":"怎样增加候选，同时让历史表示仍可比较？","sourceSlides":["l3-r2-p16"],"knowledgeBlock":"moco-two-parameter-updates","formulas":[{"latex":"\\theta_q\\leftarrow\\theta_q-\\eta\\nabla_{\\theta_q}L","definitionContext":"query编码器从当前损失接收梯度，例如 $\\theta_q\\leftarrow\\theta_q-\\eta\\nabla_{\\theta_q}L$。key编码器不沿当前loss反传，而通过参数移动平均跟随query：$\\theta_k\\leftarrow m\\theta_k+(1-m)\\theta_q$。\n令旧key参数为(0,2)，当前query参数为(2,4)，m=0.9，则key变为(0.2,2.2)。若query下一次仍不变，再更新得到(0.38,2.38)。平均的是网络参数，不是历史队列向量；也不同于优化器保存的梯度动量。\n较慢的key变化减轻历史向量的坐标系漂移。它并不把历史key全部校正为新参数下的值，也不保证任意队列容量都合适。m=0立即追上query，m=1彻底冻结；这两个极端帮助理解时间尺度，但通常不是希望采用的默认方案。"},{"latex":"\\theta_k\\leftarrow m\\theta_k+(1-m)\\theta_q","definitionContext":"query编码器从当前损失接收梯度，例如 $\\theta_q\\leftarrow\\theta_q-\\eta\\nabla_{\\theta_q}L$。key编码器不沿当前loss反传，而通过参数移动平均跟随query：$\\theta_k\\leftarrow m\\theta_k+(1-m)\\theta_q$。\n令旧key参数为(0,2)，当前query参数为(2,4)，m=0.9，则key变为(0.2,2.2)。若query下一次仍不变，再更新得到(0.38,2.38)。平均的是网络参数，不是历史队列向量；也不同于优化器保存的梯度动量。\n较慢的key变化减轻历史向量的坐标系漂移。它并不把历史key全部校正为新参数下的值，也不保证任意队列容量都合适。m=0立即追上query，m=1彻底冻结；这两个极端帮助理解时间尺度，但通常不是希望采用的默认方案。"}],"previousBlock":"b-22b3bd62dd8bd813f134bcc5","nextBlock":"b-188b115bce26f0bb92100b67","prerequisites":[{"id":"b-22b3bd62dd8bd813f134bcc5","url":"#b-22b3bd62dd8bd813f134bcc5","role":"section-title"},{"id":"b-13432c4cf8c708c91e3505e0","url":"#b-13432c4cf8c708c91e3505e0","role":"problem-context"}],"relatedBlocks":[{"id":"b-746e56cbc855023f26ebe3dc","url":"#b-746e56cbc855023f26ebe3dc","relation":"figure-and-explanation"},{"id":"b-22b3bd62dd8bd813f134bcc5","url":"#b-22b3bd62dd8bd813f134bcc5","relation":"same-reasoning"},{"id":"b-188b115bce26f0bb92100b67","url":"#b-188b115bce26f0bb92100b67","relation":"same-reasoning"},{"id":"b-649d98de0a91a63441d5cb8b","url":"#b-649d98de0a91a63441d5cb8b","relation":"same-reasoning"}]},{"id":"b-188b115bce26f0bb92100b67","type":"p","scope":"moco-two-parameter-updates","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p16.html#b-188b115bce26f0bb92100b67","content":"令旧key参数为(0,2)，当前query参数为(2,4)，m=0.9，则key变为(0.2,2.2)。若query下一次仍不变，再更新得到(0.38,2.38)。平均的是网络参数，不是历史队列向量；也不同于优化器保存的梯度动量。","sha256":"06c66c996376d8fa1b89b4e84770ae5de44e3397a0f0e5c2c6514d25104db6ff","chapter":{"id":"I","title":"学习信号与表示"},"unit":{"id":"i-dictionary-and-updates","title":"怎样增加候选，同时让历史表示仍可比较？"},"page":{"id":"h-i-ema-updates-parameters","title":"梯度和 EMA，更新的是不同对象","number":1},"sectionTitle":"梯度更新和 EMA 更新作用在不同地方","currentQuestion":"怎样增加候选，同时让历史表示仍可比较？","sourceSlides":["l3-r2-p16"],"knowledgeBlock":"moco-two-parameter-updates","formulas":[],"previousBlock":"b-d3bc02a89d932979235ea623","nextBlock":"b-649d98de0a91a63441d5cb8b","prerequisites":[{"id":"b-22b3bd62dd8bd813f134bcc5","url":"#b-22b3bd62dd8bd813f134bcc5","role":"section-title"},{"id":"b-13432c4cf8c708c91e3505e0","url":"#b-13432c4cf8c708c91e3505e0","role":"problem-context"}],"relatedBlocks":[{"id":"b-746e56cbc855023f26ebe3dc","url":"#b-746e56cbc855023f26ebe3dc","relation":"figure-and-explanation"},{"id":"b-22b3bd62dd8bd813f134bcc5","url":"#b-22b3bd62dd8bd813f134bcc5","relation":"same-reasoning"},{"id":"b-d3bc02a89d932979235ea623","url":"#b-d3bc02a89d932979235ea623","relation":"same-reasoning"},{"id":"b-649d98de0a91a63441d5cb8b","url":"#b-649d98de0a91a63441d5cb8b","relation":"same-reasoning"}]},{"id":"b-649d98de0a91a63441d5cb8b","type":"p","scope":"moco-two-parameter-updates","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p16.html#b-649d98de0a91a63441d5cb8b","content":"较慢的key变化减轻历史向量的坐标系漂移。它并不把历史key全部校正为新参数下的值，也不保证任意队列容量都合适。m=0立即追上query，m=1彻底冻结；这两个极端帮助理解时间尺度，但通常不是希望采用的默认方案。","sha256":"8b1c1dbb533ad09c59f791550563c5ff85e712ef6972bca73782b226906d9666","chapter":{"id":"I","title":"学习信号与表示"},"unit":{"id":"i-dictionary-and-updates","title":"怎样增加候选，同时让历史表示仍可比较？"},"page":{"id":"h-i-ema-updates-parameters","title":"梯度和 EMA，更新的是不同对象","number":1},"sectionTitle":"梯度更新和 EMA 更新作用在不同地方","currentQuestion":"怎样增加候选，同时让历史表示仍可比较？","sourceSlides":["l3-r2-p16"],"knowledgeBlock":"moco-two-parameter-updates","formulas":[],"previousBlock":"b-188b115bce26f0bb92100b67","nextBlock":"b-b5e1bc5d30f468694ded38d7","prerequisites":[{"id":"b-22b3bd62dd8bd813f134bcc5","url":"#b-22b3bd62dd8bd813f134bcc5","role":"section-title"},{"id":"b-13432c4cf8c708c91e3505e0","url":"#b-13432c4cf8c708c91e3505e0","role":"problem-context"}],"relatedBlocks":[{"id":"b-746e56cbc855023f26ebe3dc","url":"#b-746e56cbc855023f26ebe3dc","relation":"figure-and-explanation"},{"id":"b-22b3bd62dd8bd813f134bcc5","url":"#b-22b3bd62dd8bd813f134bcc5","relation":"same-reasoning"},{"id":"b-d3bc02a89d932979235ea623","url":"#b-d3bc02a89d932979235ea623","relation":"same-reasoning"},{"id":"b-188b115bce26f0bb92100b67","url":"#b-188b115bce26f0bb92100b67","relation":"same-reasoning"}]},{"id":"b-b5e1bc5d30f468694ded38d7","type":"h2","scope":"practice","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p16.html#b-b5e1bc5d30f468694ded38d7","content":"## 沿这个问题继续","sha256":"cf44fcc62782fb4f6bdc42c16c5187d57556a03cb7ce6065e7bd9c36d8e7d733","chapter":{"id":"I","title":"学习信号与表示"},"unit":{"id":"i-dictionary-and-updates","title":"怎样增加候选，同时让历史表示仍可比较？"},"page":{"id":"h-i-ema-updates-parameters","title":"梯度和 EMA，更新的是不同对象","number":1},"sectionTitle":"梯度和 EMA，更新的是不同对象","currentQuestion":"怎样增加候选，同时让历史表示仍可比较？","sourceSlides":["l3-r2-p16"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-649d98de0a91a63441d5cb8b","nextBlock":"b-1a914cfc536d9974e6fcc68b","prerequisites":[{"id":"b-22b3bd62dd8bd813f134bcc5","url":"#b-22b3bd62dd8bd813f134bcc5","role":"section-title"},{"id":"b-13432c4cf8c708c91e3505e0","url":"#b-13432c4cf8c708c91e3505e0","role":"problem-context"}],"relatedBlocks":[{"id":"b-746e56cbc855023f26ebe3dc","url":"#b-746e56cbc855023f26ebe3dc","relation":"figure-and-explanation"}]},{"id":"b-1a914cfc536d9974e6fcc68b","type":"p","scope":"practice","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p16.html#b-1a914cfc536d9974e6fcc68b","content":"手算两次EMA，并在图上指出梯度箭头与EMA箭头的落点；再到P17串完整一步。","sha256":"e7b20c638a8f288a4afd7f528ac97bf91e2fc3427174f2d59ba30e7ae465727e","chapter":{"id":"I","title":"学习信号与表示"},"unit":{"id":"i-dictionary-and-updates","title":"怎样增加候选，同时让历史表示仍可比较？"},"page":{"id":"h-i-ema-updates-parameters","title":"梯度和 EMA，更新的是不同对象","number":1},"sectionTitle":"梯度和 EMA，更新的是不同对象","currentQuestion":"怎样增加候选，同时让历史表示仍可比较？","sourceSlides":["l3-r2-p16"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-b5e1bc5d30f468694ded38d7","nextBlock":"b-55744bbad055f9d19a12614d","prerequisites":[{"id":"b-b5e1bc5d30f468694ded38d7","url":"#b-b5e1bc5d30f468694ded38d7","role":"section-title"},{"id":"b-13432c4cf8c708c91e3505e0","url":"#b-13432c4cf8c708c91e3505e0","role":"problem-context"}],"relatedBlocks":[{"id":"b-746e56cbc855023f26ebe3dc","url":"#b-746e56cbc855023f26ebe3dc","relation":"figure-and-explanation"}]},{"id":"b-55744bbad055f9d19a12614d","type":"h4","scope":"h-i-ema-updates-parameters","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p16.html#b-55744bbad055f9d19a12614d","content":"#### 图示条件与公式约定","sha256":"ef7f915c1ae63ed67bd23fecf050249273ba0efa8965d2c8d1c1c6b2d3178122","chapter":{"id":"I","title":"学习信号与表示"},"unit":{"id":"i-dictionary-and-updates","title":"怎样增加候选，同时让历史表示仍可比较？"},"page":{"id":"h-i-ema-updates-parameters","title":"梯度和 EMA，更新的是不同对象","number":1},"sectionTitle":"梯度和 EMA，更新的是不同对象","currentQuestion":"怎样增加候选，同时让历史表示仍可比较？","sourceSlides":["l3-r2-p16"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-1a914cfc536d9974e6fcc68b","nextBlock":"b-e2f508f9a75e52d88bf46308","prerequisites":[{"id":"b-b5e1bc5d30f468694ded38d7","url":"#b-b5e1bc5d30f468694ded38d7","role":"section-title"},{"id":"b-13432c4cf8c708c91e3505e0","url":"#b-13432c4cf8c708c91e3505e0","role":"problem-context"}],"relatedBlocks":[{"id":"b-746e56cbc855023f26ebe3dc","url":"#b-746e56cbc855023f26ebe3dc","relation":"figure-and-explanation"}]},{"id":"b-e2f508f9a75e52d88bf46308","type":"p","scope":"document","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p16.html#b-e2f508f9a75e52d88bf46308","content":"EMA 算例 m=.9、旧 θk=1、当前 θq=2，结果1.1。真实网络对参数张量更新；这不是优化器梯度动量。","sha256":"93340b8a4ae7ee0e667bd9b1d1296c8592f75ba3147b125e14fabe858da33d08","chapter":{"id":"I","title":"学习信号与表示"},"unit":{"id":"i-dictionary-and-updates","title":"怎样增加候选，同时让历史表示仍可比较？"},"page":{"id":"h-i-ema-updates-parameters","title":"梯度和 EMA，更新的是不同对象","number":1},"sectionTitle":"梯度和 EMA，更新的是不同对象","currentQuestion":"怎样增加候选，同时让历史表示仍可比较？","sourceSlides":["l3-r2-p16"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-55744bbad055f9d19a12614d","nextBlock":"b-4ddbbe98cc8a0d224b17d864","prerequisites":[{"id":"b-55744bbad055f9d19a12614d","url":"#b-55744bbad055f9d19a12614d","role":"section-title"},{"id":"b-13432c4cf8c708c91e3505e0","url":"#b-13432c4cf8c708c91e3505e0","role":"problem-context"}],"relatedBlocks":[{"id":"b-746e56cbc855023f26ebe3dc","url":"#b-746e56cbc855023f26ebe3dc","relation":"figure-and-explanation"}]},{"id":"b-4ddbbe98cc8a0d224b17d864","type":"h4","scope":"retained-reading","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p16.html#b-4ddbbe98cc8a0d224b17d864","content":"#### 这条线索的展开与后续阅读","sha256":"48b8229f7559b36ed4d5628416e3991e5c73293fbce43374db67386c8736f3bc","chapter":{"id":"I","title":"学习信号与表示"},"unit":{"id":"i-dictionary-and-updates","title":"怎样增加候选，同时让历史表示仍可比较？"},"page":{"id":"h-i-ema-updates-parameters","title":"梯度和 EMA，更新的是不同对象","number":1},"sectionTitle":"梯度和 EMA，更新的是不同对象","currentQuestion":"怎样增加候选，同时让历史表示仍可比较？","sourceSlides":["l3-r2-p16"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-e2f508f9a75e52d88bf46308","nextBlock":"b-3fce75b9972be04832faf0e9","prerequisites":[{"id":"b-55744bbad055f9d19a12614d","url":"#b-55744bbad055f9d19a12614d","role":"section-title"},{"id":"b-13432c4cf8c708c91e3505e0","url":"#b-13432c4cf8c708c91e3505e0","role":"problem-context"}],"relatedBlocks":[{"id":"b-746e56cbc855023f26ebe3dc","url":"#b-746e56cbc855023f26ebe3dc","relation":"figure-and-explanation"}]},{"id":"b-3fce75b9972be04832faf0e9","type":"ul","scope":"retained-reading","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p16.html#b-3fce75b9972be04832faf0e9","content":"- [查询编码器学得快，键编码器缓慢跟随](/supplement.html?return=l3-r2-p16/0#/l3-moco-ema/0) 本页说明","sha256":"d7dce06e9c64fd8a5461eca52b666ed33324f7810c961f6364b7ea9952575617","chapter":{"id":"I","title":"学习信号与表示"},"unit":{"id":"i-dictionary-and-updates","title":"怎样增加候选，同时让历史表示仍可比较？"},"page":{"id":"h-i-ema-updates-parameters","title":"梯度和 EMA，更新的是不同对象","number":1},"sectionTitle":"梯度和 EMA，更新的是不同对象","currentQuestion":"怎样增加候选，同时让历史表示仍可比较？","sourceSlides":["l3-r2-p16"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-4ddbbe98cc8a0d224b17d864","nextBlock":"b-34070ac0605eb181f3c60458","prerequisites":[{"id":"b-4ddbbe98cc8a0d224b17d864","url":"#b-4ddbbe98cc8a0d224b17d864","role":"section-title"},{"id":"b-13432c4cf8c708c91e3505e0","url":"#b-13432c4cf8c708c91e3505e0","role":"problem-context"}],"relatedBlocks":[{"id":"b-746e56cbc855023f26ebe3dc","url":"#b-746e56cbc855023f26ebe3dc","relation":"figure-and-explanation"}]},{"id":"b-34070ac0605eb181f3c60458","type":"li","scope":"retained-reading","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p16.html#b-34070ac0605eb181f3c60458","content":"- [查询编码器学得快，键编码器缓慢跟随](/supplement.html?return=l3-r2-p16/0#/l3-moco-ema/0) 本页说明","sha256":"d7dce06e9c64fd8a5461eca52b666ed33324f7810c961f6364b7ea9952575617","chapter":{"id":"I","title":"学习信号与表示"},"unit":{"id":"i-dictionary-and-updates","title":"怎样增加候选，同时让历史表示仍可比较？"},"page":{"id":"h-i-ema-updates-parameters","title":"梯度和 EMA，更新的是不同对象","number":1},"sectionTitle":"梯度和 EMA，更新的是不同对象","currentQuestion":"怎样增加候选，同时让历史表示仍可比较？","sourceSlides":["l3-r2-p16"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-3fce75b9972be04832faf0e9","nextBlock":null,"prerequisites":[{"id":"b-4ddbbe98cc8a0d224b17d864","url":"#b-4ddbbe98cc8a0d224b17d864","role":"section-title"},{"id":"b-13432c4cf8c708c91e3505e0","url":"#b-13432c4cf8c708c91e3505e0","role":"problem-context"}],"relatedBlocks":[{"id":"b-746e56cbc855023f26ebe3dc","url":"#b-746e56cbc855023f26ebe3dc","relation":"figure-and-explanation"}]}]}