{"version":2,"relationshipURLBase":"https://codingai-lec03.pages.dev/course/blocks/moco-two-parameter-updates.html","relationshipResolution":"Resolve relative relationship URLs against relationshipURLBase; complete records and absolute URLs are in blocks.","title":"梯度更新和 EMA 更新作用在不同地方","url":"https://codingai-lec03.pages.dev/course/blocks/moco-two-parameter-updates.html","markdown":"https://codingai-lec03.pages.dev/course/blocks/moco-two-parameter-updates.md","pages":[{"id":"reading-opening","title":"梯度更新和 EMA 更新作用在不同地方","unit":"怎样增加候选，同时让历史表示仍可比较？","unitId":"i-dictionary-and-updates","chapter":"I","number":1}],"aliases":[{"oldId":"b-2808fb3f58b1b1b9fe7a07e8","targetId":"b-2808fb3f58b1b1b9fe7a07e8","page":"reading-opening","operation":"move","beforeSHA256":"5d38f6275eea2eb1ec2e89b6260882a83d6587cf230a21ebd2faae4a5a8b8745","afterSHA256":"5d38f6275eea2eb1ec2e89b6260882a83d6587cf230a21ebd2faae4a5a8b8745","reason":"Same source body in authored semantic page"},{"oldId":"b-04d1ac2b29751c57f86f9221","targetId":"b-04d1ac2b29751c57f86f9221","page":"reading-opening","operation":"move","beforeSHA256":"ee2944b46c15b76772509b769e97f697ec292a22473941d7beba2993b59b3ab2","afterSHA256":"ee2944b46c15b76772509b769e97f697ec292a22473941d7beba2993b59b3ab2","reason":"Same source body in authored semantic page"},{"oldId":"b-4d453564867bfd7bffb45409","targetId":"b-4d453564867bfd7bffb45409","page":"reading-opening","operation":"move","beforeSHA256":"6d156c8990596f280690c01b6aebc4611b362c5f7b77a9004dd2912d3a872081","afterSHA256":"6d156c8990596f280690c01b6aebc4611b362c5f7b77a9004dd2912d3a872081","reason":"Same source body in authored semantic page"},{"oldId":"b-35ef18676986236016956c3f","targetId":"b-35ef18676986236016956c3f","page":"reading-opening","operation":"move","beforeSHA256":"06c66c996376d8fa1b89b4e84770ae5de44e3397a0f0e5c2c6514d25104db6ff","afterSHA256":"06c66c996376d8fa1b89b4e84770ae5de44e3397a0f0e5c2c6514d25104db6ff","reason":"Same source body in authored semantic page"},{"oldId":"b-d2a2ae0ceb1a4aa085ac77fd","targetId":"b-d2a2ae0ceb1a4aa085ac77fd","page":"reading-opening","operation":"move","beforeSHA256":"8b1c1dbb533ad09c59f791550563c5ff85e712ef6972bca73782b226906d9666","afterSHA256":"8b1c1dbb533ad09c59f791550563c5ff85e712ef6972bca73782b226906d9666","reason":"Same source body in authored semantic page"},{"oldId":"b-2c53cb9b5fae055450180865","targetId":"b-2c53cb9b5fae055450180865","page":"reading-opening","operation":"rewrite","beforeSHA256":"92eb5ee568e038dd4cb86569d1b3e8e1111103eb98f4ddd74afb13d82e103763","afterSHA256":"292c52e63092340af5a2e710b0c0adee41a7effb092a50b402a9029efc2eab7e","reason":"Same source identity with figure semantics or reading context enriched"},{"oldId":"b-9717a0ca8337f2467be0ff8a","targetId":"b-9717a0ca8337f2467be0ff8a","page":"reading-opening","operation":"move","beforeSHA256":"2b9ccef7e1220e2250cceff1abd8bdd736b42f8da5a223f26b029f44b962e807","afterSHA256":"2b9ccef7e1220e2250cceff1abd8bdd736b42f8da5a223f26b029f44b962e807","reason":"Same source body in authored semantic page"}],"blocks":[{"id":"b-2808fb3f58b1b1b9fe7a07e8","type":"h1","scope":"moco-two-parameter-updates","url":"https://codingai-lec03.pages.dev/course/blocks/moco-two-parameter-updates.html#b-2808fb3f58b1b1b9fe7a07e8","content":"# 梯度更新和 EMA 更新作用在不同地方","sha256":"5d38f6275eea2eb1ec2e89b6260882a83d6587cf230a21ebd2faae4a5a8b8745","chapter":{"id":"I","title":"学习信号与表示"},"unit":{"id":"i-dictionary-and-updates","title":"怎样增加候选，同时让历史表示仍可比较？"},"page":{"id":"reading-opening","title":"梯度更新和 EMA 更新作用在不同地方","number":1},"sectionTitle":"梯度更新和 EMA 更新作用在不同地方","currentQuestion":"梯度更新和 EMA 更新作用在不同地方","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":null,"nextBlock":"b-04d1ac2b29751c57f86f9221","prerequisites":[{"id":"b-04d1ac2b29751c57f86f9221","url":"#b-04d1ac2b29751c57f86f9221","role":"problem-context"}],"relatedBlocks":[{"id":"b-2c53cb9b5fae055450180865","url":"#b-2c53cb9b5fae055450180865","relation":"figure-and-explanation"}]},{"id":"b-04d1ac2b29751c57f86f9221","type":"p","scope":"moco-two-parameter-updates","url":"https://codingai-lec03.pages.dev/course/blocks/moco-two-parameter-updates.html#b-04d1ac2b29751c57f86f9221","content":"两种参数更新解决不同任务：query学当前选择，key维持较平滑的编码规则。","sha256":"ee2944b46c15b76772509b769e97f697ec292a22473941d7beba2993b59b3ab2","chapter":{"id":"I","title":"学习信号与表示"},"unit":{"id":"i-dictionary-and-updates","title":"怎样增加候选，同时让历史表示仍可比较？"},"page":{"id":"reading-opening","title":"梯度更新和 EMA 更新作用在不同地方","number":1},"sectionTitle":"梯度更新和 EMA 更新作用在不同地方","currentQuestion":"梯度更新和 EMA 更新作用在不同地方","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-2808fb3f58b1b1b9fe7a07e8","nextBlock":"b-4d453564867bfd7bffb45409","prerequisites":[{"id":"b-2808fb3f58b1b1b9fe7a07e8","url":"#b-2808fb3f58b1b1b9fe7a07e8","role":"section-title"}],"relatedBlocks":[{"id":"b-2c53cb9b5fae055450180865","url":"#b-2c53cb9b5fae055450180865","relation":"figure-and-explanation"}]},{"id":"b-4d453564867bfd7bffb45409","type":"p","scope":"moco-two-parameter-updates","url":"https://codingai-lec03.pages.dev/course/blocks/moco-two-parameter-updates.html#b-4d453564867bfd7bffb45409","content":"query编码器从当前损失接收梯度，例如 $\\theta_q\\leftarrow\\theta_q-\\eta\\nabla_{\\theta_q}L$。key编码器不沿当前loss反传，而通过参数移动平均跟随query：$\\theta_k\\leftarrow m\\theta_k+(1-m)\\theta_q$。","sha256":"6d156c8990596f280690c01b6aebc4611b362c5f7b77a9004dd2912d3a872081","chapter":{"id":"I","title":"学习信号与表示"},"unit":{"id":"i-dictionary-and-updates","title":"怎样增加候选，同时让历史表示仍可比较？"},"page":{"id":"reading-opening","title":"梯度更新和 EMA 更新作用在不同地方","number":1},"sectionTitle":"梯度更新和 EMA 更新作用在不同地方","currentQuestion":"梯度更新和 EMA 更新作用在不同地方","sourceSlides":[],"knowledgeBlock":null,"formulas":[{"latex":"\\theta_q\\leftarrow\\theta_q-\\eta\\nabla_{\\theta_q}L","definitionContext":"梯度更新和 EMA 更新作用在不同地方"},{"latex":"\\theta_k\\leftarrow m\\theta_k+(1-m)\\theta_q","definitionContext":"梯度更新和 EMA 更新作用在不同地方"}],"previousBlock":"b-04d1ac2b29751c57f86f9221","nextBlock":"b-35ef18676986236016956c3f","prerequisites":[{"id":"b-2808fb3f58b1b1b9fe7a07e8","url":"#b-2808fb3f58b1b1b9fe7a07e8","role":"section-title"},{"id":"b-04d1ac2b29751c57f86f9221","url":"#b-04d1ac2b29751c57f86f9221","role":"problem-context"}],"relatedBlocks":[{"id":"b-2c53cb9b5fae055450180865","url":"#b-2c53cb9b5fae055450180865","relation":"figure-and-explanation"}]},{"id":"b-35ef18676986236016956c3f","type":"p","scope":"moco-two-parameter-updates","url":"https://codingai-lec03.pages.dev/course/blocks/moco-two-parameter-updates.html#b-35ef18676986236016956c3f","content":"令旧key参数为(0,2)，当前query参数为(2,4)，m=0.9，则key变为(0.2,2.2)。若query下一次仍不变，再更新得到(0.38,2.38)。平均的是网络参数，不是历史队列向量；也不同于优化器保存的梯度动量。","sha256":"06c66c996376d8fa1b89b4e84770ae5de44e3397a0f0e5c2c6514d25104db6ff","chapter":{"id":"I","title":"学习信号与表示"},"unit":{"id":"i-dictionary-and-updates","title":"怎样增加候选，同时让历史表示仍可比较？"},"page":{"id":"reading-opening","title":"梯度更新和 EMA 更新作用在不同地方","number":1},"sectionTitle":"梯度更新和 EMA 更新作用在不同地方","currentQuestion":"梯度更新和 EMA 更新作用在不同地方","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-4d453564867bfd7bffb45409","nextBlock":"b-d2a2ae0ceb1a4aa085ac77fd","prerequisites":[{"id":"b-2808fb3f58b1b1b9fe7a07e8","url":"#b-2808fb3f58b1b1b9fe7a07e8","role":"section-title"},{"id":"b-04d1ac2b29751c57f86f9221","url":"#b-04d1ac2b29751c57f86f9221","role":"problem-context"}],"relatedBlocks":[{"id":"b-2c53cb9b5fae055450180865","url":"#b-2c53cb9b5fae055450180865","relation":"figure-and-explanation"}]},{"id":"b-d2a2ae0ceb1a4aa085ac77fd","type":"p","scope":"moco-two-parameter-updates","url":"https://codingai-lec03.pages.dev/course/blocks/moco-two-parameter-updates.html#b-d2a2ae0ceb1a4aa085ac77fd","content":"较慢的key变化减轻历史向量的坐标系漂移。它并不把历史key全部校正为新参数下的值，也不保证任意队列容量都合适。m=0立即追上query，m=1彻底冻结；这两个极端帮助理解时间尺度，但通常不是希望采用的默认方案。","sha256":"8b1c1dbb533ad09c59f791550563c5ff85e712ef6972bca73782b226906d9666","chapter":{"id":"I","title":"学习信号与表示"},"unit":{"id":"i-dictionary-and-updates","title":"怎样增加候选，同时让历史表示仍可比较？"},"page":{"id":"reading-opening","title":"梯度更新和 EMA 更新作用在不同地方","number":1},"sectionTitle":"梯度更新和 EMA 更新作用在不同地方","currentQuestion":"梯度更新和 EMA 更新作用在不同地方","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-35ef18676986236016956c3f","nextBlock":"b-2c53cb9b5fae055450180865","prerequisites":[{"id":"b-2808fb3f58b1b1b9fe7a07e8","url":"#b-2808fb3f58b1b1b9fe7a07e8","role":"section-title"},{"id":"b-04d1ac2b29751c57f86f9221","url":"#b-04d1ac2b29751c57f86f9221","role":"problem-context"}],"relatedBlocks":[{"id":"b-2c53cb9b5fae055450180865","url":"#b-2c53cb9b5fae055450180865","relation":"figure-and-explanation"}]},{"id":"b-2c53cb9b5fae055450180865","type":"figure","scope":"moco-two-parameter-updates","url":"https://codingai-lec03.pages.dev/course/blocks/moco-two-parameter-updates.html#b-2c53cb9b5fae055450180865","content":"图内标签与数值：P16 学习表示机制图，状态 2；查询视图；查询编码器 θq；q；对比损失；键视图；键编码器 θk；k⁺；梯度更新；键分支停止梯度；参数 EMA；θk ← m θk + (1−m) θq；0.9 × 1 + 0.1 × 2 = 1.1；这里的平均对象是网络参数；m = 0.9 仅用于教学算例。\n\n梯度只到query；EMA从query参数流向key参数。 查看对应图解状态\n\n- 查询视图\n- 查询编码器 θq\n- q\n- 对比损失\n- 键视图\n- 键编码器 θk\n- k⁺\n- 梯度更新\n- 键分支停止梯度\n- 参数 EMA\n- θk ← m θk + (1−m) θq\n- 0.9 × 1 + 0.1 × 2 = 1.1\n- 这里的平均对象是网络参数；m = 0.9 仅用于教学算例。梯度只到query；EMA从query参数流向key参数。 [查看对应图解状态](/#/l3-r2-p16/1)[查看完整图解](/#/l3-r2-p16/1)","sha256":"292c52e63092340af5a2e710b0c0adee41a7effb092a50b402a9029efc2eab7e","chapter":{"id":"I","title":"学习信号与表示"},"unit":{"id":"i-dictionary-and-updates","title":"怎样增加候选，同时让历史表示仍可比较？"},"page":{"id":"reading-opening","title":"梯度更新和 EMA 更新作用在不同地方","number":1},"sectionTitle":"梯度更新和 EMA 更新作用在不同地方","currentQuestion":"梯度更新和 EMA 更新作用在不同地方","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"figure":{"caption":"梯度只到query；EMA从query参数流向key参数。 查看对应图解状态","labels":["查询视图","查询编码器 θq","q","对比损失","键视图","键编码器 θk","k⁺","梯度更新","键分支停止梯度","参数 EMA","θk ← m θk + (1−m) θq","0.9 × 1 + 0.1 × 2 = 1.1","这里的平均对象是网络参数；m = 0.9 仅用于教学算例。"],"description":"梯度只到query；EMA从query参数流向key参数。 查看对应图解状态查询视图查询编码器 θqq对比损失键视图键编码器 θkk⁺梯度更新键分支停止梯度参数 EMAθk ← m θk + (1−m) θq0.9 × 1 + 0.1 × 2 = 1.1这里的平均对象是网络参数；m = 0.9 仅用于教学算例。","evidenceIdentity":"梯度只到query；EMA从query参数流向key参数。 查看对应图解状态","images":[]},"previousBlock":"b-d2a2ae0ceb1a4aa085ac77fd","nextBlock":"b-9717a0ca8337f2467be0ff8a","prerequisites":[{"id":"b-2808fb3f58b1b1b9fe7a07e8","url":"#b-2808fb3f58b1b1b9fe7a07e8","role":"section-title"},{"id":"b-04d1ac2b29751c57f86f9221","url":"#b-04d1ac2b29751c57f86f9221","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-9717a0ca8337f2467be0ff8a","type":"h2","scope":"moco-two-parameter-updates","url":"https://codingai-lec03.pages.dev/course/blocks/moco-two-parameter-updates.html#b-9717a0ca8337f2467be0ff8a","content":"## 继续阅读与实验","sha256":"2b9ccef7e1220e2250cceff1abd8bdd736b42f8da5a223f26b029f44b962e807","chapter":{"id":"I","title":"学习信号与表示"},"unit":{"id":"i-dictionary-and-updates","title":"怎样增加候选，同时让历史表示仍可比较？"},"page":{"id":"reading-opening","title":"梯度更新和 EMA 更新作用在不同地方","number":1},"sectionTitle":"梯度更新和 EMA 更新作用在不同地方","currentQuestion":"梯度更新和 EMA 更新作用在不同地方","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-2c53cb9b5fae055450180865","nextBlock":null,"prerequisites":[{"id":"b-2808fb3f58b1b1b9fe7a07e8","url":"#b-2808fb3f58b1b1b9fe7a07e8","role":"section-title"},{"id":"b-04d1ac2b29751c57f86f9221","url":"#b-04d1ac2b29751c57f86f9221","role":"problem-context"}],"relatedBlocks":[{"id":"b-2c53cb9b5fae055450180865","url":"#b-2c53cb9b5fae055450180865","relation":"figure-and-explanation"}]}]}