{"version":2,"relationshipURLBase":"https://codingai-lec03.pages.dev/course/blocks/training-systems-later-reading.html","relationshipResolution":"Resolve relative relationship URLs against relationshipURLBase; complete records and absolute URLs are in blocks.","title":"第六讲预备阅读：怎样把一次更新分给多设备","url":"https://codingai-lec03.pages.dev/course/blocks/training-systems-later-reading.html","markdown":"https://codingai-lec03.pages.dev/course/blocks/training-systems-later-reading.md","pages":[{"id":"reading-opening","title":"第六讲预备阅读：怎样把一次更新分给多设备","unit":"表示学习的选读问题","unitId":"i-extensions","chapter":"I","number":1}],"aliases":[{"oldId":"b-5c3eaf96b5f7555ccedbf929","targetId":"b-5c3eaf96b5f7555ccedbf929","page":"reading-opening","operation":"move","beforeSHA256":"fa54bdba828322494447f1874c09432a1c81be16823a7b2299bd55538fe8117c","afterSHA256":"fa54bdba828322494447f1874c09432a1c81be16823a7b2299bd55538fe8117c","reason":"Same source body in authored semantic page"},{"oldId":"b-76d2a6939fd9143217a78a2e","targetId":"b-76d2a6939fd9143217a78a2e","page":"reading-opening","operation":"move","beforeSHA256":"513b4dde07c19aa85a702ba5af2c592cb8b134338c12e49e6c6f6faf9adf92ed","afterSHA256":"513b4dde07c19aa85a702ba5af2c592cb8b134338c12e49e6c6f6faf9adf92ed","reason":"Same source body in authored semantic page"},{"oldId":"b-c375a51e29c12be7c0d2b8c8","targetId":"b-c375a51e29c12be7c0d2b8c8","page":"reading-opening","operation":"move","beforeSHA256":"a4c7eab21c83eccbcaefe98f60454f108c008c8b3fa6594af5b233ae84b8b0f1","afterSHA256":"a4c7eab21c83eccbcaefe98f60454f108c008c8b3fa6594af5b233ae84b8b0f1","reason":"Same source body in authored semantic page"},{"oldId":"b-a2b2ceef23542ca04a85499c","targetId":"b-a2b2ceef23542ca04a85499c","page":"reading-opening","operation":"move","beforeSHA256":"c42c6e5e5f367f9e1af5d2787a3ccb7b1d44c677ce46db623b1cdfc53a1bb235","afterSHA256":"c42c6e5e5f367f9e1af5d2787a3ccb7b1d44c677ce46db623b1cdfc53a1bb235","reason":"Same source body in authored semantic page"},{"oldId":"b-eaac5c92ffeedc2ddb0ca70c","targetId":"b-eaac5c92ffeedc2ddb0ca70c","page":"reading-opening","operation":"move","beforeSHA256":"fd9f47b23ac02134f8ba33291deefe146937e204fae8c8287e434ee874f9b302","afterSHA256":"fd9f47b23ac02134f8ba33291deefe146937e204fae8c8287e434ee874f9b302","reason":"Same source body in authored semantic page"},{"oldId":"b-559b9de1edc4287e6e9cbe60","targetId":"b-559b9de1edc4287e6e9cbe60","page":"reading-opening","operation":"move","beforeSHA256":"32927a95e6f350296075848941a2c2ad0e6cebecaf1e8f01271b917bc222ece4","afterSHA256":"32927a95e6f350296075848941a2c2ad0e6cebecaf1e8f01271b917bc222ece4","reason":"Same source body in authored semantic page"},{"oldId":"b-3da305c7b9793f067af85dcc","targetId":"b-3da305c7b9793f067af85dcc","page":"reading-opening","operation":"move","beforeSHA256":"2b9ccef7e1220e2250cceff1abd8bdd736b42f8da5a223f26b029f44b962e807","afterSHA256":"2b9ccef7e1220e2250cceff1abd8bdd736b42f8da5a223f26b029f44b962e807","reason":"Same source body in authored semantic page"}],"blocks":[{"id":"b-5c3eaf96b5f7555ccedbf929","type":"h1","scope":"training-systems-later-reading","url":"https://codingai-lec03.pages.dev/course/blocks/training-systems-later-reading.html#b-5c3eaf96b5f7555ccedbf929","content":"# 第六讲预备阅读：怎样把一次更新分给多设备","sha256":"fa54bdba828322494447f1874c09432a1c81be16823a7b2299bd55538fe8117c","chapter":{"id":"I","title":"学习信号与表示"},"unit":{"id":"i-extensions","title":"表示学习的选读问题"},"page":{"id":"reading-opening","title":"第六讲预备阅读：怎样把一次更新分给多设备","number":1},"sectionTitle":"第六讲预备阅读：怎样把一次更新分给多设备","currentQuestion":"第六讲预备阅读：怎样把一次更新分给多设备","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":null,"nextBlock":"b-76d2a6939fd9143217a78a2e","prerequisites":[{"id":"b-76d2a6939fd9143217a78a2e","url":"#b-76d2a6939fd9143217a78a2e","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-76d2a6939fd9143217a78a2e","type":"p","scope":"training-systems-later-reading","url":"https://codingai-lec03.pages.dev/course/blocks/training-systems-later-reading.html#b-76d2a6939fd9143217a78a2e","content":"后续课程预备或按需选读，不列为本讲必讲内容。","sha256":"513b4dde07c19aa85a702ba5af2c592cb8b134338c12e49e6c6f6faf9adf92ed","chapter":{"id":"I","title":"学习信号与表示"},"unit":{"id":"i-extensions","title":"表示学习的选读问题"},"page":{"id":"reading-opening","title":"第六讲预备阅读：怎样把一次更新分给多设备","number":1},"sectionTitle":"第六讲预备阅读：怎样把一次更新分给多设备","currentQuestion":"第六讲预备阅读：怎样把一次更新分给多设备","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-5c3eaf96b5f7555ccedbf929","nextBlock":"b-c375a51e29c12be7c0d2b8c8","prerequisites":[{"id":"b-5c3eaf96b5f7555ccedbf929","url":"#b-5c3eaf96b5f7555ccedbf929","role":"section-title"}],"relatedBlocks":[]},{"id":"b-c375a51e29c12be7c0d2b8c8","type":"p","scope":"training-systems-later-reading","url":"https://codingai-lec03.pages.dev/course/blocks/training-systems-later-reading.html#b-c375a51e29c12be7c0d2b8c8","content":"资源约束在本讲是方法选择的一个条件；真正需要回访它的是批内候选、MoCo缓存与MAE可见块。","sha256":"a4c7eab21c83eccbcaefe98f60454f108c008c8b3fa6594af5b233ae84b8b0f1","chapter":{"id":"I","title":"学习信号与表示"},"unit":{"id":"i-extensions","title":"表示学习的选读问题"},"page":{"id":"reading-opening","title":"第六讲预备阅读：怎样把一次更新分给多设备","number":1},"sectionTitle":"第六讲预备阅读：怎样把一次更新分给多设备","currentQuestion":"第六讲预备阅读：怎样把一次更新分给多设备","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-76d2a6939fd9143217a78a2e","nextBlock":"b-a2b2ceef23542ca04a85499c","prerequisites":[{"id":"b-5c3eaf96b5f7555ccedbf929","url":"#b-5c3eaf96b5f7555ccedbf929","role":"section-title"},{"id":"b-76d2a6939fd9143217a78a2e","url":"#b-76d2a6939fd9143217a78a2e","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-a2b2ceef23542ca04a85499c","type":"p","scope":"training-systems-later-reading","url":"https://codingai-lec03.pages.dev/course/blocks/training-systems-later-reading.html#b-a2b2ceef23542ca04a85499c","content":"本段是后续课程预备阅读，不属于Lecture03当堂必讲。它承接“训练目标要能实施”，帮助有需要的学生定位旧材料，而不是要求理解MoCo前先学分布式训练。","sha256":"c42c6e5e5f367f9e1af5d2787a3ccb7b1d44c677ce46db623b1cdfc53a1bb235","chapter":{"id":"I","title":"学习信号与表示"},"unit":{"id":"i-extensions","title":"表示学习的选读问题"},"page":{"id":"reading-opening","title":"第六讲预备阅读：怎样把一次更新分给多设备","number":1},"sectionTitle":"第六讲预备阅读：怎样把一次更新分给多设备","currentQuestion":"第六讲预备阅读：怎样把一次更新分给多设备","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-c375a51e29c12be7c0d2b8c8","nextBlock":"b-eaac5c92ffeedc2ddb0ca70c","prerequisites":[{"id":"b-5c3eaf96b5f7555ccedbf929","url":"#b-5c3eaf96b5f7555ccedbf929","role":"section-title"},{"id":"b-76d2a6939fd9143217a78a2e","url":"#b-76d2a6939fd9143217a78a2e","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-eaac5c92ffeedc2ddb0ca70c","type":"p","scope":"training-systems-later-reading","url":"https://codingai-lec03.pages.dev/course/blocks/training-systems-later-reading.html#b-eaac5c92ffeedc2ddb0ca70c","content":"最小例子：两个设备从同一参数(1,2)开始，等大小本地批次给平均梯度(2,4)和(6,0)。合并梯度为(4,2)，学习率0.1，所有副本都更新为(0.6,1.8)。若本地样本数2:6，就应按样本数加权，得到梯度(5,1)、参数(0.5,1.9)。关键是所有设备执行同一全局目标的一次更新，而不是各自学成不同模型。","sha256":"fd9f47b23ac02134f8ba33291deefe146937e204fae8c8287e434ee874f9b302","chapter":{"id":"I","title":"学习信号与表示"},"unit":{"id":"i-extensions","title":"表示学习的选读问题"},"page":{"id":"reading-opening","title":"第六讲预备阅读：怎样把一次更新分给多设备","number":1},"sectionTitle":"第六讲预备阅读：怎样把一次更新分给多设备","currentQuestion":"第六讲预备阅读：怎样把一次更新分给多设备","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-a2b2ceef23542ca04a85499c","nextBlock":"b-559b9de1edc4287e6e9cbe60","prerequisites":[{"id":"b-5c3eaf96b5f7555ccedbf929","url":"#b-5c3eaf96b5f7555ccedbf929","role":"section-title"},{"id":"b-76d2a6939fd9143217a78a2e","url":"#b-76d2a6939fd9143217a78a2e","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-559b9de1edc4287e6e9cbe60","type":"p","scope":"training-systems-later-reading","url":"https://codingai-lec03.pages.dev/course/blocks/training-systems-later-reading.html#b-559b9de1edc4287e6e9cbe60","content":"进一步的状态分片讨论谁保管权重、梯度和优化器状态，通信讨论计算前怎样取齐需要的数据。这些旧算例仍有价值，放在后续阅读入口，避免和本讲“训练答案从哪里来”的主问题混在同一默认顺序中。","sha256":"32927a95e6f350296075848941a2c2ad0e6cebecaf1e8f01271b917bc222ece4","chapter":{"id":"I","title":"学习信号与表示"},"unit":{"id":"i-extensions","title":"表示学习的选读问题"},"page":{"id":"reading-opening","title":"第六讲预备阅读：怎样把一次更新分给多设备","number":1},"sectionTitle":"第六讲预备阅读：怎样把一次更新分给多设备","currentQuestion":"第六讲预备阅读：怎样把一次更新分给多设备","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-eaac5c92ffeedc2ddb0ca70c","nextBlock":"b-3da305c7b9793f067af85dcc","prerequisites":[{"id":"b-5c3eaf96b5f7555ccedbf929","url":"#b-5c3eaf96b5f7555ccedbf929","role":"section-title"},{"id":"b-76d2a6939fd9143217a78a2e","url":"#b-76d2a6939fd9143217a78a2e","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-3da305c7b9793f067af85dcc","type":"h2","scope":"training-systems-later-reading","url":"https://codingai-lec03.pages.dev/course/blocks/training-systems-later-reading.html#b-3da305c7b9793f067af85dcc","content":"## 继续阅读与实验","sha256":"2b9ccef7e1220e2250cceff1abd8bdd736b42f8da5a223f26b029f44b962e807","chapter":{"id":"I","title":"学习信号与表示"},"unit":{"id":"i-extensions","title":"表示学习的选读问题"},"page":{"id":"reading-opening","title":"第六讲预备阅读：怎样把一次更新分给多设备","number":1},"sectionTitle":"第六讲预备阅读：怎样把一次更新分给多设备","currentQuestion":"第六讲预备阅读：怎样把一次更新分给多设备","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-559b9de1edc4287e6e9cbe60","nextBlock":null,"prerequisites":[{"id":"b-5c3eaf96b5f7555ccedbf929","url":"#b-5c3eaf96b5f7555ccedbf929","role":"section-title"},{"id":"b-76d2a6939fd9143217a78a2e","url":"#b-76d2a6939fd9143217a78a2e","role":"problem-context"}],"relatedBlocks":[]}]}