{"version":2,"relationshipURLBase":"https://codingai-lec03.pages.dev/course/notebook-blocks/ebf4defb3d804d1084f0dd9f3a91cfe9.html","relationshipResolution":"Resolve relative relationship URLs against relationshipURLBase; complete records and absolute URLs are in blocks.","title":"为什么队列需要缓慢变化的键编码器？ · 实现代码","url":"https://codingai-lec03.pages.dev/course/notebook-blocks/ebf4defb3d804d1084f0dd9f3a91cfe9.html","markdown":"https://codingai-lec03.pages.dev/course/notebook-blocks/ebf4defb3d804d1084f0dd9f3a91cfe9.md","pages":[{"id":"reading-opening","title":"为什么队列需要缓慢变化的键编码器？ · 实现代码","unit":"为什么队列需要缓慢变化的键编码器？ · 实现代码","unitId":"notebook-blocks/ebf4defb3d804d1084f0dd9f3a91cfe9","chapter":"course","number":1}],"aliases":[{"oldId":"b-2d2c53738569f3f3b5b25d82","targetId":"b-2d2c53738569f3f3b5b25d82","page":"reading-opening","operation":"move","beforeSHA256":"88a04c9ecb442ccaa48afb715979a4f7ec256d11b69e583a238c98ab9d60e977","afterSHA256":"88a04c9ecb442ccaa48afb715979a4f7ec256d11b69e583a238c98ab9d60e977","reason":"Same source body in authored semantic page"},{"oldId":"b-8e5f28e8f1d14024584779a8","targetId":"b-8e5f28e8f1d14024584779a8","page":"reading-opening","operation":"move","beforeSHA256":"d87a988a342fbc5a7a619740d26850eb951d753aab0b2255b4fee21a41a004c6","afterSHA256":"d87a988a342fbc5a7a619740d26850eb951d753aab0b2255b4fee21a41a004c6","reason":"Same source body in authored semantic page"},{"oldId":"b-19d6d17ac494ac88b283dcce","targetId":"b-19d6d17ac494ac88b283dcce","page":"reading-opening","operation":"move","beforeSHA256":"e91f24e4e4ca40e2099a0d81d30db6568f4bf2074b499eeee6e3959617bd97ac","afterSHA256":"e91f24e4e4ca40e2099a0d81d30db6568f4bf2074b499eeee6e3959617bd97ac","reason":"Same source body in authored semantic page"},{"oldId":"b-55b935f0f8afddf256c2a003","targetId":"b-55b935f0f8afddf256c2a003","page":"reading-opening","operation":"move","beforeSHA256":"2a269d38480454fbc554b17f926927738416555e36045ec5fe468ba056454f5d","afterSHA256":"2a269d38480454fbc554b17f926927738416555e36045ec5fe468ba056454f5d","reason":"Same source body in authored semantic page"},{"oldId":"b-a7242bfbd843db0bf4fe888a","targetId":"b-a7242bfbd843db0bf4fe888a","page":"reading-opening","operation":"move","beforeSHA256":"32eb91a80268f344f0da944c3e9d486d24441f4849e2c469cdfe9cc178fef276","afterSHA256":"32eb91a80268f344f0da944c3e9d486d24441f4849e2c469cdfe9cc178fef276","reason":"Same source body in authored semantic page"}],"blocks":[{"id":"b-2d2c53738569f3f3b5b25d82","type":"p","scope":"document","url":"https://codingai-lec03.pages.dev/course/notebook-blocks/ebf4defb3d804d1084f0dd9f3a91cfe9.html#b-2d2c53738569f3f3b5b25d82","content":"[LAB 07 · MoCo：历史 key、队列与两种更新](/course/notebooks/0d1b8f6fbf6a476c93ffcc169f2ec1f4.html)","sha256":"88a04c9ecb442ccaa48afb715979a4f7ec256d11b69e583a238c98ab9d60e977","chapter":{"id":"course","title":"为什么队列需要缓慢变化的键编码器？ · 实现代码"},"unit":{"id":"notebook-blocks/ebf4defb3d804d1084f0dd9f3a91cfe9","title":"为什么队列需要缓慢变化的键编码器？ · 实现代码"},"page":{"id":"reading-opening","title":"为什么队列需要缓慢变化的键编码器？ · 实现代码","number":1},"sectionTitle":"为什么队列需要缓慢变化的键编码器？ · 实现代码","currentQuestion":"为什么队列需要缓慢变化的键编码器？ · 实现代码","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":null,"nextBlock":"b-8e5f28e8f1d14024584779a8","prerequisites":[],"relatedBlocks":[]},{"id":"b-8e5f28e8f1d14024584779a8","type":"h1","scope":"document","url":"https://codingai-lec03.pages.dev/course/notebook-blocks/ebf4defb3d804d1084f0dd9f3a91cfe9.html#b-8e5f28e8f1d14024584779a8","content":"# 为什么队列需要缓慢变化的键编码器？ · 实现代码","sha256":"d87a988a342fbc5a7a619740d26850eb951d753aab0b2255b4fee21a41a004c6","chapter":{"id":"course","title":"为什么队列需要缓慢变化的键编码器？ · 实现代码"},"unit":{"id":"notebook-blocks/ebf4defb3d804d1084f0dd9f3a91cfe9","title":"为什么队列需要缓慢变化的键编码器？ · 实现代码"},"page":{"id":"reading-opening","title":"为什么队列需要缓慢变化的键编码器？ · 实现代码","number":1},"sectionTitle":"为什么队列需要缓慢变化的键编码器？ · 实现代码","currentQuestion":"为什么队列需要缓慢变化的键编码器？ · 实现代码","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-2d2c53738569f3f3b5b25d82","nextBlock":"b-19d6d17ac494ac88b283dcce","prerequisites":[{"id":"b-2d2c53738569f3f3b5b25d82","url":"#b-2d2c53738569f3f3b5b25d82","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-19d6d17ac494ac88b283dcce","type":"p","scope":"document","url":"https://codingai-lec03.pages.dev/course/notebook-blocks/ebf4defb3d804d1084f0dd9f3a91cfe9.html#b-19d6d17ac494ac88b283dcce","content":"按问题继续","sha256":"e91f24e4e4ca40e2099a0d81d30db6568f4bf2074b499eeee6e3959617bd97ac","chapter":{"id":"course","title":"为什么队列需要缓慢变化的键编码器？ · 实现代码"},"unit":{"id":"notebook-blocks/ebf4defb3d804d1084f0dd9f3a91cfe9","title":"为什么队列需要缓慢变化的键编码器？ · 实现代码"},"page":{"id":"reading-opening","title":"为什么队列需要缓慢变化的键编码器？ · 实现代码","number":1},"sectionTitle":"为什么队列需要缓慢变化的键编码器？ · 实现代码","currentQuestion":"为什么队列需要缓慢变化的键编码器？ · 实现代码","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-8e5f28e8f1d14024584779a8","nextBlock":"b-55b935f0f8afddf256c2a003","prerequisites":[{"id":"b-8e5f28e8f1d14024584779a8","url":"#b-8e5f28e8f1d14024584779a8","role":"section-title"},{"id":"b-2d2c53738569f3f3b5b25d82","url":"#b-2d2c53738569f3f3b5b25d82","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-55b935f0f8afddf256c2a003","type":"p","scope":"document","url":"https://codingai-lec03.pages.dev/course/notebook-blocks/ebf4defb3d804d1084f0dd9f3a91cfe9.html#b-55b935f0f8afddf256c2a003","content":"真实FIFO键、EMA、梯度更新及低于随机基线的保留结果","sha256":"2a269d38480454fbc554b17f926927738416555e36045ec5fe468ba056454f5d","chapter":{"id":"course","title":"为什么队列需要缓慢变化的键编码器？ · 实现代码"},"unit":{"id":"notebook-blocks/ebf4defb3d804d1084f0dd9f3a91cfe9","title":"为什么队列需要缓慢变化的键编码器？ · 实现代码"},"page":{"id":"reading-opening","title":"为什么队列需要缓慢变化的键编码器？ · 实现代码","number":1},"sectionTitle":"为什么队列需要缓慢变化的键编码器？ · 实现代码","currentQuestion":"为什么队列需要缓慢变化的键编码器？ · 实现代码","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-19d6d17ac494ac88b283dcce","nextBlock":"b-a7242bfbd843db0bf4fe888a","prerequisites":[{"id":"b-8e5f28e8f1d14024584779a8","url":"#b-8e5f28e8f1d14024584779a8","role":"section-title"},{"id":"b-2d2c53738569f3f3b5b25d82","url":"#b-2d2c53738569f3f3b5b25d82","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-a7242bfbd843db0bf4fe888a","type":"pre","scope":"document","url":"https://codingai-lec03.pages.dev/course/notebook-blocks/ebf4defb3d804d1084f0dd9f3a91cfe9.html#b-a7242bfbd843db0bf4fe888a","content":"```\ndef moco_warmup():\n    q=np.array([1.,0.]); keys=np.array([[.8,.2],[0.,1.],[-1.,0.]])\n    keys/=np.linalg.norm(keys,axis=1,keepdims=True)\n    probs={}\n    for T in [.1,.5]:\n        s=keys@q/T; p=np.exp(s-s.max());p/=p.sum();probs[str(T)]=p.tolist()\n    queue=['a','b','c','d']; queue=(queue+['e','f'])[-4:]\n    ema=.9*np.array([0.,2.])+.1*np.array([2.,4.])\n    assert queue==['c','d','e','f']; assert np.allclose(ema,[.2,2.2])\n    return {'temperature_probabilities':probs,'FIFO':queue,'EMA':ema.tolist()}\n\ndef train_moco(initial,x,train,epochs=15):\n    # Small genuine image implementation: query grads, detached EMA keys, FIFO negatives.\n    torch.manual_seed(SEED+2)\n    query=copy.deepcopy(initial); key=copy.deepcopy(initial)\n    for p in key.parameters(): p.requires_grad=False\n    opt=torch.optim.Adam(query.parameters(),lr=.002)\n    queue=torch.empty((0,24)); queue_ids=torch.empty(0,dtype=torch.long)\n    history=[]; trace=[]; m=.95; capacity=240\n    before=next(query.encoder.parameters()).detach().clone()\n    for epoch in range(epochs):\n        losses=[]\n        for ids in torch.as_tensor(train)[torch.randperm(len(train))].split(60):\n            a,b=views(x[ids]),views(x[ids])\n            q=F.normalize(query(a)[1],dim=1)\n            with torch.no_grad():\n                for pk,pq in zip(key.parameters(),query.parameters()):pk.mul_(m).add_(pq,alpha=1-m)\n                k=F.normalize(key(b)[1],dim=1)\n            # Warmup uses detached current keys. Later batches use a real FIFO dictionary.\n            if not len(queue):\n                logits=q@k.T/.2; labels=torch.arange(len(ids))\n                loss=F.cross_entropy(logits,labels)\n            else:\n                positive=(q*k).sum(1,keepdim=True)\n                negative=q@queue.T\n                # A source already in queue is NOT treated as its own negative.\n                negative=negative.masked_fill(ids[:,None]==queue_ids[None,:],-1e9)\n                logits=torch.cat([positive,negative],dim=1)/.2\n                loss=F.cross_entropy(logits,torch.zeros(len(ids),dtype=torch.long))\n            opt.zero_grad();loss.backward();opt.step()\n            with torch.no_grad():\n                queue=torch.cat([queue,k])[-capacity:];queue_ids=torch.cat([queue_ids,ids])[-capacity:]\n            losses.append(float(loss.detach()))\n            if len(trace)<5:trace.append({'step':len(trace),'queue_size':len(queue),'first_source_id':int(queue_ids[0]),'last_source_id':int(queue_ids[-1])})\n        history.append(float(np.mean(losses)))\n    assert all(p.grad is None for p in key.parameters())\n    return query,key,queue,queue_ids,history,trace,float((next(query.encoder.parameters()).detach()-before).norm())\n```","sha256":"32eb91a80268f344f0da944c3e9d486d24441f4849e2c469cdfe9cc178fef276","chapter":{"id":"course","title":"为什么队列需要缓慢变化的键编码器？ · 实现代码"},"unit":{"id":"notebook-blocks/ebf4defb3d804d1084f0dd9f3a91cfe9","title":"为什么队列需要缓慢变化的键编码器？ · 实现代码"},"page":{"id":"reading-opening","title":"为什么队列需要缓慢变化的键编码器？ · 实现代码","number":1},"sectionTitle":"为什么队列需要缓慢变化的键编码器？ · 实现代码","currentQuestion":"为什么队列需要缓慢变化的键编码器？ · 实现代码","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-55b935f0f8afddf256c2a003","nextBlock":null,"prerequisites":[{"id":"b-8e5f28e8f1d14024584779a8","url":"#b-8e5f28e8f1d14024584779a8","role":"section-title"},{"id":"b-2d2c53738569f3f3b5b25d82","url":"#b-2d2c53738569f3f3b5b25d82","role":"problem-context"}],"relatedBlocks":[]}]}