{"version":2,"relationshipURLBase":"https://codingai-lec03.pages.dev/course/notebook-blocks/7ac2f2deac214e4a8db54c94134ff009.html","relationshipResolution":"Resolve relative relationship URLs against relationshipURLBase; complete records and absolute URLs are in blocks.","title":"3. 视觉特征怎样真正进入语言模型 · 实现代码","url":"https://codingai-lec03.pages.dev/course/notebook-blocks/7ac2f2deac214e4a8db54c94134ff009.html","markdown":"https://codingai-lec03.pages.dev/course/notebook-blocks/7ac2f2deac214e4a8db54c94134ff009.md","pages":[{"id":"reading-opening","title":"3. 视觉特征怎样真正进入语言模型 · 实现代码","unit":"3. 视觉特征怎样真正进入语言模型 · 实现代码","unitId":"notebook-blocks/7ac2f2deac214e4a8db54c94134ff009","chapter":"course","number":1}],"aliases":[{"oldId":"b-8bad8720d6aecabe733a0484","targetId":"b-8bad8720d6aecabe733a0484","page":"reading-opening","operation":"move","beforeSHA256":"db08960319942d9119c9cad2cf7d0b78a68c914f8e19ab3bd375531d27c96204","afterSHA256":"db08960319942d9119c9cad2cf7d0b78a68c914f8e19ab3bd375531d27c96204","reason":"Same source body in authored semantic page"},{"oldId":"b-789f826a421769c9b35d2f32","targetId":"b-789f826a421769c9b35d2f32","page":"reading-opening","operation":"move","beforeSHA256":"6f8bbb1b978c14466df4a89c06502f67b748846f749acd340607d57f662b154d","afterSHA256":"6f8bbb1b978c14466df4a89c06502f67b748846f749acd340607d57f662b154d","reason":"Same source body in authored semantic page"},{"oldId":"b-38b000e6fe91513bc8b1edd1","targetId":"b-38b000e6fe91513bc8b1edd1","page":"reading-opening","operation":"move","beforeSHA256":"e91f24e4e4ca40e2099a0d81d30db6568f4bf2074b499eeee6e3959617bd97ac","afterSHA256":"e91f24e4e4ca40e2099a0d81d30db6568f4bf2074b499eeee6e3959617bd97ac","reason":"Same source body in authored semantic page"},{"oldId":"b-5942920d203b2da1c0f0e991","targetId":"b-5942920d203b2da1c0f0e991","page":"reading-opening","operation":"move","beforeSHA256":"6a611aef342a7535362f318ee0c33b63170ce4ab853fa44a05f76a04873fd320","afterSHA256":"6a611aef342a7535362f318ee0c33b63170ce4ab853fa44a05f76a04873fd320","reason":"Same source body in authored semantic page"},{"oldId":"b-626bd93b4db2d52f9bf42454","targetId":"b-626bd93b4db2d52f9bf42454","page":"reading-opening","operation":"move","beforeSHA256":"43eae65db43f41518c687371f8a724225e24f260be6097247b06824ef2a230c2","afterSHA256":"43eae65db43f41518c687371f8a724225e24f260be6097247b06824ef2a230c2","reason":"Same source body in authored semantic page"}],"blocks":[{"id":"b-8bad8720d6aecabe733a0484","type":"p","scope":"document","url":"https://codingai-lec03.pages.dev/course/notebook-blocks/7ac2f2deac214e4a8db54c94134ff009.html#b-8bad8720d6aecabe733a0484","content":"[LAB 13 · 看图生成一句话：训练、解码与消融](/course/notebooks/3da47f3ee3554ee0a36e0bb6a2fdc61f.html)","sha256":"db08960319942d9119c9cad2cf7d0b78a68c914f8e19ab3bd375531d27c96204","chapter":{"id":"course","title":"3. 视觉特征怎样真正进入语言模型 · 实现代码"},"unit":{"id":"notebook-blocks/7ac2f2deac214e4a8db54c94134ff009","title":"3. 视觉特征怎样真正进入语言模型 · 实现代码"},"page":{"id":"reading-opening","title":"3. 视觉特征怎样真正进入语言模型 · 实现代码","number":1},"sectionTitle":"3. 视觉特征怎样真正进入语言模型 · 实现代码","currentQuestion":"3. 视觉特征怎样真正进入语言模型 · 实现代码","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":null,"nextBlock":"b-789f826a421769c9b35d2f32","prerequisites":[],"relatedBlocks":[]},{"id":"b-789f826a421769c9b35d2f32","type":"h1","scope":"document","url":"https://codingai-lec03.pages.dev/course/notebook-blocks/7ac2f2deac214e4a8db54c94134ff009.html#b-789f826a421769c9b35d2f32","content":"# 3. 视觉特征怎样真正进入语言模型 · 实现代码","sha256":"6f8bbb1b978c14466df4a89c06502f67b748846f749acd340607d57f662b154d","chapter":{"id":"course","title":"3. 视觉特征怎样真正进入语言模型 · 实现代码"},"unit":{"id":"notebook-blocks/7ac2f2deac214e4a8db54c94134ff009","title":"3. 视觉特征怎样真正进入语言模型 · 实现代码"},"page":{"id":"reading-opening","title":"3. 视觉特征怎样真正进入语言模型 · 实现代码","number":1},"sectionTitle":"3. 视觉特征怎样真正进入语言模型 · 实现代码","currentQuestion":"3. 视觉特征怎样真正进入语言模型 · 实现代码","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-8bad8720d6aecabe733a0484","nextBlock":"b-38b000e6fe91513bc8b1edd1","prerequisites":[{"id":"b-8bad8720d6aecabe733a0484","url":"#b-8bad8720d6aecabe733a0484","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-38b000e6fe91513bc8b1edd1","type":"p","scope":"document","url":"https://codingai-lec03.pages.dev/course/notebook-blocks/7ac2f2deac214e4a8db54c94134ff009.html#b-38b000e6fe91513bc8b1edd1","content":"按问题继续","sha256":"e91f24e4e4ca40e2099a0d81d30db6568f4bf2074b499eeee6e3959617bd97ac","chapter":{"id":"course","title":"3. 视觉特征怎样真正进入语言模型 · 实现代码"},"unit":{"id":"notebook-blocks/7ac2f2deac214e4a8db54c94134ff009","title":"3. 视觉特征怎样真正进入语言模型 · 实现代码"},"page":{"id":"reading-opening","title":"3. 视觉特征怎样真正进入语言模型 · 实现代码","number":1},"sectionTitle":"3. 视觉特征怎样真正进入语言模型 · 实现代码","currentQuestion":"3. 视觉特征怎样真正进入语言模型 · 实现代码","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-789f826a421769c9b35d2f32","nextBlock":"b-5942920d203b2da1c0f0e991","prerequisites":[{"id":"b-789f826a421769c9b35d2f32","url":"#b-789f826a421769c9b35d2f32","role":"section-title"},{"id":"b-8bad8720d6aecabe733a0484","url":"#b-8bad8720d6aecabe733a0484","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-5942920d203b2da1c0f0e991","type":"p","scope":"document","url":"https://codingai-lec03.pages.dev/course/notebook-blocks/7ac2f2deac214e4a8db54c94134ff009.html#b-5942920d203b2da1c0f0e991","content":"真实CNN–GRU条件训练与greedy/beam生成、遮图/错图消融","sha256":"6a611aef342a7535362f318ee0c33b63170ce4ab853fa44a05f76a04873fd320","chapter":{"id":"course","title":"3. 视觉特征怎样真正进入语言模型 · 实现代码"},"unit":{"id":"notebook-blocks/7ac2f2deac214e4a8db54c94134ff009","title":"3. 视觉特征怎样真正进入语言模型 · 实现代码"},"page":{"id":"reading-opening","title":"3. 视觉特征怎样真正进入语言模型 · 实现代码","number":1},"sectionTitle":"3. 视觉特征怎样真正进入语言模型 · 实现代码","currentQuestion":"3. 视觉特征怎样真正进入语言模型 · 实现代码","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-38b000e6fe91513bc8b1edd1","nextBlock":"b-626bd93b4db2d52f9bf42454","prerequisites":[{"id":"b-789f826a421769c9b35d2f32","url":"#b-789f826a421769c9b35d2f32","role":"section-title"},{"id":"b-8bad8720d6aecabe733a0484","url":"#b-8bad8720d6aecabe733a0484","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-626bd93b4db2d52f9bf42454","type":"pre","scope":"document","url":"https://codingai-lec03.pages.dev/course/notebook-blocks/7ac2f2deac214e4a8db54c94134ff009.html#b-626bd93b4db2d52f9bf42454","content":"```\nclass Captioner(nn.Module):\n    def __init__(self):\n        super().__init__()\n        self.cnn = nn.Sequential(nn.Conv2d(3,12,3,padding=1),nn.ReLU(),nn.MaxPool2d(2),nn.Conv2d(12,24,3,padding=1),nn.ReLU(),nn.MaxPool2d(2),nn.Flatten(),nn.Linear(24*8*8,48),nn.LayerNorm(48),nn.ReLU())\n        self.word = nn.Embedding(len(VOCAB),24,padding_idx=PAD)\n        self.initial = nn.Linear(48,64)\n        self.rnn = nn.GRU(24+48,64,batch_first=True)\n        self.output = nn.Linear(64,len(VOCAB))\n    def encode(self, pixels):\n        return self.cnn(pixels - 0.5)\n    def forward(self, pixels, prefix):\n        visual = self.encode(pixels)\n        features = visual[:,None,:].expand(-1,prefix.shape[1],-1)\n        hidden,_ = self.rnn(torch.cat((self.word(prefix),features),-1),self.initial(visual)[None])\n        return self.output(hidden)\n\nmodel = Captioner()\noptimizer = torch.optim.Adam(model.parameters(),lr=.001)\ncurve=[]\nfor epoch in range(45):\n    model.train(); order=torch.randperm(len(Xtrain)); total=0.; tokens=0\n    for ix in order.split(60):\n        logits=model(Xtrain[ix],Ytrain[ix,:-1])\n        targets=Ytrain[ix,1:]\n        # Sum over real targets then divide by their count, not padded positions.\n        loss=F.cross_entropy(logits.reshape(-1,len(VOCAB)),targets.reshape(-1),ignore_index=PAD,reduction='sum')/targets.ne(PAD).sum()\n        optimizer.zero_grad(); loss.backward(); nn.utils.clip_grad_norm_(model.parameters(),5); optimizer.step()\n        count=int(targets.ne(PAD).sum()); total+=float(loss.detach())*count; tokens+=count\n    model.eval()\n    with torch.no_grad():\n        logits=model(Xtest,Ytest[:,:-1]); gold=Ytest[:,1:]\n        test_loss=float(F.cross_entropy(logits.reshape(-1,len(VOCAB)),gold.reshape(-1),ignore_index=PAD))\n    curve.append({'epoch':epoch+1,'train_token_ce':total/tokens,'heldout_teacher_forced_ce':test_loss})\n    if epoch in [0,5,11,23,44]: print(curve[-1])\ntorch.save({'state_dict':model.state_dict(),'vocab':VOCAB,'seed':1303},OUT/'lab13-caption-weights.pt')\n```","sha256":"43eae65db43f41518c687371f8a724225e24f260be6097247b06824ef2a230c2","chapter":{"id":"course","title":"3. 视觉特征怎样真正进入语言模型 · 实现代码"},"unit":{"id":"notebook-blocks/7ac2f2deac214e4a8db54c94134ff009","title":"3. 视觉特征怎样真正进入语言模型 · 实现代码"},"page":{"id":"reading-opening","title":"3. 视觉特征怎样真正进入语言模型 · 实现代码","number":1},"sectionTitle":"3. 视觉特征怎样真正进入语言模型 · 实现代码","currentQuestion":"3. 视觉特征怎样真正进入语言模型 · 实现代码","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-5942920d203b2da1c0f0e991","nextBlock":null,"prerequisites":[{"id":"b-789f826a421769c9b35d2f32","url":"#b-789f826a421769c9b35d2f32","role":"section-title"},{"id":"b-8bad8720d6aecabe733a0484","url":"#b-8bad8720d6aecabe733a0484","role":"problem-context"}],"relatedBlocks":[]}]}