Blogs/Own scribbling
- Neural nets tips and tricks
- WeightNorm vs LayerNorm vs BatchNorm (Key thing: in deep learning anything is legal as long as it is differentiable, and all these normalization operations are differentiable.)
- Self-supervised learning reading list Blog 1 Blog 2
- Some notes on freezing: when fine-tuning, simultaneously training entire network for initial epochs can lead to catastrophic forgetting especially if some new layers in the model have been introduced or some weights have been randomly initialized. Hence it is advised to for initial one or two epochs, we should freeze the weights of the pretrained layers and train only the new randomly initialized weights.
Summary of papers (mainly in NLP and Machine Learning) that I read:
- Batch Normalization (The summation sign over the mini-batch size is important while computing the derivatives of mean and variance of mini-batch, which in turn are required for computing the derivative with respect to a particular x_i, more suitable for conv nets, but with recurrent nets a problem that the test sequence may be larger than any of the training sequence, then how to normalize? Goodfellow found that using something called as virtual batch normalization is essential for training GANs to be able to generate good images. References for batch norm: Goodfellows lecture for discussion(link from the book website), the deep learning book, and yannic kilcher's review of the paper)
- Layer Normalization (overcomes the shortcomings of batch norm for RNNs, though for CNN batchnorm is still more effective, basically helps in making the model more robust to scaling and initializations of parameters, and the input values (invariant to weight rescaling and recentering and input rescaling and recentering), while solving internal covariate shift. A nice table given in this paper comparing layernorm, batchnorm and weightnorm. If layer normalization is working on the outputs from a convolution layer, the math has to be modified slightly since it does not make sense to group all the elements from three distinct channels together and compute the mean and variance. Each channel is considered as an “independent” sample and all the normalization was done for that specific channel only within the sample.)
- Weight Normalization (aims to alleviate the significance of batch sizes as a design decision as in batch normalization, and claims to be faster for CNNs, and applicable to noise sensitive applications like deep reinforcement learning, but is affected by the way parameters are initialized.)
- What BERT LEARNS (Attention heads show some general patterns like attending to the previous and the next word, a large number of heads attend to the SEP token, especially in the middle layers, while the higher layers attend to periods and commas. Attention to SEP token can be thought of as a no operation, implying that that particular word is not much affected by any other word. Moreover the attention heads in the lower layers in particular dont focus on any particular word, rather the attention is much more dispersed across the whole sentence, giving a bag-of-vectors representation. Also the CLS token in the last layer has a very dispersed attention, making it an ideal candidate to be input to a classifier. There are some attention heads which learn some syntactic feature, like object of verb attending to the verb, preposition attending to its verb, coreference is also handled well by some heads. Another interesting point is that heads in the same layer tend to behave together (probably making them an ideal candidate to be pruned). This can be an artifact of dropout as well as we force the model to be not dependent on one head. Making attention non-uniform has been studied in link which seems to improve the performance of machine translation.)
- BERT (First true bidirectional embedding in the sense that in elmo, either the token attends only its left context or only its right context, and then the 2 embeddings are concatenated, while in BERT, the token can attend to its left and right context simultaneously.)
- ULMFit blog (It should be mentioned that while Howard and Ruder [10] were not the first to apply inductive transfer via fine-tuning in NLP, they were one of the first to do so successfully with regard to performance and efficiency. First some general domain language modelling pretraining, then target domain langauge modelling pretraining, and finally target task specific fine-tuning. They use AWD-LSTM and a bunch of other techniques like freezing, learning rate scheduling and discriminative fine-tuning (discriminative fine-tuning means using different learning rates for different layers of the network, higher learning rates for higher layers since higher layers are the ones which learn semantic high-level features and more complex information, while lower learning rates for lower layers since lower layers learn syntactic information) to make this work.)
- Lottery Ticket Hypothesis Video (paper shows that instead of one-shot pruning, iterative pruning is more useful for finding sparse networks. The hypothesis is: A randomly-initialized, dense neural network contains a subnet-work that is initialized such that—when trained in isolation—it can match the test accuracy of theoriginal network after training for at most the same number of iterations. The key thing here is that the subnetwork needs to have the same initialization as the original network, random initialization does not lead to same performance, suggesting that the weights that were not pruned won kind of a lottery at the time of initialization. It shows that 10-20% weights only can lead to same or even better performance when retrained. Another important thing is they show that the weights that remain in the subnetwork are the ones that travel the farthest from their initialization in the optimization procedure, compared to the weights that are pruned, so if w_0 was retained and had the final value w_0f while w_1 was pruned and had the final value w_1f, then w_0f-w_0 was much greater than w_1f-w_1.) Deconstructing Lottery Ticket Hypothesis Video (this paper further shows that the weights that are pruned are not the ones which were initialized to 0, but rather are the ones which were initialized to relatively large positive or negative values, but had value close to 0 after the training.)