Winter 2021 CS291A Introduction to Deep Learning

Instructor and Venue

• Instructor: William Wang
• Time: M W 11:00-12:50
• Location: Zoom (private link to be shared)
• Instructor Office Hours: M 2-3pm starting 01/23.
• Prerequisites:
  • Machine Learning (CS 165B) or equivalent
  • Good programming skills in Python and knowledge of data structure (e.g., CS 130A)
  • Solid background in machine learning, linear algebra, probability, and calculus.
  • Comfortable with deep learning platforms such as TensorFlow, Torch, Theano, MXNet, Caffe etc.
  • Prior experiences with AWS / Google Cloud is not required, but could be very useful.

In-class Presentation

• 01/25 Word embeddings
  • Conner : Efficient Non-parametric Estimation of Multiple Embeddings per Word in Vector Space, Neelakantan et al., EMNLP 2014
  • Sanjana : Glove: Global Vectors for Word Representation, J Pennington, R Socher, CD Manning - EMNLP, 2014
  • Wenhu : AutoExtend: Extending Word Embeddings to Embeddings for Synsets and Lexemes, Rothe and Schutze, ACL 2015
• 01/30 Neural network basics (Project proposal due to Grader: Ke Ni < ke00@ucsb.edu> , HW1 out)
  • Jashanvir : Learning representations by back-propagating errors, Nature, 1986
  • Metehan : An overview of gradient descent optimization algorithms, Sebastian Ruder, Arxiv 2016
  • Vivek P.: Dropout: A simple way to prevent neural networks from overfitting (2014), N. Srivastava et al., JMLR 2014
• 02/01 Recursive Neural Networks
  • April : Semantic Compositionality through Recursive Matrix-Vector Spaces, Socher et al., EMNLP 2012
  • Zhiyu : Parsing with Compositional Vector Grammars, Socher et al., ACL 2013
  • Andy : Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank, Socher et al., EMNLP 2013
• 02/06 RNNs
  • Lukas : A Learning Algorithm for Continually Running Fully Recurrent Neural Networks, Ronald J. Williams and David Zipser, 1989
  • Yifu : Recurrent neural network based language model
  • John : Generating Sequences With Recurrent Neural Networks, Alex Graves, 2013 arxiv
• 02/08 LSTMs/GRUs
  • Liu : Long short term memory, S. Hochreiter and J. Schmidhuber, Neural Computation, 1997
  • Nidhi : On the Properties of Neural Machine Translation: Encoder–Decoder Approaches, Cho et al., 2014
  • Vivek A.: Gated Feedback Recurrent Neural Networks, Chung et al., ICML 2015
• 02/13 Sequence-to-sequence models and neural machine translation (HW1 due and HW2 out)
  • Ryan : Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation, Cho et al., EMNLP 2014
  • Yanju : Sequence to Sequence Learning with Neural Networks, Sutskever et al., NIPS 2014
  • Karthik : Achieving Open Vocabulary Neural Machine Translation with Hybrid Word-Character Models, Luong and Manning, ACL 2016
• 02/15 Attention mechanisms
  • Jing : NEURAL MACHINE TRANSLATION BY JOINTLY LEARNING TO ALIGN AND TRANSLATE, Bahdanau et al., ICLR 2015
  • Abhay : Teaching Machines to Read and Comprehend, NIPS 2015
  • Ashwini : End-to-end memory networks, NIPS 2015
• 02/20 Convolutional Neural Networks (Mid-term report due to Grader: Ke Ni <ke00@ucsb.edu>)
  • Esther : Natural Language Processing (Almost) from Scratch, Collobert et al., JMLR 2011
  • Maohua : A Sensitivity Analysis of (and Practitioners’ Guide to) Convolutional Neural Networks for Sentence Classification, Zhang and Wallace, Arxiv 2015
  • Jiawei : Convolutional Neural Network Architectures for Matching Natural Language Sentences, Hu et al., NIPS 2014
• 02/22 Language and vision
  • Sai : Show and Tell: A Neural Image Caption Generator, CVPR 2015
  • Xiyou : Deep Visual-Semantic Alignments for Generating Image Descriptions, Andrej Karpathy and Li Fei-Fei, CVPR 2015
  • Richika : Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books, Zhu et al., ICCV 2015
• 02/27 Deep Reinforcement Learning 1 (HW2 due)
  • Sharon : Deep Reinforcement Learning for Dialogue Generation, Li et al., EMNLP 2016
  • David : Improving Information Extraction by Acquiring External Evidence with Reinforcement Learning, Narasimh et al., EMNLP 2016
  • Michael : Deep Reinforcement Learning with a Natural Language Action Space, He et al., ACL 2016
• 03/01 Deep Reinforcement Learning 2
  • Trevor : Playing Atari with Deep Reinforcement Learning, Mnih et al., NIPS workshop 2013
  • Calvin : Continuous control with deep reinforcement learning, Lillicrap et al, ICLR 2016
  • Chani : Mastering the game of Go with deep neural networks and tree search (2016), D. Silver et al., Nature
• 03/06 Unsupervised Learning
  • Hongmin : Generative Adversarial Nets, Goodfellow et al., NIPS 2014
  • Burak : Auto-encoding variational Bayes, Kingma and Welling, ICLR 2014
  • Pushkar : Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks, Redford et al., 2015
  • Liu : Semi-supervised Sequence Learning, Dai et al., NIPS 2015

Course Objective

At the end of the quarter, students should have a good understanding about basic deep learning models, and should be able to implement some fundamental algorithms for simple problems in deep learning. Students will also develop an understanding of the open research problems in deep learning, and be able to conduct cutting-edge research with novel contributions to improve existing solutions.

Piazza

http://www.piazza.com/ucsb/winter2018/cs291a

Syllabus

Winter 2018 CS291A Syllabus

Course Description

Deep learning has revolutionized many subfields within AI. DeepMind's AlphaGo combined convolutional neural networks together with deep reinforcement learning and MCTS, and won many games against top human Go players. In computer vision, most of the leading systems in ImageNet competitions are based on deep neural networks. Deep learning has also changed the game in NLP: for example, Google has recently replaced their phrase-based machine translation system with neural machine translation system. Throughout the quarter, we will go over some of the basics in neural networks, and we will also go through the deep learning revolution after 2006. In this graduate class, we will also emphasize on the development of graduate student's paper reading and presentation abilities: each student will need to present research papers related to this topic. Last but not least, the most important aspect of this course is for students to work on a novel research project in open problems related to NLP and deep learning, and gain hands-on experiences of doing cutting-edge research.

Text Book

No textbook is required, but the following optional textbook is recommended:

• Deep Learning, An MIT Press book, Ian Goodfellow and Yoshua Bengio and Aaron Courville
• HTML version of the book: http://www.deeplearningbook.org/

Project

One key aspect of this class is to have students to gain hands-on experiences in open research problems. To do this, each student will need to propose a research project. The teaching staff will provide the feedback on the proposal, and track the progress of each student. Computing resources will be provided: each team will be provided with sufficient amount of Google Cloud credits for their projects.

In the project proposal, each team must clearly mention the following aspects of their project:

• What is the motivation of the problem?
• What is the exact definition of the problem? How do we formulate the problem in machine learning?
• What are some existing approaches to this problem?
• What are some existing datasets that you can work on?
• What is the novelty in your project? New problem? New approach? New dataset?
• How are you going to implement your approach and verify the idea?

Good places to look for project inspirations:

• Recent papers from ACL, EMNLP, NAACL from ACL Anthology: http://aclweb.org/anthology/
• Recent papers from ICML, NIPS, and ICLR conferences: http://jmlr.org/proceedings/ http://papers.nips.cc/

FAQ: Can I use my existing research projects / thesis research project as the project in this class? A: I would prefer students to get out of their comfort zone and try something new in this class. If you are using existing techniques from your existing project, it is unlikely that you will be able to learn anything new during the course project. However, you may still draw the inspiration from your research problem to formulate your class project.

Available Datasets

• Wikipedia Harassment/Personal Attack Dataset https://figshare.com/projects/Wikipedia_Talk/16731 Ex Machina: Personal Attacks Seen at Scale, https://arxiv.org/abs/1610.08914
• Stance Detection / Fake News Detection / Automated Fact-Checking, email William.
• Deep learning for abstractive humor generation. Dataset: https://www.cs.ucsb.edu/~william/papers/meme.pdf
• NELL Knowledge Graph http://rtw.ml.cmu.edu/
• Relation Prediction / Reasoning FB15K-237 https://www.microsoft.com/en-us/download/details.aspx?id=52312
• Abstractive summarization datasets https://www.aclweb.org/anthology/D/D15/D15-1044.pdf
• WikiHow: learning processes from lists and free text https://github.com/paolo7/KnowHowDataset

Grading

There will be two homework assignments (20%), one project (65%), and an in-class paper presentation (15%). The in-class presentation includes 12 minutes presentation (12 slides max) and 3 minutes QA. The breakdown of project grading includes: 1-page proposal (10%), 2-page mid-term report (10%), final presentation (15%), and a final report (30%). Four late days are allowed with no penalty. After that 50% will be deducted if it is within 4 days after the due day, unless you have a note from the doctors' office. Homework assignment submissions that are five days late will receive zero credits.

Final Report Format

You must use the ICML 2018 latex style files for writing the report. The final report must be 3-5 pages long including references. It is encouraged to include the following components in your reports (not necessarily this order): abstract, introduction (motivation, task definition, your novel contributions), related work, your technical approach, such as math formulation of the problem, algorithms, theorems (if any), experiments, discussion, and conclusion.

Academic Integrity

We follow UCSB's academic integrity policy from UCSB Campus Regulations, Chapter VII:``Student Conduct and Discipline"):

• It is expected that students attending the University of California understand and subscribe to the ideal of academic integrity, and are willing to bear individual responsibility for their work. Any work (written or otherwise) submitted to fulfill an academic requirement must represent a student’s original work. Any act of academic dishonesty, such as cheating or plagiarism, will subject a person to University disciplinary action. Using or attempting to use materials, information, study aids, or commercial “research” services not authorized by the instructor of the course constitutes cheating. Representing the words, ideas, or concepts of another person without appropriate attribution is plagiarism. Whenever another person’s written work is utilized, whether it be a single phrase or longer, quotation marks must be used and sources cited. Paraphrasing another’s work, i.e., borrowing the ideas or concepts and putting them into one’s “own” words, must also be acknowledged. Although a person’s state of mind and intention will be considered in determining the University response to an act of academic dishonesty, this in no way lessens the responsibility of the student.

More specifically, we follow Stefano Tessaro and William Cohen's policy in this class:

You cannot copy the code or answers to homework questions or exams from your classmates or from other sources; You may discuss course materials and assignments with your classmate, but you cannot write anything down. You must write down the answers / code independently. The presence or absence of any form of help or collaboration, whether given or received, must be explicitly stated and disclosed in full by all involved, on the first page of their assignment. Specifically, each assignment solution must start by answering the following questions:

• (1) Did you receive any help whatsoever from anyone in solving this assignment? Yes / No.
  • If you answered 'yes', give full details: (e.g. ``Jane explained to me what is asked in Question 3.4")
• (2) Did you give any help whatsoever to anyone in solving this assignment? Yes / No.
  • If you answered 'yes', give full details: (e.g. ``I pointed Joe to section 2.3 to help him with Question 2".
• No electronics are allowed during exams, but you may prepare an A4-sized note and bring it the exam.
• If you have questions, often ask the teaching staff.

Academic dishonesty will be reported to the highest line of command at UCSB. Students who engage in such activities will receive an F grade automatically.

Accessibility

Students with documented disability are asked to contact the DSP office to arrange the necessary academic accommodations.

Discussions

All discussions and questions should be posted on our course Piazza site.