Difference between revisions of "Spring 2017 CS292F Deep Learning for NLP"

Revision as of 17:26, 10 February 2017

Instructor and Venue

• Instructor: William Wang
• TA: TBA
• Time: T R 1:00pm - 2:50pm
• Location: PHELPS 2510
• TA Office Hours: TBA
• Instructor Office Hours: Tu 3-4pm HFH 1115
• Prerequisites:
  • Good programming skills and knowledge of data structure (e.g., CS 130A)
  • Solid background in machine learning, linear algebra, probability, and calculus.
  • Comfortable with deep learning platforms such as TensorFlow, Torch, Theano, MXNet, Caffe etc.
  • Prior experiences with AWS is not required, but could be very useful.

Course Objective

At the end of the quarter, students should have a good understanding about basic deep learning models, and should be able to implement some fundamental algorithms for simple problems in deep learning. Students will also develop an understanding of the open research problems in deep learning, and be able to conduct cutting-edge research with novel contributions to improve existing solutions.

Piazza

piazza.com/ucsb/spring2017/cs292f/home

Syllabus

• 04/04 Introduction, logistics, NLP, and deep learning.
• 04/06 Tips for a successful class project
• 04/11 Word embeddings (HW1 out)
• 04/13 Knowledge base embeddings
• 04/18 Neural network basics (Project proposal due)
• 04/20 Neural networks language models
• 04/25 RNNs (HW1 due and HW2 out)
• 04/27 LSTMs/GRUs
• 05/02 Convolutional Neural Networks (1)
• 05/04 Convolutional Neural Networks (2)
• 05/09 Project: mid-term presentation (1)
• 05/11 Project: mid-term presentation (2) (HW2 due)
• 05/16 Language and vision
• 05/18 Information extraction
• 05/23 Speech recognition and understanding
• 05/25 Sequence-to-sequence models and neural machine translation
• 05/30 Attention mechanisms in NLP
• 06/01 Question answering
• 06/06 Project: final presentation (1)
• 06/08 Project: final presentation (2)

Course Description

Deep learning has revolutionized many subfields within AI. DeepMind's AlphaGo combined convolutional neural networks together with deep reinforcement learning and MCTS, and won many games against top human Go players. In computer vision, most of the leading systems in ImageNet competitions are based on deep neural networks. Deep learning has also changed the game in NLP: for example, Google has recently replaced their phrase-based machine translation system with neural machine translation system. Throughout the quarter, we will go over some of the basics in neural networks, and we will also go through the deep learning revolution after 2006. In this graduate class, we will also emphasize on the development of graduate student's paper reading and presentation abilities: each student will need to present research papers related to this topic. Last but not least, the most important aspect of this course is for students to work on a novel research project in open problems related to NLP and deep learning, and gain hands-on experiences of doing cutting-edge research.

Text Book

No textbook is required, but the following optional textbook is recommended:

• Deep Learning, An MIT Press book, Ian Goodfellow and Yoshua Bengio and Aaron Courville
• HTML version of the book: http://www.deeplearningbook.org/

Project

One key aspect of this class is to have students to gain hands-on experiences in open research problems. To do this, each student will need to propose a research project. The teaching staff will provide the feedback on the proposal, and track the progress of each student. Computing resources will be provided: each team will be provided with sufficient amount of AWS credits for their projects.

In the project proposal, each team must clearly mention the following aspects of their project:

• What is the motivation of the problem?
• What is the exact definition of the problem? How do we formulate the problem in machine learning?
• What are some existing approaches to this problem?
• What are some existing datasets that you can work on?
• What is the novelty in your project? New problem? New approach? New dataset?
• How are you going to implement your approach and verify the idea?

Good places to look for project inspirations:

• Recent papers from ACL, EMNLP, NAACL from ACL Anthology: http://aclweb.org/anthology/
• Recent papers from ICML, NIPS, and ICLR conferences: http://jmlr.org/proceedings/ http://papers.nips.cc/

Available Datasets

• Wikipedia Harassment/Personal Attack Dataset https://figshare.com/projects/Wikipedia_Talk/16731 Ex Machina: Personal Attacks Seen at Scale, https://arxiv.org/abs/1610.08914
• Stance Detection / Fake News Detection / Automated Fact-Checking, see William.

Grading

There will be two homework assignments (20%), one project (65%), and an in-class paper presentation (15%). The breakdown of project grading includes: 1-page proposal (10%), mid-term presentation (10%), final presentation (15%), and a final report (30%). Four late days are allowed with no penalty. After that 50% will be deducted if it is within 4 days after the due day, unless you have a note from the doctors' office. Homework assignment submissions that are five days late will receive zero credits.

Final Report Format

You must use the ICML 2017 latex style files for writing the report. The final report must be 4-8 pages long. It is encouraged to include the following components in your report: abstract, introduction (motivation, task definition, your novel contributions), related work, your approach, experiments, discussion, and conclusion.

Academic Integrity

We follow UCSB's academic integrity policy from UCSB Campus Regulations, Chapter VII:``Student Conduct and Discipline"):

• It is expected that students attending the University of California understand and subscribe to the ideal of academic integrity, and are willing to bear individual responsibility for their work. Any work (written or otherwise) submitted to fulfill an academic requirement must represent a student’s original work. Any act of academic dishonesty, such as cheating or plagiarism, will subject a person to University disciplinary action. Using or attempting to use materials, information, study aids, or commercial “research” services not authorized by the instructor of the course constitutes cheating. Representing the words, ideas, or concepts of another person without appropriate attribution is plagiarism. Whenever another person’s written work is utilized, whether it be a single phrase or longer, quotation marks must be used and sources cited. Paraphrasing another’s work, i.e., borrowing the ideas or concepts and putting them into one’s “own” words, must also be acknowledged. Although a person’s state of mind and intention will be considered in determining the University response to an act of academic dishonesty, this in no way lessens the responsibility of the student.

More specifically, we follow Stefano Tessaro and William Cohen's policy in this class:

You cannot copy the code or answers to homework questions or exams from your classmates or from other sources; You may discuss course materials and assignments with your classmate, but you cannot write anything down. You must write down the answers / code independently. The presence or absence of any form of help or collaboration, whether given or received, must be explicitly stated and disclosed in full by all involved, on the first page of their assignment. Specifically, each assignment solution must start by answering the following questions:

• (1) Did you receive any help whatsoever from anyone in solving this assignment? Yes / No.
  • If you answered 'yes', give full details: (e.g. ``Jane explained to me what is asked in Question 3.4")
• (2) Did you give any help whatsoever to anyone in solving this assignment? Yes / No.
  • If you answered 'yes', give full details: (e.g. ``I pointed Joe to section 2.3 to help him with Question 2".
• No electronics are allowed during exams, but you may prepare an A4-sized note and bring it the exam.
• If you have questions, often ask the teaching staff.

Academic dishonesty will be reported to the highest line of command at UCSB. Students who engage in such activities will receive an F grade automatically.

Accessibility

Students with documented disability are asked to contact the DSP office to arrange the necessary academic accommodations.

Discussions

All discussions and questions should be posted on our course Piazza site.