The Craft of
(Software Engineering) Research


Yanyan Jiang

Welcome!

Happy to see you all!


Why am I standing here?

  • You guys are about to make some serious changes to the world
  • But perhaps you're not quite ready yet

Notes

Don't forget to check the course homepage

  • We'll have video replays (with transcriptions) available
  • Assignment 1 is due: Oct 9

In today's lecture

  • Introduction to software engineering research
  • How to read research papers

What is Research?

In the broadest terms, we do research whenever we gather information to answer a question that solves a problem.
——The Craft of Research

Example

  • PROBLEM: To settle a bet, I need to know when Michael Jordan was born.
  • RESEARCH: You Google “Michael Jordan birthday.”

Academic Research

Solving significant problems with novel insights.

  • And research papers should...
    • Teach a lesson to our peers (researchers, practitioners, ...)
  • Sadly, most folks don't get this
    • Ever tried to find some nonexistent novelty in your capstone project?

Therefore...

Research is hard.

  • Super hard
  • Even for the most clever and hard-working ones
    • Because they generally aim higher

Grad Life: The Good, The Bad, The Ugly

Good

  • Get loads of time to explore deep, dark, fantastic stuff

Bad

  • Things just never seem to go as planned

Ugly

  • End up having to do a ton of workarounds...

Software Engineering Research:
Three Easy Pieces


Reading, Writing, and Hacking

Before We Dive In...

  1. What is software anyway?
  2. What is software's engineering?

What's Up with Our Software?

We can't seem to build good software quickly.

Software engineering: making life easier for code peasants

  • Faster (productivity)
  • Better (quality)

The ultimate dream

  • Kill humans in software development
  • 80% already achieved by copilot

For fun and profits

Solve hard problems!

Build useful tools!

  • A quick test for evaluating your work: will real software engineers (e.g., Google/Meta guys) care about your work?

Examples

“Empirical and human studies for better understanding”

“Technical solutions for hard problems”

The Three Easy Pieces

Reading

  • Stay in the loop with the latest research

Writing

  • Share your ideas with the others (and the world)

Hacking

  • Code to prove your idea works

1 - Reading (40% pts)

  • We have a reading list of good papers
    • Studies
    • Compilers; static analysis and checking
    • Dynamic analysis and trace; debugging
    • Testing and validation
    • Verification and synthesis
  • Assignment: paper presentation
    • Cycled broadcast on Bilibili for 1 week
    • <10% Distinguished Presentation Awards 🏅

2 - Writing (60% pts)

  • Bad writing practices
    • Broken logic
    • Imprecise wording
    • Verbose writing
  • Assignment: research proposal
    • <10% Distinguished Proposal Award 🏅

SUGGESTIONS

Start writing sooner rather than later

Rush into your advisor's office and make them revise your manuscripts face-to-face

  • If they refuse → drop PhD immediately

3 - Hacking (0% pts)

Drop course if you don't know how to hack real systems

  • (Programming assignments moved to group-private courses)

Interview questions

  • What's the most memorable bug you've ever debugged?
    • Drawing a blank?
  • Can you explain what Git is?
    • Just something for downloading source code from the Internet?

How to Read

The Sad Truth

Over 1/2 of our students aren't reading enough

  • even after 2-3 years into their PhD

But we need to keep reading.

Reading Materials (From Easy to Hard)

Textbooks and courses

Magazines, blog posts

  • Communications of the ACM; Hacker News/Medium/Reddit/...

Research papers

  • Top conferences (ICSE/FSE/ASE/ISSTA, SOSP/OSDI/EuroSys/ATC, PLDI/POPL/OOPSLA, ...) and talks; some journal papers

How to Find Relevant Papers

All papers form a citation graph

  • Your advisor should have given you some seed papers
    • If not → rethink your PhD
    • If of low quality → rethink your PhD
    • Use these papers to find more by traversing the graph
      • Favor papers in good venues (top conference/journals)

My Reading Journey (2011-now)

As a PhD student

  • Textbooks (open courses): 40/year
  • Magazines: 24/year (less dense)
  • Papers: 100/year (this is the tough part)

As a faculty member

  • I read papers/magazines/textbooks a lot faster (by cheating: invoke a domain-expert to explain)

Sounds Scary, Right?


How is that even possible?


Not that bad.

You'll read a lot faster after training yourself with >100 papers

  • And you don't need to understand everything 100%

How to Read a Paper in Three Steps

S. Keshav. How to read a paper, ACM SIGCOMM Computer Communication Review, 3(37), 2007.

  1. Quick scan
    • category, context, correctness, contributions, clarity
  2. Jot down the key points
    • read with care, but ignore details as proofs
  3. Virtually re-implement
    • re-create the work as if you were the author

How to Read a Paper (cont'd)

A “machine-learning” approach

  • This paper identifies a new problem!
    • Back-prop: why didn't I notice this important problem?
  • This paper tackles an existing problem!
    • Hide the solution and try to solve it yourself (like auto-regressive models)
    • Back-prop: why didn't I come up with this solution?
  • We are “transformers”

Traps and Pitfalls in Paper Reading

Only papers solving my problem are relevant

  • Good problem solvers have lots of tools in their toolbox

This paper is limited in XXXX. It's useless!

  • Every paper has limitations, be constructive in your criticism

I give up!

  • Don't give up easily--grab some background materials

Reading in the GPT-age

LLMs are amazingly good at breaking things down

  • I turn to GPT-4 if I can't explain things to my kid

    (冷冻电子显微镜) 可以投射电子到蛋白质分子上，而蛋白质分子是由很多小颗粒组成的，就像一堆小石头组成的小山一样。这些电子会与这些小颗粒相撞，并反弹回来，就好像你向一个山丘丢石头，石头会弹回到你跟前一样。然后，这个机器会记住所有反弹回来的电子的路径和速度……就好比你闭着眼睛，通过听石头落地的声音判断山丘的形状一样。

Quick Recap

  1. Read a ton of papers
  2. Read really good papers (to develop a good taste in research)
    • The training-data quality matters
    • “Textbooks are all you need”, but...
      • 90% of “top-conference” papers have almost no “real” contributions (maybe doing well, but not groundbreaking)
      • Lots of papers can mislead junior researchers
  3. Use tools wisely (citation manager, symbolic engines, GPT, ...)

Example: FlashFill

Sumit Gulwani. Automating string processing in spreadsheets using input-output examples (POPL'11, Most Influential Test-of-Time Paper Award in 2021)

  • Good paper, but extremely hard to read
  • Trust the authors: papers are self-contained
    • Reading them only requires textbook knowledge
    • [Corollary] Go find textbooks (and open courses) if you can't understand a paper

More Comments on Reading...

Manuel Blum's advice to a beginning graduate student

  • Just don't panic

End.


(Drop course.)