original title: how to be good at research

Original by Vivek, AI Analyst

Original language: MinLi, AI Builder

Nobody really taught you how to do research. You get a desk, a question that someone else picks up, and a vague instruction to make something new。

So most people do reverse engineering of the job through what they can see (e.g. papers, posts and announcements), and eventually they learn just how to "look" like a researcher, not how to "be" a researcher. Real research capacity is a stack of small skills, and almost every one can be developed through deliberate exercise。

Pick your own problems

Richard Hamming has a habit at Bell Lab, which makes him very unwelcome at lunch. He would ask those sitting next to him what the important questions in their field were, and then why they did not study them. So people changed their tables to eat。

This is a very stingy question, because most of us have no good answers. We are not choosing the problem, but we are absorbing it — from our mentors, from the bulletins issued by a major laboratory last quarter, from the papers that have been transmitted to us all this week。

The problem with absorption is that you have only conclusions, without knowing the logic behind them. You know a famous lab that cares about a certain direction, but you don't know why, or what they expect to find, or what will make them abandon that direction。

When they turn around, you won't notice until a year later. And, on a matter that's already popular, you're running with 1,000 people who start earlier and count better than you。

The Machine Learning Research Guide for John Schulman divides this work into two models. First, you read the literature and look for places to improve. SecondYou choose a result you really want to achieve, and then you push it back to design the experiment。

He argued for the second because it created originality. A goal that you really care about will drag you into a territory not covered by any general paper。

as for taste, it is often discussed as a gift. but it's more like a muscle。

Before running each experiment, predict its results; cover the results of a paper and guess the data in its own way; record what results published this month will remain important after two years, and then verify your hit rate later. One prediction, one error, and a few hundred times -- that's how every good model is trained, including the one in your head。

Upgrade your input

Shared reading lists generate shared ideas. If your info ration is just an arXiv's hot list plus a group chat. What remains of the election is bound to come to the same conclusions as everyone, which makes them almost worthless。

The value of old information has been seriously underestimated. This is an area that is constantly repeating its own past: The Mixed Expert Model (MoE) dates back to 1991 and STM to 1997, when reverse transmission became mainstream in 1986。

Rich Sutton wrote "The Bitter Lesson" in 2019 with just a thousand words, and its predictions of the trajectory of development in this area are ten times more accurate than it is. Claude Shannon gave a speech on creative thinking in 1952, and his first move was to narrow down the problem to an almost insignificant level, crack the reduced version, and then add back a little bit of the difficulty。

This alone will help you break more walls than any modern productivity proposal。

The breadth and depth are as important。Explanatory research draws on the neurosciences in an undisguised way; evaluation (Eval) design is a mechanism design in white masts; as long as there is a practical understanding of how the GPU is moving the memory, you will be able to determine which architecture papers are destined to fail before the results of the benchmarking test are available; and honest statistics may already be the most scarce skills in the field of machine learning, where many of the publicly published "quiet" are simply a feeling of error。

One more thing。Read the paper itself, not the post summarizing it。Appendices are the place where secrets are buried, and the "restrictive" part is usually the most honest part of the entire document。

Write everything down

Paul Graham pointed out that an idea was always very mature before you tried to translate it into words. But black and white will reveal the holes in your brain that you've never tested, that you're not in a consistent position, that you're in conflict with one another。

The principle of Feynman is that the first person you must avoid cheating is yourself, because you are the easiest target。Writing is the cheapest defence mechanism ever invented。

Darwin went further, and he programmed it: any fact contrary to his theory would be written on the spot, because he discovered that his memory could be removed at a much faster rate than that of favourable evidence. So is your memory for your failed running records。

Maintaining a log habit: assumptions, settings, expectations, results, updated perception. Rereading the last month's record would make you feel extremely humble and no reviewer would be able to do so。

Original Link