Some pitfalls on the way of research
Jerry, a caveman (well, did they have such names? who knows! let’s make an assumption, for the time being) got up after his usual nap, just to see a mesmerizing display of colors appearing in the sky, right in front of the cave entrance. He had seen it many times before, but did not know why it was there. “Maybe it could be the meat I ate”, he thought, which he often gathers by hunting near the lake on rainy days, “why would it appear on my doorstep otherwise?”. You already would have guessed that Jerry saw a rainbow, which was given a far more reasonable explanation through the observations made by Sir Isaac Newton, many many years after the demise of Jerry.
Now, we are no different than Jerry at reasoning “things” or explaining “things” (things meaning the natural or man-made phenomena around us) unless we have an acceptable way to make some reasonable explanations of things. These “ways” evolved separately in the ancient eastern world and the western world. Let’s talk about the version of the western world as it has become the dominant “way” in the world now. The ideas of this domain initiated from philosophical endeavors (during the Greek civilization mainly). “Epistemology” is a related sub-category where people try to formalize the methods on how to acquire true knowledge.
The field of epistemology, has a lot to talk about (you could have a look at an earlier post that I made related to epistemology here ) and it might carry us away from the original intention of this write-up, which is about a challenge in research. Now, that term has become common, we would use it unintentionally sometimes. But it turns out that it started be mentioned more often only after 1900s (not in the times of Jerry) according to Google Dictionary. It is defined in the Cambridge dictionary as, “a detailed study of a subject, especially in order to discover (new) information or reach a (new) understanding”.
As I started out as a research student in the field of NLP, there were ideas which was vague (to me) such as, how to be sure whether I’m doing it in the right way (a justifiable way) or will this work lead me to a useful end. The general concepts in the method of research (or as some call it the “Scientific method”) would help anyone in any scientific domain, to understand whether things are in right places. I guess some beginners might be tricked into a pitfall (as I too did) which is, some initial idea or assumption that we make is always correct just because it fits with some of our expectation and we draw quick conclusions out of them. Below is an interesting video that I saw only recently on YouTube.
So we are quick in drawing conclusions by only the fact that it matched what we imagined how it would be, our current knowledge or our perception about it. But that makes us “a Jerry”.
Hypothesis
First we have a hypothesis (conjectures; like an assumption), in order to find answers to a question that we have, based on our current knowledge about the question. Then we can test whether our hypothesis is correct or not. Jerry thinking that meat he ate brought him the colorful display could be a some hypothesis. He made it up using the knowledge he had related to the problem-“why it was there?” But any hypothesis should be “falsifiable”, which means it could be contradicted by any logically possible observation. Or in other words, Jerry’s hypothesis is falsifiable, as there could be an event when the rainbow appears on a day which Jerry had no meat. (In theory, a hypothesis would not be falsifiable, if it doesn’t have a property called “inter-subjective verifiability”; which means, that it should be available for other subjects (say humans) to observe and also be reproducible)
Experiments
Next, we need to find (or devise) experiments to test how far our hypothesis is able to provide a reasonable explanation to our initial question. Here, what I learnt is that we should try with every possible experiment which could falsify our hypothesis or we might fall into the pit-fall I mentioned above, (also like in the video clip). We have to try every possible method that would falsify our assumption and try to prove ourselves wrong. There are instances that we get satisfied that our novel idea turned out to be true just because a single experiment proved it, but there could be hundreds of instances where our assumption is completely wrong. It would not also serve our purpose well, if we only try to devise experiments that would support our hypothesis (or simply how we like it). Then we might end up in a wrong conclusion. Rules of the things around us are as they are, not what we want or like them to be. Not only the variety of experiments should be sufficient in order to uncover the real truth but also they all should be fair. Like the fair coin experiment in the textbooks learnt during the high school. We would not want to conduct experiments in a way such that they are biased towards what we like to see.
Next, we can analyze the experiment results and draw conclusions from them, whether we need to form new hypotheses and devise more experiments on them. Likewise, the research process could run iteratively.
A simple example
In ML, averaging results over several samples of data is a common practice followed to get a fair evaluation of a model’s performance (say k-fold cross validation). We cannot be so sure whether a model generalizes well, just by its good performance on a particular sample drawn out of a population. Perhaps, it might perform well, only on a specific sample but not on the others. Furthermore, we make sure that the model doesn’t see any test samples during its training, to be fair such that we get to see the true performance of the model or how well it infers the population.
(In the case of Jerry, what would happen if Jerry just conducted an experiment of observing the sky for the rainbow, only on days he ate meat, to verify his initial hypothesis. Perhaps that might make the legend of the magical creature by the lake whose meat gave you free colorful displays on rainy days…)
Skepticism
Being skeptic can be your friend in pursuing true knowledge of a particular subject. Here, what is referred by the word “skepticism” is, the behavior of questioning things and not easily getting convinced of something. Sometimes the same word might refer to far more extreme concepts such as in religion but let’s be limited to the context of research here. As I learnt, we should always question our own work. Can we do better…? is it correct what I have done…? It is not a mere doubt, but questions based rational thinking and arguments. The feedback of peers and supervisors is important here since there might be corners we miss in the process.
Rationales
Sometimes, we work based on a hunch rather than facts. This could be a maze of desperation in the realm of research. Suppose we are devising an experiment based on some hypothesis. The hypothesis is inter-subjective verifiable and all that but after experimenting hours and hours it appears to be a lost cause. The tricky part is that, sometimes our intuition might be very wrong and its dependent on the knowledge and the experience we currently have. Working only based on a hunch, could be a sign of desperation sometimes as well and there are better ways to act in such situations than doing random things. So, another thing I learnt is that, it’s better to always move forward with some rationale, especially when building hypothesis and experimenting them. But it doesn’t mean that we should not discard the power of our intuition at all. Sometimes it’s the intuition that helps us with finding our way when we are lost. Yet, it might take some time and effort to develop a “accurate & broad” intuition related to the domain we are working on. Now, we cannot use our intuition we use to play soccer on the playground to carry out some scientific research, right? Intuition is also different in context.
Summary
Phew, it was some little things that I learnt through experience only recently and I think it is better to be late than never. The attitude of rational thinking and questioning is important while not trapped by our cognitive biasness (we will tend to bias towards our subjective reality or our perception of things). But lacking of these might trap you especially when you have to work on you own and even though they seem little things they are common pitfalls that a beginner might fall into. I am also a beginner and I think what drives research forward is peer collaboration and feedback that we get. Let me know your thoughts and experience in the comments! We might be able to have Jerry for another time as well. (Or was my assumption wrong, to call him Jerry?)
