Computer Architecture Research-105 (Gollu’s Kutti Story- Experimental skills of interest for Computer Architecture Research)

5 min readFeb 26, 2023

Preamble: This blog is a continuation of my previous blogs: Two cents on Computer Architecture Research [1][2] [3][4]. It is primarily meant for Indian undergraduate/early graduate students who have taken a course on computer organization/architecture and want to pursue research in computer architecture. There is a long gap between the previous blog on Computer Architecture Research 104 and this one. Gollu was busy doing Computer Architecture Research, and it took around two years of consistent hard work to get some good/excellent research results.

This blog is about all the learning that Gollu had in the last two years regarding experimental skills for publishing research at top A/A* computer architecture forums like ISCA, MICRO, ASPLOS, HPCA, PACT, ICS, ICPP, IPDPS, and ISPASS.

Computer Architecture research is empirical/experimental. Yes, computer systems research is empirical/experimental. However, compared to other computer systems areas, computer architecture is highly empirical, with top conference research papers having many plots, analyses, and whatnot. Ideas in top conference papers are just a page or two long, but the motivation and experiments are six to eight pages out of 13 [5].

Here is the list of skills that Gollu thinks are useful.

Faithful coding and documentation: You should be able to code your idea and existing ideas without compromising the subtle nuances of ideas. A simple rule is to write modular code, have “assert” statements, and document your code: one paragraph for every k lines of code or function. Make sure you have to debug flags and additional microarchitecture events of interest if it is on a microarchitecture simulator. Code and write what you have coded in simple English sentences so that it can help you and your advisor and collaborator. While coding, talk to yourself, “why did you code? what did you code?” Once, Gollu claimed he had written 1000 lines of code, implementing a flagship paper within a month. Later, his advisor found the implementation was elegant but functionally incorrect and gibberish.
Verify, whatever you have coded is (i) functionally correct. (ii) micro architecturally correct. For example, your idea has an additional microarchitecture structure, but you still need to add latency. Similarly, ensure the interactions between your new microarchitecture unit and the existing ones are correctly modeled as additional events with their respective latency and energy consumption. Events should be functionally correct. Just because your code, simulations, or experiments run fine does not mean all is well.
Version control. Once you finish coding, push it to your Github account and do proper version control. Nah, start doing version control from day-1 of your coding. Gollu has seen his friends/collaborators/ sending code by email through a zip file and another student changing that code and sending another email with a new zip file. These friends are from IIX doing architecture research :) Once coding is done and properly pushed to a GitHub account, it is time to run experiments. Students spend days doing wrong experiments that lead to wrong insights/results. You should have automated scripts that can run experiments with various parameters faithfully. The results should be ported into a Google sheet or any excel sheet or .csv file, and then another small script that can plot the results with proper legends, Y-axis, X-axis, etc. Ensure all the scripts are properly checked for their sanity; otherwise, random and adhoc numbers will come out of plots.
Faithful evaluation. Use all the benchmarks of interest and defend why you picked what you picked. Do not cherry-pick your benchmarks to extrapolate your idea. It won’t fly at top conferences. Simulated instructions are too few for any microarchitecture unit to warm up is another mistake. If 10M instructions are good enough for a small cache, 100M may not be enough for TLBs. Do not do experiments blindly. Make sure the benchmarks are simulated for their respective region of interest, or else you are creating an artificial problem and solving it with an artificial solution. Do not forget Amdahl’s law :) For experiments on real systems, perform it atleast k times and take the average to nullify the system noise.
Plots. Make sure to make legible plots: no tiny font sizes unless you provide a microscope for free with your paper submission :) and technically correct Y and X axes. Please refer to any top conference paper for the quality of plots [5]. Make sure plots are normalized to a baseline. The baseline should be properly defined, or you will end up comparing apples with oranges. Summarize the results: Arithmetic mean, Geometric mean, or harmonic mean. Percentage, ratio, normalized ratios, speedup, and slowdown are the keywords to understand and use appropriately and not blindly. Do not take an average of normalized numbers to extrapolate your results:) Do have the flexibility of getting plots, insights of interest in minutes, if you change a parameter, etc. Test your simulations, experiments with small experiments, and then repeat for the large ones.
Why? Now that you have your magical numbers. But can you convince yourself why did you get it? Let’s say you proposed a cache optimization, and performance improves by 200%, but cache misses have improved only by 2%. Is it possible? Yes, and NO. Yes, if the cost of these 2% cache misses is high. No, otherwise. Can you convince yourself with end-to-end performance, power, or security claims? For security paper, the threat model may assume many and then claim that idea is secure :) A common trend in low-quality conferences nowadays. Make sure your excel sheets, plots, and data match. Gollu has seen his friends would show performance improvement in plots and performance degradation on sheets, and the script does not check all of these :) If you get x% improvement, then you should be able to explain from where and why you got this x%.
Artifical problem. Do not create an artificial problem and propose artificial solutions and publish in low-quality international conferences where the expertise of reviewers could be better. Even the reviewers may have yet to write/publish a research paper at the flagship forums. You may get the best paper award in those conferences and be happy with your myopic world of computer architecture research. If that is the goal, then this blog is not for you. All the best for your best papers :)
Slides. Finally, prepare slides for your research. Let the slides tell you how to do that [6][7].

PS: If you have these skills, then there is no guarantee you will still get your research published at the top forums. However, if you do not have these skills, you won’t be able to make it to flagship forums. If you are interested in mediocre and low-quality non-architecture (mostly VLSI and others), then this blog is not for you. This blog is only meant for students who want to do top-quality research. Acquiring these skills, like Gollu, takes time. However, it is fun and rewarding to have these skills.

Computer Architecture Research-105 (Gollu’s Kutti Story- Experimental skills of interest for Computer Architecture Research)

Written by Biswabandan Panda