Over the course of the semester you will write five documents and then combine them into a single cohesive final report.

Report 0: Propose Ideas

Briefly describe three small project ideas by writing one paragraph per idea. The proposed projects should be small enough to ensure completion within the remainder of the semester. Notably, the project should not require substantial implementation – they should be based on evaluating some algorithm, system, or model that already exists. The goal of the project should be to understand the behavior (e.g., performance, accuracy, cost, etc) of some system or algorithm under different conditions. Your description should briefly describe the idea and explain what aspect you will evaluate for your project. You should also briefly detail the what existing code/benchmarks/data sets you will be able to make use of, and what you will need to construct yourself (hopefully not much).

If you have a strong preference to work on one of the three proposed ideas, then make a note of that in your writeup. However, you should still provide 3 possible ideas, perhaps listing out topics you might explore in future work.

Your report should be 1-2 pages in length, single spaced one-column format and submitted as PDF.

Lost? Try talking with your advisor for possible ideas. Don’t try to propose too large a project - instead, break a big project down into pieces and discuss a few of those as different ideas.

Due Date: Sunday January 27th at 11:59PM

Report 1: Research Description

A “solution” (i.e., a system, an algorithm, etc.) is affected by its inputs and its environment. As the researcher, you control the solution, a user would define the inputs, and the environment is typically outside either of your control. This report should first present the overall problem you are trying to solve, and then break the proposed research down into these categories and explain each of them.

You should describe your solution in enough detail that the reader can understand the high level design. The reader does not need sufficient information to implement the solution. Your report should also provide the basic background information needed so that a CS graduate student not in your research area can understand the context of the problem and solution. Include a section describing the current state of the solution’s implementation and note any aspects that you will need to complete before being able to do your evaluation.

Describe the aspects of the environment and task which are under your control and which you will not be able to modify. Identify the “performance” metrics (speed, accuracy, cost, etc) that you will evaluate. Note any other metrics that may provide insights into the effectiveness of the solution.

Your report should be 3-4 pages in length, double spaced one-column format, with additional space for diagrams and tables. You should include references as appropriate.

Due Date: Tuesday February 12th at 11:59PM

Report 2: Experimental Design

Having a clear experimental design gives structure to your investigation. Having a set of hypotheses to evaluate ensures your solution is robust and by predicting preliminary results you can more easily “sanity check” your experiments to be sure they are behaving correctly.

For this report you must propose a set of experiments that will evaluate your solution. Your report should list out your hypotheses for the solution’s behavior in each experiment. The hypotheses must be falsifiable and if possible, target causal hypotheses. You do not need to have an exhaustive list of hypotheses to test; instead, focus on the key metrics or aspects of the system that you expect will have the most impact. Describe the metrics you will analyze and the experimental setup including both the task and environment variables that you will adjust.

Your report should be 2-3 pages in length, single spaced one-column format, with additional space for diagrams and tables. The experimental plans in the document do not limit what you evaluate over the coming weeks–you are free to modify your design based on what you discover.

Due Date: Tuesday March 5th at 11:59PM

Report 3: Literature Review

Doing a literature survey ensures you understand the state of the art work in your problem area and helps identify common issues and opportunities that can motivate your work.

For this report you must synthesize the information from at least four research papers to present an overall literature summary. Make an effort to organize/classify the work into categories to help relate the pieces to each other and to your own work.

The report should first provide background for the problem domain, and then analyze each of the papers in detail. It is likely that your survey will involve more than four papers, but you only need to describe the four most relevant ones in detail. Explain the key results from these prior papers and describe how they relate to your own work. Identify any gaps, oversights, or errors that appear in the papers. Finally, describe how your research will innovate on top of this prior work.

This report should be 2-3 pages in length, single spaced one-column format, with additional space for diagrams and tables. You should include references.

Due Date: Wednesday March 27th before class

Report 4: Experimental Results

This report should analyze the results of your experiments. Beyond simply showing data, you should strive to show the meaning of the results and what they mean for the solution you are testing. Your report should evaluate the results of your previously proposed hypotheses and note any surprises.

You should make an effort to present your results in a visually clear and compelling fashion. Avoid misleading graphs and ensure that they are accurately representing both the strengths and weaknesses of your solution.

Due Date: Wednesday April 10th before class

Final Report

This final report will pull together all of your prior reports into a cohesive document. This does not mean simply copying and pasting – you will need to go back and revise and update your prior writing. The full report should be approximately 10 pages, single spaced one-column format, with additional space for diagrams, tables, and references. Your report should not exceed 20 pages.

Due Date: Sunday May 5th at 11:59PM

These assignments are based on the Research Methods for Empirical Computer Science course by David Jensen