CSCE 190 Computing in the Modern World Research Methodologies in Computing

1. CSCE 190Computing in the Modern WorldResearch Methodologies in Computing Spring 2012 Marco Valtorta mgv@cse.sc.edu

2. Computer Science as a Discipline • Science • The study of information processes • Not (necessarily) related to physical computers • Engineering • The study of computers • The study of computer applications • Mathematics • The engineering of mathematics • Taking mathematical objects, especially algorithms, and making them do useful things, as computer programs • Therefore, different ways of doing research! • Investigate the criteria for research evaluation in your field!

3. Systems and Theory • Systems: Programming Languages, Compilers, Operating Systems • Modeling • Simulation • Analysis • Experiments • Theory: Algorithms, Theory of Computation • Definitions • Conjectures • Proofs • Complexity • Ph.D. qualifier exams traditionally include these two parts

4. An example systems paper Nicklaus Wirth. “The Design of a Pascal Compiler.” Software: Practice and Experience, vol. 1, issue 4, pages 309-333, Oct/Dec 1971. Abstract: The development of a compiler for the programming language PASCAL1 is described in some detail. Design decisions concerning the layout of program and data, the organization of the compiler including its syntax analyser, and the over-all approach to the project are discussed. The compiler is written in its own language and was implemented for the CDC 6000 computer family. The reader is expected to be familiar with Reference 1.

5. An example theory paper Antonello Monti, Ferdinanda Ponci, and Marco Valtorta. “Extending Polynomial Chaos to Include Interval Analysis.” IEEE Transactions on Instrumentation and Measurement, vol. 59, no. 1, Jan. 2010, 48-55. Abstract—Polynomial chaos theory (PCT) has been proven to be an efficient and effective way to represent and propagate uncertainty through system models and algorithms in general. In particular, PCT is a computationally efficient way to analyze and solve dynamic models under uncertainty. This paper presents a new way to use a polynomial expansion to incorporate uncertainties that are not expressed in terms of a probability density function (pdf). This paper presents the formalization of the process and some simple applications. The authors show that, within the framework introduced in this paper, it is possible to incorporate interval analysis. The long-term goal of this paper is to support the claim that the proposed framework can extract and represent uncertain behaviors in a form more general than previously used for these engineering problems. The proposed approach is first applied to an algebraic model and then to a differential equation model. The results thus obtained are analyzed in two different perspectives: 1) interpreting the PCT expansion as a fully probabilistic method and 2) in the framework of possibility theory. The conclusions in these two cases are compared and discussed.

6. A Model of the Research Process: Machine Learning • Thomas Dietterich, “Exploratory Research in Machine Learning,” Machine Learning 5, 5-9 (1990). • Exploration, usually driven by specific problems in specific domains, leads to clear definition of tasks (problems to be solved) • Development of algorithms and methods to solve the problems • Empirical evaluation of the methods that have been developed • Comparison of algorithms and methods under controlled situations • Theoretical analysis of the task (problem) and the methods • Theoretical foundation for the field

7. Examples in Inductive Concept Learning Exploration: Hunt, E.B., Marin, J., & Stone, P.J. (1966). Experiments in induction. New York: Academic Press. Development: Quinlan, J.R. (1983). Learning efficient classification procedures and their application to chess endgames. In R.S. Michalski, J.G. Carbonell, & T.M. Mitchell (Eds.), Machine Learning: An Artificial Intelligence Approach (Vol. 1). San Mateo, CA: Morgan Kaufmann. Empirical evaluation: The two previous papers (Hunt et al: training set/test set; Quinlan: sample complexity) Comparison: Quinlan, J.R. (1988). An empirical comparison of genetic and decision-tree classifiers. Proceedings of the Fifth International Conference on Machine Learning (pp. 135-141). Ann Arbor, MI: Morgan Kaufmann. Theoretical analysis: Valiant, L.G. (1984). A theory of the learnable. Communications of the ACM, 27, 1134-1142. Theoretical foundation: Rissanen, J. (1978). Modeling by shortest data description. Automatica, 14, 465-471.

8. Publication • Types of publication • (archival) journals • E-journal • (archival) conferences • workshops • technical reports • patents • informal communication (not a publication?) • Peer review • Regular, blind or double-blind • Conflicts of interest • Refereeing

9. Funding • Competitive grants from funding agencies • NSF • Most common source for our department: http://www.cse.sc.edu/news/ • DARPA • Often larger grants than NSF • Often more applied than NSF • DOD, IARPA, NIH, etc. • Non-competitive sources • Industry • Often equipment donation • Sometimes graduate student support • Government • SC DoJ • Military, e.g. SPAWAR

10. Proposal Writing • Highly stylized form of writing • Sometimes pre-proposals are required • Sometimes Requests for Information (RFI) are issued • Proposal evaluation process is very long and uncertain

11. Example of RFI (purpose section) The Intelligence Advanced Research Projects Activity (IARPA) often selects its research efforts through the Broad Agency Announcement (BAA) process. This request for information (RFI) is intended to provide information relevant to a possible future IARPA program, so that feedback from potential participants can be considered prior to the issuance of a BAA. Respondents are invited to provide comments on the content of this announcement to include suggestions for improving the scope of a possible solicitation to ensure that every effort is made to adequately address the scientific and technical challenges described below. Responses to this request may be used to support development of, and subsequently be incorporated within, a future IARPA Program BAA and therefore must be available for unrestricted public distribution. The following sections of this announcement contain details of the scope of technical efforts of interest, along with instructions for the submission of responses.

12. Example of Funding Opportunity Announcement (FOA) This FOA, issued by the National Institute of General Medical Sciences (NIGMS) and the National Institute of Mental Health (NIMH), solicits applications for developing and testing innovative theories and computational, mathematical, or engineering approaches to deepen our understanding of complex social behavior. This research will examine phenomena at multiple scales to address the emergence of collective behaviors that arise from individual elements or parts of a system working together.  This FOA will support research that explores the often complex and dynamical relationships among the parts of a system and between the system and its environment in order to understand the system as a whole. To accomplish the goals of this FOA we encourage applications that build transdisciplinary teams of scientists spanning a broad range of expertise. Minimally this team should include senior investigators with expertise in the behavioral or social sciences as well as in computational and systems thinking (computer science, mathematics, engineering, systems-level methodology). Research can involve model organisms or humans. The FOA will support small research projects focusing on theory building and testing, development and testing of innovative methods or methodological approaches, as well as small infrastructure projects focusing on development and testing of shared resources (in the context of a driving biological, basic behavioral or social, or human health issue). The FOA also will fund larger and more integrative research projects focusing on the modeling of complex social behavior.

13. Decrease in proposal success rates coincides with increases in proposal submissions and average award size/duration

14. Funding Rate for Select Directorates Competitive Research Grants 50% 40% NSF BIO 30% CISE Funding Rate ENG GEO 20% MPS SBE 10% 0% 1997 1998 1999 2000 2001 2002 2003 2004 2005 All R&RA directorates experienced a decline in funding rates between FY 2000 and FY2005