Foundation:Planning:Education:Evaluation

Evaluation methodologies for Mozilla education

This section is VERY DRAFT. Much discussion still needed.

As noted above the two high-level hypotheses we wish to test are that we can

We potentially have multiple ways of testing whether students have rich learning outcomes:

Student self-assessment. For example, we could survey students and ask them (among other things) to rate the value and relevance of their learning experience.
Instructor assessment. For example, instructors (and mentors) could rate students on their demonstrated levels of enthusiasm, dedication, and achievement.
Peer assessment. For example, students could nominate classmates for special recognition based on their overall contributions to the class experience.
Student behavior. For example, we could look at the level of student involvement in class activities as measured by the amount of code checkins, bug comments, forum postings, IRC presence, etc.

We also have several possible ways to measure resulting new ideas and contributors:

Mentor assessment. For example, we could ask mentors (and/or module owners and peers) to rate students and/or their projects based on their level of contribution to project goals.
Contributor assessment. For example, we could ask Mozilla project contributors in general to nominate students and their projects for special recognition.
New contributor metrics. For example, we could determine how many students achieve various contributor-related milestones (getting "canconfirm" status on bugs, having patches successfully reviewed and accepted, gaining commit access, etc.). If possible, we could compare students with a "control group" of contributors who came into the project through more traditional paths.

There are other dimensions on which we might evaluate our overall efforts, for example:

Student participation. This would measure the overall reach of the program and the extent to which participating students take on more active roles within the project. More specifically, some potential measures include
- Total number of students who participate to some degree.
- Number of students who successfully complete at least one substantial project.
- Number of students who continue participation past the end of their class(es).
- Number of students who become core project contributors.
Geographic reach. This would measure the extent to which we can achieve worldwide success in reaching potential student populations.
- Number of countries with participating institutions.
- Number of countries with participating students. (This assumes the possibility of some students participating outside a formal institutional framework.)
- Number of languages in which instruction occurs and for which educational material is localized.
Institutional adoption. This would measure the extent to which this paradigm is adopted within particular education institutions.
- Number of instructors participating within given institutions.
- Number of relevant classes within given institutions.
- Number of students participating within given institutions.
Breadth of offerings. This would measure the extent to which instruction occurs beyond just traditional computer science programs. For example, do we have participation by business schools? Design schools? And so on...