Drumbeat/p2pu/Assessment and Accreditation/Webcraft Assessments - detailed/Methodology v1

Determining skills of interest – a methodology

Background

For the P2PU/Mozilla School of Webcraft project, we need to identify a suite of skills (competencies, habits, knowledge, etc) that are of particular interest to employers and collaborators who need to evaluate the potential for someone to be a good professional fit for their needs. In actuality, this problem is ubiquitous for most disciplines, especially when it comes to non-knowledge-based abilities, but we are focused on the School of Webcraft here.

We have talked to experts in the field, and we plan to continue doing so. But there is a need for a more comprehensive, replicable methodology for determining the "desirable qualities" of someone in an open-source programming environment. The following approach has not been tested, to our knowledge, but seems worthwhile, and may prove to be generally applicable.

Methodology

1. Using a professional social network (such as LinkedIn), identify to classes of people (there is likely to be some overlap): 1) people who identify as web developers, and 2) people who manage or hire web developers.
2. In each case, browse the following items, if present:
   1. Recommendations from other people.
   2. Recommendations for other people.
   3. Job ads they have posted (that are relevant).
   4. Descriptions of themselves.
   5. Any other information that seems relevant.
3. Cut and paste into a separate document or spreadsheet those parts of the browsed items that refer to specific skills and habits, which should (as a rule) be positive.
   1. This process will result in a database of sorts composed of text snippets which refer to skills and habits that are considered desirable by web developers and those who employ them.
4. The resulting database can be analyzed by hand, or by text-analysis software, to extract common words and phrases that refer to desired competencies. This process will produce a new database.
5. This final database serves as the raw material for any additional consideration of the listed attributes. It will likely need some additional sorting and aggregating, as well as some evaluation from experts and the broader community as a form of validation for the results.

Comments

We believe this process is robust and replicable. In aggregating the individual texts, the origin URLs and other relevant data should be captured as well so that others can perform their own analyses of the information if they wish.

There may be ways to streamline this even more, for example by querying the social network itself for certain types of words and phrases (the reverse of what is suggested above) and then noting the relative incidence of the different text-items.

If we really want to demonstrate the potential for this approach, we should recruit several different people to perform parallel analyses (i.e., pursue the same questions on the same system, but independently) so that we can evaluate the extent to which the resulting conclusions differ from each other. Some diversity is fine, but if this methodology is sufficiently objective, we should see broadly similar results for each of the efforts.

Data

First analysis of LinkedIn data - UK company ads

We generated a spreadsheet of ad copy within LinkedIn for many different web developer positions posted by companies. The file is available here: File:P2PU work(uk).odt.

Using the AntConc 3.2.1 program, we generated a word list based on frequencies of use within the ads. There are other types of analyses we could perform as well, but it makes more sense to first explore the raw data for patterns and then analyze specific text snippets based on those analyses. The top words (in descending order to a minimum of 3 hits) associated with desirable "habits" were:

1. Team (8 hits) - as in a team player or working as part of a team.
2. Agile (7 hits) - as in agile software development.
3. Creative (5 hits) - as in strong, flexible, creative flair.
4. Writing, written, and verbal (5,3,3 hits) - as in excellent and proven writing skills or excellent written and verbal communication skills.
5. Communication (4 hits) - as in excellent communication skills.
6. Analytical (3 hits) - as in strong analytical skills.
7. Attention (3 hits) - as in attention to detail.
8. Attitude (3 hits) - as in positive attitude.
9. Designing (3 hits) - as in experience designing and optimizing code.
10. Fast (3 hits) - as in working in a fast paced environment.
11. Thinker (3 hits) - as in logical thinker or critical thinker.

A quick review of this list reveals no real surprises in terms of the types of skills and habits that are in demand by employers of web developers. Most employers want creative and analytical thinkers who work well on teams, have strong communication (written and verbal) skills, and can work quickly and flexibly. The bigger challenge, of course, is figuring out ways to authentically identify these traits in the employee pool.

We will be performing a companion analysis using recommendations from colleagues which are more likely to contain words and phrases that are less standardized and perhaps more revealing. As the dataset grows, we will also be able to group and outline typical words and phrases into larger categories and gain some analytical power as a result, especially for better understanding exactly which types of web developer positions favor certain skills and habits over others.

Second analysis of LinkedIn data - personal recommendations in LinkedIn

This spreadsheet contains snippets from personal recommendations that people wrote for each other and posted to LinkedIn. All of the people here have some responsibility for web development.

Again using the AntConc 3.2.1 program, we generated a word list based on frequencies of use within the recommendations. The top words (in descending order to a minimum of 3 hits) associated with desirable "habits" were:

1. Team (4 hits) - as in a team player or working as part of a team.
2. Leader (3 hits) - as in strong group leader.
3. Communicator (3 hits) - as in great communicator.

These were the only words in common for the raw descriptions to the minimum standard of 3 hits. Obviously, we could use more data. But this small list is also reflective of the vastly greater diversity in the terminology people use to describe each other, as opposed to the semi-standardized terminology that you find in job descriptions (see above). It would be worth taking some time to browse the descriptions manually and do some grouping and outlining.