User:Grondemar/Essays/FAC/End user expectations and process capability

From WP:FA: "Featured articles are considered to be the best articles Wikipedia has to offer, as determined by Wikipedia's editors." Featured article candidates (FAC) as a process is designed to be a final quality check: before an article gets the star icon and is considered worthy of appearing on the Main Page, it must meet certain quality standards specified in the WP:Featured article criteria. All article improvement work is expected to have been completed by the time an article is submitted at FAC; while basic improvements are often made as part of the FAC process, any major concerns generally result in the archiving of the review.

Before discussing anything else about FAC, we need to answer two basic questions:

What do our end users expect out of the FAC process?
How capable is the FAC process of meeting end user expectations?

Below I will develop a framework to answer both of these questions.

End user expectations

The FAC process, like any other process whether in business, government, or the non-profit sector, has end users that have expectations as to what the process must deliver. Different end users will have different expectations and requirements, and will make different tradeoffs between the three fundamental factors listed below:

Quality
Cost (i.e., how much one must invest in order to receive the product. On Wikipedia this is measured in time and effort expended)
Speed (i.e., how frequently a product can be made available, which can also be measured as a rate such as X FAs per month)

In general, in order to increase any one of the factors, you have to decrease one or both of the others. For example, if you improve FA quality by tightening quality standards, you will end up either increasing the cost of the process (the time that needs to be invested by nominators and reviewers to improve articles to meet the new standards), decreasing the speed of the process (article reviews take longer to complete; fewer articles can be promoted), or both to varying degrees. Conversely, if you want to increase the number of articles reviewed and passed per month, you will either need to lower quality standards, increase the number of reviewers and/or the amount of time those reviewers spend reviewing, or both. The only way to avoid this tradeoff is if you can improve process efficiency. I will discuss this topic (waste reduction) in a future essay.

Types of end users

As I see it, there are four basic end users of FAC:

The Main page
FAC nominators
The general readership
The Wikipedia community

Main page

The Main page's requirement from FAC is to produce enough featured articles so that the Today's Featured Article slot never runs out of new articles. Right now there is still of backlog of 1,322 articles per WP:FANMP, meaning that if we stopped promoting featured articles entirely as of today (January 10, 2012), we would be OK until August 24, 2015. However, if we want to ensure the backlog never runs dry, we would need to promote a minimum of one featured article each day (365-366 per year).

Quality: Meets WP:Featured article criteria
Cost: N/A (Since all work is freely donated, the Main page is indifferent to the cost of the article)
Speed: 1 FA per day

FAC nominators

Nominators at FAC seek to have their work recognized as the best Wikipedia can offer. When they nominate an article at FAC, they are saying that they believe their article is a complete and comprehensive summary of the article subject that fully meets the featured article criteria. Nominators in general are indifferent to the number of articles promoted through FAC each month; they only care what happens to their article. They would prefer to have to invest the minimum required time to pass an article through FAC; this does not mean that they don't want to meet the featured article criteria or are looking to "game the system", just that they don't want to spend more time working on the article than is necessary to meet the criteria.

Quality: Meets WP:Featured article criteria
Cost: Minimize
Speed: In the micro (i.e. their article), as fast as possible; less concerned about the macro (i.e. total FAs per month)

General readership

It is more difficult to judge the end user demand for Featured Articles from the general readership. In the business world, sales is a good arbiter of what customers demand. Since articles on Wikipedia are free, however, this type of metric is unavailable. Page hits provide one kind of approximation, but since the cost of producing the article is not necessarily correlated with the number of hits the article gets, this type of metric is of limited utility. One of the basic assumptions of the FA process is that the general readership does demand high-quality articles; exactly how high-quality is difficult to answer.

Quality: High, at least complete and accurate if not FA quality
Cost: N/A (no cost is passed on to the general readership)
Speed: Hard to judge

Wikipedia community

The last end user of the FAC process is the Wikipedia community. If you reading this essay, you are probably a member of the community. The community has determined the quality standards at FAC by reaching consensus on the featured article criteria. Cost is harder to judge; certainly the less time and effort it would take to produce an FA while still meeting the criteria, the better. Speed... speed is the fundamental question, which will be the topic of the next section.

Quality: Meets WP:Featured article criteria
Cost: As low as possible while still meeting criteria
Speed: See below

Takt

Takt is a measure of end user demand. It is simply the number of units needed to be produced over a period of time in order to meet end user demand. For example, if you run a widget factory, customers demand 1,000 widgets per week, and you have 40 working hours per week (8 per working day) where you can run the factory, your takt would be 1,000 widgets per work week, 200 widget per work day, and 40 widgets per hour. If your production capacity does not allow you to produce 40 widgets per hour, you will not be able to meet demand. To maximize sales, you would need to increase capacity by either expanding the factory, adding a second shift, or improving the efficiency of your production process.

Takt is more difficult to measure for a process such as FAC, since end user demand is more vague. There are only two definite sources of end user demand: the Main page, and the Wikipedia community. If we use the Main page's requirement as our takt, it would be 365/366 per year, or about a net gain of 30.5 featured articles per month. Recently there have been suggestions from within the community at WT:FAC that a higher rate of promotion would be desired, often expressed in terms of the percentage or ratio of Featured Articles to the total number of articles on Wikipedia. Ratios as high as 10% (1 out of every 10 articles) have been expressed as desirable. In order to determine how many articles must be promoted per month in order to reach a particular ratio target, a deadline to reach the target must be set. For example purposes I selected December 2015 as the target date, or about four years from now. Assuming a continued linear growth of total articles at the past 3-year average rate, the below table shows how many articles would need to be passed each month in order to reach various target ratios.

(I apologize that the below table is an image versus a Wikitable. If someone could show me an easy way to convert an Excel file to a Wikitable, I would appreciate it.)

Since achieving a 1% (1 out of 100) ratio of Featured Articles required a monthly promotion rate of over 1,000 articles, which is probably safely beyond the capacity of the process at this point, I did not expand the table further. Note also that all of the FA counts were rounded to the nearest whole, since it is not possible to promote a portion of an article.

Process capability

The chart above shows what it would take on a monthly basis to reach various ratio goals by December 2015. However, before the community can decide on a takt, another question must be answered: what is the capability of the FAC process?

I examined the past three years of FACs (2009–2011) using the information available at WP:FAS along with the archived nominations log. Over the past three years, while there has been a total average monthly article increase of just over 32,000, there have been an average of about 71.7 FACs submitted per month. 38.6 FACs are promoted to Featured articles, while 33.1 fail to reach consensus and are archived. Over the same period, an average of 8.9 FAs per month have been demoted at WP:FAR. This results in an average FA monthly increase of 29.7, or 30 as rounded up in the chart above. This is less that the 30.4 articles needed to simply insure that the Main Page never runs out of FAs to display. As of today, FAC is not capable of meeting the takt of Today's Featured Article.

(Subsequent note: Two methodological errors have been subsequently pointed out by others in the above statement. First, the above calculation does not take into account articles demoted at FAR that were already featured on the Main Page. The ratio of demotions at FAR of articles that have already appeared on the Main Pages versus articles that have not has not been measured. Additionally, the above statement fails to take into account a margin of error that would make the difference between the current FAC takt and the Main Page takt not statistically significant. The above statement can therefore not be made with the certainly expressed. I sincerely regret the error. –Grondemar 01:59, 16 January 2012 (UTC))

Process control

The control chart is a tool to see whether a process is in statistical control. A range of three standard deviations from the mean in either direction forms what is called the upper and lower control limits (UCL and LCL). Assuming that the data follows a normal distribution, 99.73% of process results should fall within the six-sigma range. Any results falling outside the control limits indicate that most likely a special cause or external event has caused a change in the process. Variation within the six-sigma range is called common-cause variation, and reflects events internal to the process rather than outside forces. A narrower range of two standard deviations from the mean in either direction, called the upper and lower warning limits (UWL and LWL), can be used to see when results are approaching the outside limits of the process. 95% of events should fall within the four-sigma range; an event outside the warning limits, while not as significant as an event outside the control limits, is still something to keep an eye on.

The first control chart I will share is the number of FACs passed each month. Note that this is a gross count of FAC passes and does not take into account demotions at FAR.

Per the chart FAC has remained in statistical control at least until November 2011, when the number of FAs promoted dropped below the lower warning limit. I do note however that there appear to be patterns in this data: from January 2009 until September 2010 the number of FAs promoted clustered around an average of about 45. There was a notable decline in the count of FAs promoted in October 2010; a new average seems to form over the next several months at about 32. Another decline seems to occur in September 2011, eventually trending to the all-time low in November 2011. This chart would lead me to investigate whether there were events around those dates that caused the decline of FAs promoted.

The second control chart shows the number of FACs submitted per month. This includes both promoted FAs and archived nominations, but does not include any nominations that were deleted rather than archived.

This control chart shows a similar pattern to the last one, with the number of FAs submitted crossing the lower warning limit in November 2011. It is notable that the decline seen in the first control chart between September – October 2010 is even more pronounced in this chart: from an average of about 90 FACs submitted per month, submissions declined to an average of under 60 per month, before cratering in November 2011. This chart greatly increases my suspicion that events of significance occurred in September – October 2010 and November 2011 that had a significant impact on FAC participation.

The third and final control chart I would like to share is of the FAC yield rate; i.e. the percentage of submitted FACs that are promoted to FA status.

I find this control chart especially interesting for two reasons: first, at two occasions (December 2009 and September 2011) the yield rate cross the lower warning limit. In fact, in September 2011 the yield rate came perilously close to crossing the lower control limit, not just the warning limit. The other interesting aspect of this chart is that there is little if any correlation between the yield rate and the number of FACs submitted. If the problem was just one of reviewer participation, one would figure that a decrease in the number of submissions would increase the yield rate. There does not appear to be any significant yield increase attributable to the decrease in submissions. Of course, an equal decrease in reviewers would explain why yield would not increase.

Process yield

A key takeaway for me from the yield control chart is that the three-year average yield has been just over 53%. In the past three years, 2581 FACs have been submitted. 1390 have become Featured Articles; 1191 reviews were archived without being promoted. Think about that for a moment. Think about all the effort expended by nominators on over 1000 articles over the past three years, in order to bring them to featured article status. Think about how discouraged they must feel, having come so far and invested so much toil and sweat, only to see the ultimate prize denied. Clearly something is wrong when so much effort expended by so many editors goes to such waste.

Looking at the situation from another perspective, the yield issue is a clear opportunity for improvement. If we could improve submissions to the three-year trailing average of 72, and improve yield to 80%, we would achieve an average promotion rate of 60 articles per month. Not only would this promotion rate satisfy the takt of the Main Page, it would allow us to celebrate our 4000th featured article in September of this yield, promote our 5000th featured article in February 2014, and enjoy the fact that by the end of 2015 0.1175% of articles will be featured articles, about 1 out of 850 articles, and the highest percentage recorded in the history of the Featured Article program.

However, before we can celebrate, we must first answer another question: why do FACs fail? What are the specific issues that cause the majority of archived nominations to fail to achieve promotion? That will be the subject of the next essay.

Next: Why do FACs fail?