Wikipedia:Page Reformat Crisis

By the end of 2013, the Page-Reformat Crisis had become a major problem, with some templates flooding the wp:Job_queue(s) with so many pages triggered, when updating the templates, that it has taken over 6 weeks(!) to update all related pages (which formerly ran within 2-3 days in 2008). The prior, slow Edit-preview Crisis, which used 12-45 seconds to preview changes to most major articles, was solved, in April 2013, by fast-cite templates, now using Lua script in wp:CS1 Module:Citation/CS1, plus fast Lua-based Module:Navbox and Module:Infobox or Template:Weatherbox (etc.) to cut edit-preview of pages as 3x-4x times faster than February 2013. Meanwhile, the epidemic growth of ever-more templates (plus Lua modules) has bogged down the page-reformat queues with millions of jobs to re-re-re-reformat the same pages for multiple changes to whichever megatemplates they are using. A recent victim has been Template:Convert, transitioned from 3,000 tiny subtemplates, to instead use a gargantuan Lua Module:Convert (at 02:15, 11 December 2013) but still reformatting the final 7,000 of the 554,000 related pages, almost 21 days later.

Potential solutions to page-reformat crisis: There are several possible options, but Step 1, is to beware people imagining, "It's not a problem" (no), because a delay of 2-to-7 weeks to show results of updated templates is too long, even if people try to rationalize the mega-slow results. You might recall, when the page "Canada" edit-preview ran 28 seconds or "Israel" ran 42 seconds, some people were saying not-a-problem and recommending to edit each major article by section only, but no longer seeing the whole page reformatted to proofread for typesetting. So, such solutions by lowered-expectations are a reluctant option, but would treat this website as a miserly operation. Instead, let's try to gain major performance improvements:

• Perhaps reformat most articles daily: It might be possible to reserve some job-queues to simply force every page to reformat daily (or within 3 days) rather than hope the regular queues finish within 2-to-7 weeks.
• Trigger reformat of major articles daily: Perhaps all major articles could link to a dummy template, to be edited every day (or two), and then process those 300,000(?) pages daily, which would by-the-way relink to any updated templates/modules which were enhanced during the forced daily reformatting.
• Add a job-priority option for major updates: With a privileged user-right, it might be possible to have an admin submit a requested major template update into a high-priority queuing status, so that a new feature, or major template bugfix, could be propagated system-wide faster by reformatting a million related pages within a week or such.
• Re-split large templates for use in fewer articles each: The sub-optimized design of the former, markup-based Template:Convert proved a feature could be added into one subtemplate and appear within 28,000 pages a day later. So perhaps some of the mega-templates could be split into forks which affected only 50,000 pages each.

Fortunately, most templates rarely change major features, and so people have not been complaining (much), as they wait for a new option to appear in the final 7,000 pages within a half-million. However, as we upgrade to have smarter, auto-correcting templates, rather than issuing 1950s-style error messages ("ERROR: Invalid data; DOES NOT COMPUTE") as busy work to hand-edit, there is the need to re-install system-wide waves of auto-correcting updates (see: "wp:Autofixing cites"). In those cases, a high-priority of reformatting could post results weeks sooner, to check for unexpected consequences, and then re-release the next wave of smarter template within the week. Also, the flood caused by wp:data hoarding has multiplied thousands of minor pages with tedious data-table templates. Anyway, this is just a quick overview of the crisis, where corrections or updates to some major templates do not post current data for over 6 weeks of reformatting.

• Things to do before Wikipedia becomes popular: In terms of overall priorities, the above Page-Reformat Crisis is an issue which should be resolved as Wikipedia grows larger, and the delay to reformat pages, after updating a mega-template, expands from a few weeks into months of waiting. During 2010, some users imagined the English WP would not exceed 4.4 million articles, already exceeded in late 2013. Along with plans to fix the auto-merging of many wp:edit-conflicts, the faster reformatting of major pages is a primary issue which affects many users everyday, seeing the outdated results from old templates linger week after week, or month after month.

• Mass reformatting slows updates for smaller templates: As part of the "ripple effect" from reformatting millions of pages for mega-templates, the changes to smaller templates cause their related pages to wait longer in queues, such as reverting hack edits which show adverts in popular templates. In prior years, an advert could be removed from a thousand related articles within an hour, but now it might linger for days due to waiting along with each mega-template to meanwhile reformat a million pages during 3-5 weeks.

See also

• wp:Overlink crisis
• wp:Wikimedia Foundation error (WFE)