Wikipedia talk:Village pump (proposals)/Sidebar redesign/programming

Easy vs. Hard

Changing items in the navigation box (or creating new boxes above search) is easy and requires only on wiki changes to certain parts of the Mediawiki: namespace. By contrast, changing the toolbox or creating new structures is hard and requires developer contributions (and some proposals may well be rejected for technical reasons).

Given this, I'd like to ask this group to consider making two rounds of proposals. First a proposal for easy changes that take no developer intervention, and then afterwards work on the hard stuff. Dragons flight 05:13, 28 August 2006 (UTC)[reply]

While that makes sense, I seem to recall that somewhere way back in ~~the dawn of time~~ this discussion, Rob said that the code needed an overhaul anyway. Besides, even if we did get the first half done, I can easily see the later half being procrastinated until Christmas and beyond. (sounds familar?) --DavidH Oz Au 10:25, 28 August 2006 (UTC)[reply]

Except that the programming will take a long time regardless, so why not do the parts that don't require programming sometime sooner? Dragons flight 10:52, 28 August 2006 (UTC)[reply]

That's what this project page and discussion is. However, I haven't got anything against making as much changes as possible and letting the developers code in the extra features when they have time for it. Heck, one possible solution is to use a hack where the PHP code just spits out the HTML (no wikicode parsing) in a fixed format with the decided layout? I admit that it is a rather ugly hack, but just think of all the code we don't have to write! ;-) --DavidH Oz Au 11:32, 28 August 2006 (UTC)[reply]

not possible for the toolbox, and certainly not desirable. --gatoatigrado 16:48, 28 August 2006 (UTC)[reply]

the coding shouldn't take longer than a few weeks; the design discussion has been going on for a long time anyway, and the current sidebar has been there a long time, plus a few weeks shouldn't make a difference. I experimented only changing the navigation and it didn't look very nice; many additions will put the search box below the end of an 800 by 600 window. let's just do it right. --gatoatigrado 16:50, 28 August 2006 (UTC)[reply]

reprogramming design

to maintain backwards compatibility, I think we should add a $this->data['sidebar2'], which will be displayed by an updated monobook.php.

should we parse regular mediawiki code and cache it? should we make a quick parser using regular expressions? How should we handle variables? This is what I'm thinking for example syntax

===== navigation ====
* [{{{mainpage-url}}} {{{mainpage}}}]
* [http://link listitem0]
* [http://link1 listitem1]
<hr />
* [http://asdf otherlistitem0]

===== search =====
{{#searchbox}}

===== toolbox =====
{{#if: {{#uploadsenabled}}|[http://specialpageupload upload]}}

a few things I realized while partially implementing this

it's inefficient to use variables as we do now (the {{{mainpage-url}}} in my syntax above), as the system has to make another sql query and get MediaWiki:Mainpage-url or whatever. Because sidebar is language specific anyway, should we just implement it there?
I am converting some of this to a data structure, so that individual style pages such as monobook.php can have their own <div> tags for the contents under each header or whatever.
there needs to be some new or differently used syntax. I think {{{variable}}} is consistent with templates, but some things aren't variables, such as the predefined searchbox html. If's also have to be implemented, and what about special variables that aren't in the mediawiki namespace, such as wguploadsenabled?

once I get my code to a reasonably working state I'll post it, or if I don't have time to work on it I'll post it. I've been gone for a while. --gatoatigrado 05:14, 28 August 2006 (UTC)[reply]

cache code

The simplest way would be to parse wikicode and then cache it inside a dynamically generated PHP file (with some logic inside said page if need be). Following that, add a bot or an admin-only purge function that updates said file whenever the layout changes. (However, you'll need a way to bypass it when something goes astray.) The other way is of course to do as you suggest. Caution: don't take this suggestion seriously, I am not a PHP rockstar; I am far more comfortable with Java or C/C++. --DavidH Oz Au 12:18, 28 August 2006 (UTC)[reply]

Thanks for the suggestion, but I don't agree completely. we should read

includes/ParserCache.php

before continuing, and note how the professionals do it. --gatoatigrado 17:24, 28 August 2006 (UTC)[reply]

there are more advanced caching libraries than do-it-yourself, I think apc might be one; hopefully they use memory instead of hd
writing files requires writing permissions in a directory
Using files too much also slows things down
The toolbox needs to be dynamic
Caching html ignores the needs of other style pages

list item style

I was thinking it might be neat to have list items with style, like

*|image="mainpage_icon_maybe_nuvola_home.png"|style="margin-bottom:3px;" [[main page]]

and different style pages could display it differently. --gatoatigrado 17:25, 28 August 2006 (UTC)[reply]

No icons. Please please please no icons. Too subjective, too distracting. Keep It Simple. --Quiddity 18:28, 28 August 2006 (UTC)[reply]

If icons were desired, they can already be added in CSS, no need to change the code for that. Dragons flight 18:35, 28 August 2006 (UTC)[reply]

icons for every list item? I don't think so. --gatoatigrado 02:46, 29 August 2006 (UTC)[reply]

If you look at the HTML, you'll notice that each list item acquires it's own id, which could theoretically be used to insert an icon or change the bullet into an icon. Dragons flight 03:38, 29 August 2006 (UTC)[reply]

Technical primer

The following is a technically oriented discussion of the Wikimedia setup and the consequent requirements for programming changes.

Wikipedia runs on Mediawiki software developed by Wikimedia staff member Brion Vibber and a number of volunteers. Every Wikimedia site runs on the same codebase. The differences between sites are controlled by 1) a LocalSettings.php file unique to each site, 2) MessagesXX.php files where XX is an appropriate language code and defines the default interface attributes for each language setting, 3) customization of the Mediawiki namespace. That's it. Everything else for every Mediawiki site is the same. This is done to keep the code base managable rather than having 100 different forks for all the different Mediawiki sites. As a result, any changes to files other than LocalSettings.php for the purposes of customizing this wiki need to be backwards compatible with all other sites using the same code.

really? explain the italian mediawiki then. i mean really explain it, with the code and everything. they must have changed monobook. there is no way to edit the mediawiki namespaces and put the "this page in wiktionary", etc. below the search box, as far as i am seeing the code in mediawiki 1.7 without modifying monobook.php. unless it's using something other than monobook. --gatoatigrado 03:00, 29 August 2006 (UTC)[reply]

Did you perhaps notice that those links only appear on the Italian main page? They are embedded at the bottom of the main page code and moved the side using some CSS tricks. If you look at the source for their main page you'll see a section title Progetti which accomplishes the trick. Kind of neat actually. Technically Wikipedia is running 1.8alpha at the moment. Dragons flight 03:50, 29 August 2006 (UTC)[reply]

oh, very neat, I had no idea that was possible. I guess we could do the same. --gatoatigrado 04:32, 29 August 2006 (UTC)[reply]

I guess with that method there's no way to put it between the search and the toolbox, but I think that's probably a minor issue. Anyway, if I ever code anything good I'll tell you; maybe it can eventually be adopted. --gatoatigrado 04:46, 29 August 2006 (UTC)[reply]

Mediawiki's basic layout is controlled by the "skin". Monobook is the default, but presently seven different skins are supported (look under My Preferences for options). These operate through Skin.php/SkinTemplate.php and skin specific files (e.g. Monobook.php). In particular, Skin/SkinTemplate do things like look up what should be in the toolbox and pass that information to Monobook.php, which then arranges it in the current format. Skin/SkinTemplate should be used for parsing things like Mediawiki:Sidebar into internal arrays (e.g. $bar, $nav_urls), that controls the selection of items on the Sidebar. After this is passed to a particular skin, e.g. Monobook, then that decides how the information is used. Extensibility is key. Functional upgrades should ideally be supported by all 7 distributed skins (hence it is prefered to modify Skin/SkinTemplate to the extent possible, before modifying individual skin files), and such upgrades should not break third party skins that work under the current version of Mediawiki.

The actual custom sidebar is built in includes/Skin.php. Variables are set with the "set" function in skintemplate.php and are available as the $this->data array in monobook.php. bar and navurls are local variables in includes/Skin.php:buildSidebar():1145 only. check with the latest code next time, unless wikipedia isn't running mediawiki 1.7. I don't mean to be disrespectful, but correctness counts. --gatoatigrado 03:00, 29 August 2006 (UTC)[reply]

My apologies, you are correct. Dragons flight 03:50, 29 August 2006 (UTC)[reply]

If the basic syntax used in Mediawiki:Sidebar or any of the other existing customization locations is to be changed then it either needs to be backward compatible, or a converter needs to be written that will allow existing Mediawiki sites seemlessly upgrade to the new syntax. Note that Mediawiki:Sidebar presently has an *A|B syntax, where wfMsg (and it's variants) are used to resolve the meaning of A and B by looking first at Mediawiki:A, then at MessagesXX.php, then at the default Messages.php file, and lastly treats it as an internal link if no intrinsic meaning is found. This is intended to ensure that interlanguage compatibility, and similar compatibility should be built into any modified code. Though this may look clunky, since the Sidebar is cached after the initial rendering, it doesn't significantly add to overhead.

yeah i figured this out; took me a while. i wish you had told me earlier lol. --gatoatigrado 03:00, 29 August 2006 (UTC)[reply]

yes it's cached if you enable it, and it only parses navigation anyway. --gatoatigrado 03:00, 29 August 2006 (UTC)[reply]

Lastly, and I cannot stress this enough, changing Mediawiki code means changing the code that operates all of the Wikimedia sites and the code that is distributed to third parties. This means thinking carefully about customizability, backwards compatibility, and interlanguage requirements. Changes to the code that break any of the other sites will not be accepted. Dragons flight 17:48, 28 August 2006 (UTC)[reply]

Good point, but it doesn't prevent a change of code. --gatoatigrado 03:00, 29 August 2006 (UTC)[reply]

PS. I forgot to note the existence of the CSS files, which are also a big part of the appearance. These include a global file for all distributions, a skin specific file for all distributions, a local global file (Mediawiki:Common.css), and a local skin specific file (e.g. Mediawiki:Monobook.css). Skin elements are generally implemented as divs to make full use of CSS. Dragons flight 18:38, 28 August 2006 (UTC)[reply]

None of the proposals have changed the css a lot --gatoatigrado 03:02, 29 August 2006 (UTC)[reply]

caching

okay, how do we do this dragons? the navigation was the only thing cached before because it was static. should we leave the individual toolbox items to be generated by includes/skinTemplate.php:buildNavUrls():771 and leave them out of the MediaWiki:Sidebar?

I don't like buildNavUrls() for a few reasons

It uses wfMsg for langauge compatibility; this is slow. It should be cached using something like

$parserMemc->get($wgLanguageCode.$varname)

It requires editing php to change it
It is unnecessarily skin-specific

the $parserMemc functions look easy enough; does this use APC? --gatoatigrado 03:28, 29 August 2006 (UTC)[reply]

if anyone is wondering, parsercache.php appears to be using $parserMemc from &singleton():16 which creates a new ParserCache with $parserMemc. --gatoatigrado 03:32, 29 August 2006 (UTC)[reply]

I'm not entirely sure what you are asking. When fully enabled, caching occurs at several levels. The whole rendered HTML is cached (in squids), the parsed version of each page is cached, and all of the messages (i.e. wfMsg results) are cached. (And possibly other places I forgot.) Could things be more optimized? Probably, but I wouldn't consider it your job to accomplish that. Mediawiki supports multiple caching frameworks, but I believe memcached is the most important in use for Wikipedia. Dragons flight 04:08, 29 August 2006 (UTC)[reply]

you're right, wfMsg is cached. I dunno why, but I looped it 10000 times and it was significantly slower than the preg functions. --gatoatigrado 04:59, 29 August 2006 (UTC)[reply]

are you sure it's squids? the wikipedia article looked like that was an html caching thing, and I thought during setup mediawiki said it was using APC to cache things. the mMemc variable used by all caching looks like it accepts arrays. --gatoatigrado 05:02, 29 August 2006 (UTC)[reply]

Squids are definitely used (as the last line of caching, but only for not logged in queries), but APC might get stuck in there somewhere too. Or perhaps APC is used unless stronger alternatives are set up. Dragons flight 05:28, 29 August 2006 (UTC)[reply]

sorry, maybe it's not APC. apc says it's a intermediate language thing, not a caching thing. i think you're right that it's memcached. anyway, i dunno why wfMsg was a bit slow, but I guess it's all a matter of microseconds lol. --gatoatigrado 07:17, 29 August 2006 (UTC)[reply]

(giggle) It's all of them. APC, shared memory caching, and Squid caches. 86.134.92.120 04:48, 30 August 2006 (UTC)[reply]

no, I think dragon was right, APC is not a caching thing. --gatoatigrado 18:32, 31 August 2006 (UTC)[reply]

Um, no, APC is a caching thing, just a different kind of caching thing. Whoops, what a stupid mistake, not bothering to check the facts. 86.143.70.12 14:36, 3 September 2006 (UTC)[reply]

sorry, where are you getting this? APC's website says it is an intermediate language parser. --gatoatigrado 18:14, 3 September 2006 (UTC)[reply]

never mind, you're right. sorry... but what did you mean by "different kind"? --24.7.86.143 06:10, 13 September 2006 (UTC)[reply]

Well, there's caches, and there's caches. APC caches on different levels to, e.g. the regular parser caches, or the Squid caches, or the nice, old-fashioned, client-side caches. There's caching all over the shop! 164.11.204.52 22:22, 24 September 2006 (UTC)[reply]

gatoatigrado's work

if anyone wants to see what I'm working on: User:Gatoatigrado/sidebarcode. This is intended to be a simple parser that allows all HTML and parses the result to a data structure so that individual pages can handle it separately; e.g. if one page doesn't want to create <div>s for the contents under a header.

please do not modify it or I will not see the changes. Feel free to annotate by breaking a pre section with some text, and make sure to break the nowiki tags as I described at the top. post suggestions here please and I'll change the code on the site and my code. thanks. --gatoatigrado 03:48, 29 August 2006 (UTC)[reply]

Simpler proposal

(Added from: Wikipedia talk:Village pump (proposals)/Sidebar redesign/Final draft vote#Simpler proposal, as requested)
I think that the work on the "toolbox" has been great. How about if we only implement that change, and allow for further discussion of the rest? I think that that has unanimous consent at this point. And it has the added benefit of being less work for the developers. - jc37 00:30, 16 September 2006 (UTC)[reply]

So including everything except the "other" (third) box, and it's content, and adding a divider in the upper box, for clarity (for now - until the other box is implemented), and of course including all the toolbox variations, it would look something like this:

(j)

search

toolbox

This could be implemented, while the discussion concerning the content and title of the "other" box continues. - jc37 15:06, 25 September 2006 (UTC)[reply]

No, the current version does not support hr's. As I said, just be patient, or help out. I will post some code up soon; I've been taking a different path (using lambda functions) and it's going pretty well. It's just a bit slow because there's no decent debugger for php that I know of. Most of the problems are small things which I could have fixed a lot earlier if I had a decent debugger. There will be some syntax weird things; I think these are still in effect with the current version, such as

{{#if: {{{something}}}|* list item}}

will need to be

{{#if: {{{something}}}|
 * list item
 }}

The headers will have a similar condition. --gatoatigrado 16:32, 25 September 2006 (UTC)[reply]

if you want to do something

If you want to help me without having to read my code, etc., I would appreciate someone finding out how MediaWiki implements hooks. --gatoatigrado 16:40, 25 September 2006 (UTC)[reply]

never mind, it's wfProfileIn($fname)...I found it. --gatoatigrado 17:46, 25 September 2006 (UTC)[reply]

I would also appreciate an expert opinion about the speed at which this needs to run. When I finish putting all of the features in, I think it should be able to evaluate the cached lambda expression maybe 2000 times / second on my Athlon 64 3000. I read Wikipedia gets 2000 requests / second and has 100 servers, but I don't know how well php works across a number of computers - could I just multiply the 2000 by 100 and aim for 1% of the processing time? Is that too much? --gatoatigrado 16:40, 25 September 2006 (UTC)[reply]

Would it be worth it to only parse the if statements, and leave all the rest to HTML? It wouldn't have the "syntactic sugar" but might be a more effective solution. --gatoatigrado 16:40, 25 September 2006 (UTC)[reply]

quick code needed

(unresolved) how to use parser_functions for {{#if }}, probably containing an efficient way of handling adjacent and sub-elements. --gatoatigrado 04:30, 30 September 2006 (UTC)[reply]

possible approaches

Here are some options I could think of. --gatoatigrado 02:10, 28 September 2006 (UTC)[reply]

html and if statements, cache after static replacements (e.g. main page link)
full parsing, cache a lambda function
keep most of syntax intact, edit MonoBook.php.

new mediawiki page

To maintain backwards compatibility and sanity, I think that this should use MediaWiki:Sidebar2 or something like that. I have coded this, as it is very simple, and if anyone wants it I can post it. --gatoatigrado 02:10, 28 September 2006 (UTC)[reply]

html and if statements (approach 1)

append this code to Skin.php before the end of the class. Image:Mediawiki html sidebar code.pdf. One last thing to do - make Monobook.php read $this->data['htmlsidebar']. There's also a small modification to SkinTemplate.php to store the htmlsidebar variable if it's not false. --gatoatigrado 17:49, 25 September 2006 (UTC)[reply]

parsing if statements (approach 1, 2)

If anyone has a good way to parse the if statements so they are nested correctly it would help a lot. I think it's done somewhere in the parser functions, but I can't understand their code. I would appreciate knowing how to use it so I don't have to write something that does the same thing, and probably not as well. --gatoatigrado 04:41, 27 September 2006 (UTC)[reply]

the quick hack (approach 3)

code at User:Gatoatigrado/sidebarhack for the first three boxes. I would like Wikipedia's toolbox because it has a modification to add the cite field. Thanks. --gatoatigrado 17:18, 29 September 2006 (UTC)[reply]

why the MediaWiki:Monobooksidebar is necessary. it couldn't be cached because the "#sidebar" box will show up as a header for an empty box. If it was quickly removed (because the skin wasn't monobook), the search box might be pushed down too far. If this could be coded more elegantly, it would be nice, but please ask first so it doesn't lose efficiency (or worse, accuracy). I didn't test out every case for the caching if's yet - it's sort of a pain to delete MediaWiki:Monobooksidebar and wait for all of the caches to be updated (usually involves a server restart). --gatoatigrado 04:24, 30 September 2006 (UTC)[reply]

the toolbox isn't quite done yet. The line doesn't disappear if there are no elements above or below. --gatoatigrado 15:57, 3 October 2006 (UTC)[reply]

Everything should work now. The code for the toolbox is much more concise. --gatoatigrado 20:27, 4 October 2006 (UTC)[reply]

bugzilla report

A request for review has been filed at http://bugzilla.wikimedia.org/show_bug.cgi?id=7493. --gatoatigrado 20:40, 4 October 2006 (UTC)[reply]

Looks good and I can't find anything wrong with it. The only thing missing from the proposal is 'cite this page', however, Special:Cite is not an official part of MediaWiki as you said earlier; the likelihood of its inclusion in MediaWiki's phase3 trunk is remote, so I'm not worried about it. Good job. :) --DavidH Oz Au 12:27, 5 October 2006 (UTC)[reply]