Jump to content

Wikipedia talk:STiki/Archive 1

Page contents not supported in other languages.
From Wikipedia, the free encyclopedia
Archive 1Archive 2Archive 3→Archive 5


Broken report to AIV

This report to AIV wasn't formatted properly – it seems that “/r” was inserted there instead of a newline. Svick (talk) 15:19, 26 March 2010 (UTC)

This was noticed a couple days ago and since has been fixed. Updating (re-downloading) to the most recent version will clear up the problem. Thank you for your continued use of the tool. West.andrew.g (talk) 15:44, 26 March 2010 (UTC)

A bit slow?

Just started experimenting with STiki on my mac. Going well, and nice work. Two comments. One that it looks like there are too many false positives, and also it feels a bit slow although I am on broadband. prashanthns (talk) 11:41, 24 March 2010 (UTC)

Hi Prashanthns -- Thank you for your pro-active interest in the tool. As the STiki page notes, improving the hit-rate (false-positives) is one of my major goals moving forward. I wanted to have a working system across the board before I delved seriously into any one part -- though the vandalism detection rate is likely the single most important! I'm glad to hear things worked on a Mac -- I've only seen the tool on Windows and Linux machines to date. Sorry to hear the edits advanced slowly. This isn't something I've ever experienced (maybe because my laptop resides just across town from my server?) -- but I am going to look into some caching. It is a bit puzzling though -- each button press hits my server once -- and the back-end Wikipedia databases twice (both for very small data exchanges). Just how latent would you estimate the edit advancement to be? Thanks, West.andrew.g (talk) 04:01, 25 March 2010 (UTC)
On my PC i find it takes 2 seconds to load each edit(this is at 9am and 2pm uk time), and about 3 seconds for the edit to take effect on the contributions screen. I find that huggle on my pc acts at about the smae speed.Ottawa4ever (talk) 15:11, 25 March 2010 (UTC)
The STiki version of 2010/4/2 eliminates any speed issues. Every non-visual action is now multi-threaded, and similarly, background threads are used to pre-fetch edits so there is no network latency when a new one needs to be displayed. West.andrew.g (talk) 21:06, 1 April 2010 (UTC)

My feedback

Hi, I'm trying STiki and it seems like a really interesting idea, but there are two issues I have with it:

  1. Quite often, I click on one of the three classification buttons and nothing seems to happen. Once I clicked “Vandalism (revert)” and the edit was actually reverted (and the user notified), but the GUI didn't change to another edit and won't change even if I repeatedly click “Pass”. Restarting the application fixes the problem, but it very soon surfaces again.
  2. The tool should use the standard “Month year” headings and warning with appropriate level, not {{uw-v2}} all the time.

Also I like to watch the talk page I post warnings to, so I would welcome this as an option. (STiki actually does this.)

Thanks for an interesting tool. Svick (talk) 21:53, 22 March 2010 (UTC)

Hi there Svick. Thanks for your interest and download of the tool!
  1. On your first point (jamming) -- this isn't a behavior I've seen before. Edits always advance on my machine (Linux and Windows) -- and this isn't the same machine that hosts the back-end processing and database. I'm not sure of your technical expertise (so just ignore if I confuse you) -- but if your running the *.JAR from a terminal, does anything get printed to the terminal (stderr?) when these jam-ups occur? The reports of latency I've heard are puzzling -- each RID-advance pings my server once and the WikiMedia database twice (each with trivial payloads).
  2. On your second point (warnings) -- I spent a considerable amount of time today overhauling the warning structure. I now parse the UserTalk page of the offending editor, looking for previous user-warnings (by month, year), and then append to the appropriate section, incrementing the warn-level by one. If one vandalizes with a relevant uw-4 still active - they get reported at AIV. It's a best-effort though: between Huggle, Twinkle, and all types of home-brewed strategies there are many different formats out there for doing this, but I am pretty satisfied.
Again, thanks for your review -- and hopefully after many iterations with users like you, I will have a tool that both fulfills my academic needs and those of the Wikipedia community. Thanks, West.andrew.g (talk) 04:26, 25 March 2010 (UTC)
  1. Strange, I'm unable to reproduce the problem now. Everything works fine.
  2. Thanks a lot for this change.
Svick (talk) 23:22, 25 March 2010 (UTC)

I've been experiencing the above mentioned problem again. This time, I ran STiki from console and got the following messages: 

com.mysql.jdbc.CommunicationsException: Communications link failure due to under lying exception:

** BEGIN NESTED EXCEPTION **

java.net.SocketException
MESSAGE: Software caused connection abort: socket write error

STACKTRACE:

=== SNIPPED ~100 lines (see history) === (West.andrew.g)

Svick (talk) 22:29, 29 March 2010 (UTC)

Hi Svick. I know this is a bit of a long-shot, but had you had the program open for 8+ hours when this error occurred? Or has it recently happened multiple times in short succession? All the documentation I can find pointing to this error indicates it is the result of connection time-out (which is set at 8 hours). As you might have noticed -- it is happening deep within an external library, so I haven't had much luck tracking it down in my own code. Again, thank you for your support. West.andrew.g (talk) 07:03, 30 March 2010 (UTC)
No, I certainly didn't have the program open for more than 15 minutes. But yes, usually, when it happens once, then it soon happens again. Svick (talk) 16:45, 30 March 2010 (UTC)
Hi again Svick, I just posted a new version of STiki. With respect to the error you were getting, I changed the JDBC (database connector) from mysql-connector-3.1.14 to version 5.1.12. I am not positive this has fixed the problem (it has never happened on my machine). However, the MySQL forums are dense with discussion about this error in 2006 and 2007 -- but nothing recently -- so I suspect it might have gotten handled somewhere along the line. I was only using the older version because it has a smaller footprint, and well, "if it ain't broke, don't fix it". Thanks again for your support of STiki, and let me know if issues continue. West.andrew.g (talk) 23:01, 30 March 2010 (UTC)
It doesn't seem it helped:
Failed to populate metadata object:
RID in question is: 352977278
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link fai
lure

The last packet successfully received from the server was 21á793 milliseconds ag
o.  The last packet sent successfully to the server was 21á234 milliseconds ago.

== SNIPPED ~100 lines (see history) == (West.andrew.g)
Svick (talk) 23:51, 30 March 2010 (UTC)
Hi again Svick. I've spent a good 6 hours trying to track down this bug today, to no avail. You seem like you have some programming experience. Can you characterize any environmental characteristics when it happens? Do you have a fast and reliable Internet connection? Anything weird about your network setup? From what I've learned, it seems certain that the connection between my server and your machine is dying or getting killed (rather than an in-code error), but the time-elapsed seems too brief to trigger any sever time-outs. Sorry to bug you again, but I am really grasping at straws on this one. Thanks, West.andrew.g (talk) 05:11, 31 March 2010 (UTC)
Yes, I have programming experience, although not in Java. I can try to debug the problem (I know it's hard to do if you can't reproduce the problem yourself), if you tell me what to look for. There's nothing unusual when it happens. I'm using Windows Vista and Java 1.6.0_18. I have fast (measured now at 25 MBit/s download and 4 Mbit/s upload and 102 ms average ping to www.cis.upenn.edu) and reliable internet connection. It's a university network, so there are rules forbidding some kinds of traffic, but I think that shouldn't be causing this. Svick (talk) 09:35, 31 March 2010 (UTC)
Hi again Svick. I've made some changes and I'm confident I've made progress. I've wrapped every DB-call in a try-catch block looking for this specific error. If the error occurs, it seamlessly resets the connection and re-attempts the operation under the new connection. The fixed version is available for download (2010/04/01). Of course, this only corrects things when the error happens, rather than preventing the error in the first place. Why the error occurs is still a bit perplexing. One thought: Your stack trace indicated it had been 21,000 milliseconds (20 seconds) since the last database communication. Do you always see figures this high, or has a stack trace ever shown a small number (say, less than 5,000 milliseconds?). If it is always high, it could still be a time-out issue somewhere along the line -- and maybe I could prevent it by spawning a thread that just pings the server once every 1--5 seconds. Similarly, do these errors occur during frequent use? or maybe after you've navigated away from STiki to fix something manually on Wikipedia, initiate a rollback, etc.? Thanks again for your help and support. West.andrew.g (talk) 21:38, 31 March 2010 (UTC)
So far I haven't seen this problem with the new version. It may be possible that it appeared always after I didn't use STiki for several seconds, but I don't know this for sure. Is the exception logged if it is catched? In other words, should I run STiki from the command line to see if it writes anything? Svick (talk) 22:06, 31 March 2010 (UTC)
Nope, no output. As long as the program keeps running -- I'm going to consider it a success. There is always the chance that a different error will be thrown, though I haven't heard anything about this. Thanks, -AW West.andrew.g (talk) 22:30, 31 March 2010 (UTC)
Unfortunately, the error still appears in the latest build:
Failed to populate metadata object:
RID in question is: 353066363
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link fai
lure

The last packet successfully received from the server was 21á739 milliseconds ag
o.  The last packet sent successfully to the server was 21á292 milliseconds ago.

== SNIPPED ~100 lines (see history) == (West.andrew.g)
Svick (talk) 22:34, 31 March 2010 (UTC)
Hello again Svick. Another try (version is 2010/4/2). I noticed your last two stack traces had "last communicated" durations of just over 20 seconds. I implemented the "ping" strategy I discussed above. The client now pings the MySQL server once every 5 seconds with a trivial query. In my research I found its not always the client or the server that can terminate idle connections -- but network switches sometimes also get involved. I hope this attempt works. You'll probably also notice that STiki runs much much quicker this version. It is now multi-threaded with all non-visual processing happening in the background. Most critically, edits are pre-fetched into a small cache so there is no network latency when the edit needs advanced. Hopefully the multi-threading doesn't cause you any problems that it didn't give me. Thanks again for everything, West.andrew.g (talk) 21:03, 1 April 2010 (UTC)
So far no problems with the new version. And I really like the speedup. I only have one small issue – I have gotten few times the same edit twice to review. I guess this could be some issue with the caching. 02:45, 2 April 2010 (UTC)
A big whew on the dropped connection point for the time being. Now about this edit repeating: There are two possibilities: (1) The same person is committing the same kind of edit on many pages (category vandalism/spam are frequent with this), just making it look the same edit, or (2) You are editing a little slower than I had anticipated. When you ask the server for edits, you get a "reservation block" with a time-to-live (10 minutes for 20 edits). If you haven't finished those 20 edits in that time-frame, the database will give them to the next person who asks (and this could very well be "you" again -- but the first copy may still exist in the queue, just unclassified, leading to two copies of the same edit in the queue). Either way, its trivial to fix by just making a list of what's been seen -- I've already uploaded a new version with the fix (but under the same/old filename). Thanks, West.andrew.g (talk) 03:38, 2 April 2010 (UTC)

Personal Experinces and some bugs noticed

Ive given STiki about 3 tries so far. Each version seems to improve on the other. I thought it prudent to share my experiences about this tool, as obviously other vandal fighters are considering new tools all the time. The software does exactly what its said to do in that it compliments tools like huggle. Where huggle gives you an up to date as it happens queue, this digs a bit deeper in the history and catches older vandalism (I do not not use twinkle so i will not make any comparisons to that software) that’s usually 3 hours old. It often catches vandalsim about 5 minutes old as well. This is very significant. As we have many huggle/twinkle warriors out there but its difficult to catch those that are overlooked in the queue as these tools are used very speedily, so for a vandal fighter who prefers to spend time finding and spotting older embedded vandalism this is an extremely useful tool. That’s what I like.

Now what I’ve noticed that i think needs improvement (at the time of writing i am referring to the march 25th 2010 version); I think there is a small glitch in the formation of reporting users to AIV. I found that my report was sent after the individual was block, it would be nice if there was a way to check that if a user was blocked before filling (minor incidence) Also i think the formatting is a bit skewy when posting at AIV there. What id also like to see is a warning that you are also editing as an IP when you start up. The time of edit would also be useful (is that available anywhere that i may have missed?)The final comment i have regards to a rollback function. A few edits i did reverted the initial edit of the individual but left other incidences of vandalism (since rollback is not featured). At a slower pace this is fine because youd go thorough your edits and this can often happens in huggle anyway with rollback, when people game the system. It would be beneficial i think to incorporate the tool using the rollback much like huggle does. The program is young though. And constantly changing I applaud this program and encourage others to give it a thorough test to. Sorry if this sounds like a blog but i think for a program thats just starting out its nice to share abit of feedback and experiences from other users. Happy editing, and once again well done. Ottawa4ever (talk) 15:11, 25 March 2010 (UTC)

Thank you for your kinds words (the changes I mention here will be present in the next update):
  1. I've now corrected the AIV problems (formatting). Further, I make an explicit blockage check before reporting over at AIV (For some reason, late last night I had it in my head "such a check is stupid -- if they were blocked -- they couldn't have edited in the first place!". Of course, the contradiction here is that STiki may be displaying edits that occurred long ago, and the user may have been blocked since.
  2. Displaying the time of the edit? Ditto, I'll get right on that. I think it may be most useful to put this in relative terms: "This edit was made X hours and Y minutes ago".
  3. Rollback. This is something I'm thinking about. Since the edit-window only shows a single 'diff' -- it would be hard to know exactly how much content you are rolling back if you hit the button. Right now, if I see multi-edit vandalism I usually just click on the page history link and initiate a rollback myself.
West.andrew.g (talk) 20:56, 25 March 2010 (UTC)

Anonymous users

This can be used without logging in, right? Isn't that potentially hazardous? We could have IP edit wars or 4chan attacks using it. Can this functionality be removed?  fetchcomms☛ 02:57, 26 March 2010 (UTC)

It also trips the editfilter about every edit.  fetchcomms☛ 03:22, 26 March 2010 (UTC)
It is correct that the tool can be used without logging in. However, this tool provides IP-users no functionality they wouldn't have otherwise. So in that respect, I see no harm done. Further, Wikipedia makes it clear that we should not hold bias against users who choose to edit anonymously. It is ultimately the responsibility of the user operating any tool to act properly (and other tools do permit anonymous use). If someone wanted to be malicious, they could abuse the API in *much* worse ways than my tool could facilitate.
On the second point, I assume it is only anonymous use of the tool that trips the 'editfilter'. Exactly which filters are being hit? -- I assume those involving the edit rate of anonymous editors? This is more concerning to me. I guess a solution is to encourage users to log-in via a warning dialog -- which also notes the potential problems with the edit filter. West.andrew.g (talk) 05:08, 26 March 2010 (UTC)
It allows IPs to rapidly revert edits--this is potentially disruptive, because simply editing as an IP does not allow me to click a button and revert edits easily. Instead of manually vandalizing, they could just click the "revert" button as fast as they wanted, switching IPs after becoming blocked and causing more damage, faster. Obviously, anyone who knew how could make a vandalbot or some sort of device, but the amount of vandalizing IPs who would not know how to do that is far greater than those who do. How many IPs actively fight vandalism and would require this tool? In addition, an IP marking vandalism as "innocent" would mess up the predictions, right? The filter hit is the unregistered user rapidly reverting edits one. Personally, I feel that making this sort of tool available to unregistered users should require community consensus first.  fetchcomms☛ 15:03, 26 March 2010 (UTC)
Anonymous users probably shouldn't be allowed to use this tool. It could be applied to destructive purposes with very little consequence.-168.9.25.23 (talk) 15:37, 26 March 2010 (UTC)
I agree. Having definitive user login will also enable me to prevent abuse of the prediction system. I will take the steps to disable this functionality (it could take a small amount of time) and perhaps open discussion somewhere regarding if it should be re-enabled. Thank you for interest and discussion regarding my tool. (For posterities sake, note that the above IP throwing in his/her opinion is a vandal that I and others have reverted on many occasions). West.andrew.g (talk) 15:50, 26 March 2010 (UTC)

← And thanks for understanding. I do like the feel of this tool, and hope that it's uses will be expanded (different warning types, etc.) I also like that it's built on Java, which I feel makes it a more lighter interface and easier to use.  fetchcomms☛ 16:02, 26 March 2010 (UTC)

This feature change has been made per the version of 2010/03/28. West.andrew.g (talk) 22:20, 28 March 2010 (UTC)
Thanks; is there any way to disable anonymous use of the previous versions, or should one just hope that the original doesn't fall into the wrong hands?  fetchcomms☛ 22:32, 30 March 2010 (UTC)
No, there is no way to disable the functionality in prior versions. However, I am able to monitor who is using the tool. Thus, if an IP address was to start using an older version of the tool, I would certainly give these reverts some examination. 99% of the use I have seen is not by IP users and the older STiki versions are not available for download. Further, the tool is yet to be advertised, so the user-base is not large. In other words, I am confident there will be no issues. Thanks, West.andrew.g (talk) 22:51, 30 March 2010 (UTC)
OK, I thought something like that, and it should be fine, as you said. Thanks!  fetchcomms☛ 23:20, 30 March 2010 (UTC)

Relationship to Huggle

Unlike the bulk of existing anti-vandalism tools, STiki does not rely heavily on natural language processing (e.g., regular expressions) over the article or diff text.

nor does Huggle

Instead, STiki spatio-temporally processes primarily revision metadata (edit timestamp, article, editing-user, and revision comment) to arrive at vandalism predictions.

so does Huggle

Just saying... Gurch (talk) 09:22, 17 May 2010 (UTC)

Hi Gurch. I have experimented with Huggle and believe you have an excellent anti-vandal tool. My comments (i.e., what you quoted above), are a bit out of context on STiki's Wikipedia talk page, because they were basically copied verbatim from my academic writings. In that realm (the academic one), the use of metadata is quite rare, and the use of NLP is far more common. My apologies for not handling my description with more clarity. Indeed, as I note elsewhere I have no intention to compete with your tool and its massive user-base -- mine is far more proof-of-concept.
On a related note, I am sure your tool is the most effective at anti-vandalism on Wikipedia (both in terms of raw revisions, and hit-rate). A white-paper describing your technique and your statistics would be well received (and probably generate plenty of citations). I am sure your vandalism statistics, when aggregated, would tell a quite interesting story. West.andrew.g (talk) 18:30, 18 May 2010 (UTC)
I'm not saying that Huggle is more effective than STiki (or anything else). If effectiveness is defined in terms of successful classification as vandalism/not-vandalism, Huggle fails miserably because it doesn't actually do that. :) Speaking of a hit-rate is also meaningless with Huggle, because the decision to revert is made by the user, so you're actually just measuring how good the user is at spotting vandalism. Your tool likely has the potential to be very effective at classification, since it has a learning element of sorts. (I would certainly hope that a fancy Java spacio-temportal processing engine is better at that than a few hundred lines of Visual Basic, anyway). But as I'll explain below, that doesn't really matter.
I mainly wanted to point out that most existing patrolling tools work the same way -- not so much because it better, but because so many edits are made that loading the text of every diff in real time isn't really feasible. So what the introduction presents as something new, isn't so much, though it looks like by far the best implementation of it. (The learning aspect isn't completely new either, in that I can think of at least two attempts to make tools based on it, neither of which really worked). Of course, bots have to compromise, because they have to do the task of reading the diff that a human would do with other patrolling tools, but automated vandalism-detecting bots that take action aren't really something I agree with.
Huggle's main focus is on improving the efficiency of the patrolling process, not of detecting vandalism. After all, if the process is efficient enough that one person can keep up with all changes to the wiki (more or less possible with Huggle at present), it hardly matters. So a single keystroke will:
* carry out a bunch of checks to try to avoid mistakes
* revert all consecutive revisions by the same user
* parse that user's talk page to find previous warnings
* leave an appropriate higher-level warning
* report the user if they were already warned enough
Diffs are pre-loaded before the user views them, so usually the wait time for one to load is close to zero. Web requests are all made in parallel in the background, so Huggle doesn't have to wait until it's done warning one user to revert another, and the user doesn't have to wait for anything -- they can get straight on with viewing the next diff while the reverting/warning happens. For this specific activity, this is an order of magnitude more efficient than the standard MediaWiki UI (and even tools like Twinkle), where the limitations of the Web interface get in the way. Other features have ended up in MediaWiki having first been implemented in Huggle (for example, showing the diff of a rollback after doing it, something Huggle originated to try to make users spot mistakes).
Anyway, as for the processing, which you're probably more interested in:
* automatic page-blanking summaries are listed first
* reported/blocked users are listed next
* warned users are listed next
* anonymous edits are listed next
* edits to articles are listed next
* then everything else is listed
* users with more than 500 edits are ignored
That's it. And yet it doesn't matter that this isn't very good because the patroller is keeping pace with recent changes and will see the revision within seconds anyway. Huggle does not look for key words or do pattern matching on edit summaries, user names, page titles, article content, diffs or anything else. It's far too subjective for a computer program to do in an error-free manner.
It's worth me disclosing at this point my views on the abuse filter, which does do pattern-matching of those things, and patterns implemented by overzealous administrators with a poor understanding of regular expressions at that. The abuse filter is in my opinion a poorly-designed, hurriedly-implemented, fundamentally flawed, misused piece of junk that has driven away useful potential contributors without actually benefiting the project in any way; the revisions it is catching are either harmless or would have been reverted within seconds anyway by Huggle or similar tools (or even a bot).
Gurch (talk) 09:31, 19 May 2010 (UTC)
Some changes made to the STiki description to address this discussion. Thanks, West.andrew.g (talk)

Razi High School edit

As for the Razi High School, Why did you revert my post? Please let me know your reasons for doing that? I had just added the jersey color of the school team! BTW I am a Razi graduate and used to be in Razi High School Varsity team. Good Old Days!--71.62.41.187 (talk) 00:43, 19 July 2010 (UTC)

Ill take a liberty and respond here breifly, West.andrew did not make that edit (though the program he wrote allowed the other editor to make the change), you can directly query the user who did the reversion ; here at User:Dripping Flame/ir. Ottawa4ever (talk) 08:54, 19 July 2010 (UTC)

Suggestion regarding classifications

There are some more possibilities to consider in the "classification" buttons.

  • The edit may not be vandalism, but is otherwise deficient and needs to be reverted while preserving good faith.
  • The edit may be questionable but in a way that can't be determined solely from a diff - small changes to dates or numeric values are a common example.
  • The edit may not be vandalism, and may not be something we want to revert, but is still deficient in such a way as to require further (re)working.
  • The edit may be perfectly fine, but in seeing the part of the article given as diff context, there may be something else wrong.

Other than that, looks like a great tool so far, keep up the good work. Triona (talk) 06:45, 7 August 2010 (UTC)

Thanks for your kind words, Triona. Multiple edit classifications (vandalism, spam, good-faith revert), or even taggings, have long been on my radar for STiki. The rigors of graduate school have prevented me from implementing them thus far. I'll note that this would take care of your 1st and 3rd points above. Regarding point #2: I have tried to preference my machine-learning classifier against displaying edits that have JUST numbers, because I know this is a hard case for non-experts. Nonetheless, they still show up some times. Regarding point #4: This is a difficult case. Nearly all review tools (STiki, Huggle, etc.) -- seem to stick the 'diff' strategy -- and honestly this is something I have no intention of changing. West.andrew.g (talk) 14:27, 11 August 2010 (UTC)

Warning Notices

I don't think the warning given to the user should link back to STiki... It's fine to do it in the edit summary, but putting it in the warning that the user gets will just confuse them, not to mention the fact that it makes this page prone to vandalism, and it could be considered spam. - EdoDodo talk 12:06, 7 August 2010 (UTC)

The requested changes have been completed in-code. This code will become public-available at the next release. I can't do much about those that have already downloaded a previous version, though. Thanks for your comments, West.andrew.g (talk) 14:31, 11 August 2010 (UTC)
Okay :). - EdoDodo talk 16:29, 11 August 2010 (UTC)
You might want to consider adding a "kill" for older versions, by checking a page or within the backend. Generally, this shouldn't be necessary, but if a bug is found that may actually damage the wiki, or if the backend communication changes, it should prevent or strongly warn against continued use of the old version. Triona (talk) 17:47, 11 August 2010 (UTC)
I agree. Forcing users to upgrade, like AWB does, is annoying, but can be useful. - EdoDodo talk 17:54, 11 August 2010 (UTC)
I have this functionality (though un-intentionally). STiki clients must ask the server for a RID to review. If disaster were ever to strike, I could simply blank this queue (and STiki wouldn't work). More elegantly, I could populate the queue with only a single RID (perhaps in my user-space), which instructs them to upgrade. Then, the next version would point to a different queue, and all would be well. I don't see it being relevant for this minor change, though, correct? West.andrew.g (talk) 18:00, 11 August 2010 (UTC)
No, clearly not for this, just making sure you have a way to clear out "bad" versions if such an event ever happens. Triona (talk) 18:04, 11 August 2010 (UTC)
Yeah, not for this, just if some major change has to be done. Sounds good :). - EdoDodo talk 22:08, 11 August 2010 (UTC)

U IOWA Vandalism detection research

Didn't know if you saw this:

ABSTRACT: This paper proposes an active learning approach using language model statistics to detect Wikipedia vandalism. Wikipedia is a popular and influential collaborative information system. The collaborative nature of authoring, as well as the high visibility of its content, have exposed Wikipedia articles to vandalism. Vandalism is defined as malicious editing intended to compromise the integrity of the content of articles. Extensive manual efforts are being made to combat vandalism and an automated approach to alleviate the laborious process is needed.

This paper builds statistical language models, constructing distributions of words from the revision history of Wikipedia articles. As vandalism often involves the use of unexpected words to draw attention, the fitness (or lack thereof) of a new edit when compared with language models built from previous versions may well indicate that an edit is a vandalism instance. In addition, the paper adopts an active learning model to solve the problem of noisy and incomplete labeling of Wikipedia vandalism. The Wikipedia domain with its revision histories offers a novel context in which to explore the potential of language models in characterizing author intention. As the experimental results presented in the paper demonstrate, these models hold promise for vandalism detection.

Ocaasi (talk) 11:46, 24 September 2010 (UTC)

Indeed, I happen to have a copy of that paper on my desk right now! I haven't delved into it in great detail yet, but plan to do so in the coming days. Would you happen to be one of the authors? West.andrew.g (talk) 18:18, 24 September 2010 (UTC)
No, intra-article linguistic frequency data... it just looked like it was right up your alley.
BTW, I don't know if you tweaked the algorithm, or if STiki's machine-learning is just working, but the vandalism ratio is down to about 1 in 3 from 1 in 10. Much more enjoyable to use. Also, definitely nicer without all of the number-only diffs.
A side-effect of the blazing speed is that the chance of accidentally reverting to another bad edit has gone up. Rollback would obviously be ideal, but an inbetween fix would just be including the i.p./username of the prior edit. It's usually a dead give-away when a vandal-edit was preceded by the same author; very unlikely they're not both bad.
I was trying to think of people in the community you might want to talk to if you have time.
  • RobLa WMF developer, into all of the big projects
  • Werdna I think he's done WP:edit filter specific work, among other things
  • Gurch The main guy behind Huggle, which probably accounts for 60% of the recent-changes patrolling. He's pretty much seen it all when it comes to vandalism.
Looks like things have quieted down on the controversy front, which is good. I think I can speak for those with less itchy outrage-fingers that your continued presence around here can do a lot for some of the persistent problems this project faces.
Cheers, Ocaasi 00:15, 25 September 2010 (UTC)
Thanks for the pointers. The major change in performance was likely due to a shift in ML approach. Previously I was using Support Vector Machines and I've now switched to Alternating Decision Trees (you may have also noticed that the occasional edit from a registered user now appears). ADTrees have inherent support for missing features and enumerated ones, which is a big help.
Another interesting fact is how the shared edit-queue effects perceived performance -- it's a Cache-22 of sorts. If no one knows about STiki and it has only a few users, they all think it is great, because they sign on and there is a large amount of vandalism at the top of queue waiting to be handled. These people tell others about the performance -- and the user-base expands. Then, the vandalism is distributed among this larger set -- and they don't perceive the performance as being nearly as great. Thanks, West.andrew.g (talk) 16:44, 25 September 2010 (UTC)

Error when running

I got the following error when running for the first time: 

$ java -jar STiki_2010_07_17.jar 
Exception in thread "main" java.lang.UnsupportedClassVersionError: Bad version number in .class file
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:260)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:56)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:268)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)

I'm running Mac OS X 10.4 with JRE version 1.5.0_19.

Errr... I might have compiled with Java 1.6. That would explain the "UnsupportedClassVersion" exception. Time for an upgrade? You could also recompile from the source version, but that would require a couple of libs. West.andrew.g (talk) 14:11, 17 August 2010 (UTC)
I think the last time I tried to upgrade to 1.6, it told me I needed OSX 10.5, which is not an option for me. (X! · talk)  · @704  ·  15:54, 17 August 2010 (UTC)
I am working on a new version, due for release in the next couple days, at the latest. I'll compile the *.JAR with Java 1.5 -- and give you a ping when that is available. (Edit: And sorry about the TB template, just saw that message the second time through) Thanks, West.andrew.g (talk) 15:57, 17 August 2010 (UTC)
Took a little longer than expected, but the front-end has been recompiled with a Java 1.5 target. Thanks, West.andrew.g (talk) 18:26, 26 August 2010 (UTC)

Still getting an error: 

Exception in thread "main" java.lang.NoClassDefFoundError: java/awt/Desktop
        at executables.stiki_frontend_driver.initialize_button_panel(stiki_frontend_driver.java:457)
        at executables.stiki_frontend_driver.<init>(stiki_frontend_driver.java:215)
        at executables.stiki_frontend_driver.main(stiki_frontend_driver.java:177)

(X! · talk)  · @705  ·  15:54, 5 September 2010 (UTC)

User warning for old edits

I just reverted several cases of serious vandalism that were 10 hours old – way to go STiki! However, STiki refused to warn the users because the edits were too old, even though the user talk pages were empty. I went ahead and did the warnings with Twinkle. It would be great if STiki did warnings like Huggle in actually comparing the time of the last warning on the talk page and assigning the next higher warning level if the reverted edit is newer. —UncleDouggie (talk) 09:06, 3 October 2010 (UTC)

Probably be nice if you can simply customize (or have the option to customize) the warning template (Havent used stiki in a while, i think you can...). I agree though that the user on stiki should have an option to over rule if neceesary though, sometimes you know its a static IP/ same person. But Im worried over the fact that users might simply tag a talk page without considering it being a dynamic IP (using the approporate warning for this), which if its the wrong template can be bitey to a passer by. So a prompt ask to warn would give the user a few seconds more to consider customizing a warning or not. Ottawa4ever (talk) 10:31, 3 October 2010 (UTC)
I would rather just integrate the Huggle code. —UncleDouggie (talk) 04:12, 4 October 2010 (UTC)
I like the Huggle idea. But yes, as Ottawa pointed out, the real purpose of the parameter was for dynamic IP addresses. As you observed, STiki sometimes displays vandalism that is 10+ days old. It seems inappropriate to warn an IP such a situation, no? Registered users are always warned, regardless of elapsed time. I'll add this to a growing TODO list. Nonetheless, what do you think is a good default duration when IP warning is still appropriate? West.andrew.g (talk) 06:28, 4 October 2010 (UTC)
24 hours sounds about right. An exception is if the talk page has one of those glaring University shared IP templates on it, but I suspect that Huggle already deals with this properly. —UncleDouggie (talk) 09:17, 4 October 2010 (UTC)

Bug report

I don't think this is vandalism. I don't know how this script works, so I've contacted the user and those here as well.—Ryūlóng (竜龙) 21:11, 8 November 2010 (UTC)

Thanks Ryulong. You can understand why 'Inoue Joe HELLO!.jpg' looks a little spammy. It'd be great if you or someone (me perhaps) would add an edit-note above it so that no one is confused. <-This is the real album name... !> that kind of thing. regular STiki user, Ocaasi (talk) 21:41, 8 November 2010 (UTC)
Hi Ryulong. STiki isn't a "script" per se -- it just shows users diffs which *may* contain vandalism. Thus the responsibility of making the vandalism/innocent distinction falls to individual users. Any issues should be taken up with the reverting user. Thanks, West.andrew.g (talk) 23:05, 8 November 2010 (UTC)
Sure, blame it on the alcohol Ocaasi (talk) 23:15, 8 November 2010 (UTC)

Stiki down?

I'm not getting any revisions. Arthena(talk) 08:57, 26 November 2010 (UTC)

Now it works again. Arthena(talk) 12:05, 26 November 2010 (UTC)
I was getting some weirdness last night, also. No good explanation at this point. However, I was working on a updated version of STiki -- and an error of some kind might have trashed the server state. I'm not going to fret about it unless it appears again. However, do look for the improved version in the next couple of days. I've been (a) implementing rollback, and (b) handling all the bug reports filed on this page and my talk page. Thanks, West.andrew.g (talk) 14:44, 26 November 2010 (UTC)

Need rollback

I just reviewed an edit that was obviously vandalism. In the diff, I saw that the old rev also had different vandalism. The history showed that the previous vandalism was added by the same IP on the previous edit. So, I did a rollback from the history page and then clicked "Vandalism" in the classifier to help out the STiki rating algorithm, even though I knew there was no longer an edit for it to undo. It would be nice to be able to go back to previous diffs right in the diff-browser and execute a rollback, just like when using Twinkle. —UncleDouggie (talk) 08:45, 3 October 2010 (UTC)

It is my intention to include the "rollback" option in STiki much as Huggle does (either using "rollback" naturally for those who have it -- or implementing "in software rollback" for those who do not). I don't know how to best integrate "diff display" so you can be sure you got it all though? Currently I just use the link I provide to the page history and use rollback there -- but I recognize this is inelegant. West.andrew.g (talk) 06:28, 4 October 2010 (UTC)
Rollback functionality (both "native" and "software") are now implemented in STiki. See the notes about the 2010/11/28 release. Thanks, West.andrew.g (talk) 03:28, 28 November 2010 (UTC)

Failure to revert

I just classified this edit as vandalism and STiki failed to revert it saying that I had been beaten to the revert. However, this was still the latest edit on the page! After this, I did a manual rollback. A contributing factor may have been that I sat on the prior edit I was reviewing for about an hour while I reworked that editor's good faith attempt into something presentable. I then clicked innocent and was immediately shown the Fahd of Saudi Arabia edit. —UncleDouggie (talk) 10:37, 3 October 2010 (UTC)

I just had the same thing happen. In this case, the same edit was made twice, with the first immediately reverted by someone else. Why did STiki not revert the second edit? —UncleDouggie (talk) 11:00, 3 October 2010 (UTC)

The only way out of this was to close and reopen STiki, after which it started doing reverts again. It showed me logged in the whole time. —UncleDouggie (talk) 12:31, 3 October 2010 (UTC)

I suspect your problem is related to the "I sat on the prior edit for about an hour" bit. When one uses STiki, they "reserve" blocks of ~10 edits at a time that they are given to work on (so the client can go ahead and process the API calls, so network latency doesn't slow the user experience). However, after a period of time these reservations expire (on the assumption the client died unexpectedly, without releasing the reservation). I'm assuming this expiration screwed things up -- but I cannot wrap my head around the minutae of the problem at this moment. I'll think about this for a while. West.andrew.g (talk) 06:39, 4 October 2010 (UTC)

Minor edits?

I think it would be good if there was an option to mark STiki's edits as minor. By Wikipedia:Minor#When_to_mark_an_edit_as_a_minor_edit, removal of blatant vandalism should be marked as a minor edit. Arthena(talk) 19:55, 6 October 2010 (UTC)

Acknowledged, in the next release I will ensure that both the reversion and the user-talk warnings are marked as minor (though I believe the latter already are). It shouldn't even need to be an option, should it? Thanks, West.andrew.g (talk) 14:46, 7 October 2010 (UTC)
True, It doesn't need to be an option. I was just thinking that sometimes I come across an edit in STiki that is not vandalism, but I do want to revert it for some other reason, and then it would not be a minor edit. But in those cases I should just tag it as 'innocent' and make the revert manually in my browser, because reverting it in STiki would mistrain the algorithm. Arthena(talk)
When a change is required that STiki can't handle (non-vandalism, rollback, etc.), I make the change manually and then "pass" so as to not mess up the training. The edit won't actually go to anyone else because it's no longer the most recent. It might be better to click "vandalism" in some cases where I do a rollback, but I've stopped this because STiki sometimes added a second user talk page warning right after the warning I had just added from the rollback. —UncleDouggie (talk) 17:16, 7 October 2010 (UTC)
Aha, interesting. When I don't want to give a warning, I just disable STiki warnings by clicking on the checkbox in the bottomleft corner. Arthena(talk) 08:27, 8 October 2010 (UTC)
This is a better solution for the vandalism rollback case. I'm torn on the non-vandalism problems. I want to train STiki to catch them, but I'm hesitant to actually click vandalism for fear of messing up its detection of real vandalism. We need more buttons! —UncleDouggie (talk) 09:42, 8 October 2010 (UTC)
Fortunately, in non vandalism cases where the edits are worth undoing for other reasons, it's not a 'terrible' thing if STiki catches them. The salient traits of those edits are likely not altogether different from most vandalism. I also use Arthena's method of toggling the user warning. A simple good-faith revert button would solve all of the issues.Ocaasi69.142.154.10 (talk) 00:21, 10 October 2010 (UTC)
Closing note here. STiki now marks its reversions as minor. See the notes about the 2010/11/28 release. Thanks, West.andrew.g (talk) 03:31, 28 November 2010 (UTC)

Bug report: ampersand in article title breaks 'Current-Page' and 'Page-Hist' buttons

If an article name contains an ampersand ("&"), then the 'Current-Page' and 'Page-Hist' buttons do not function correctly. For instance, for the article "Maram & Aabroo", the mentioned buttons will send you to "Maram" instead. This is a very insignificant bug, but I thought I would mention it for completeness' sake. Arthena(talk) 19:07, 8 October 2010 (UTC)

I was wondering why the buttons sometimes don't work! I'm not sure that this is the only problem, but I'll now know what to look for. Thanks. —UncleDouggie (talk) 23:31, 8 October 2010 (UTC)
Hello everyone. Thanks for reporting this bug. I realize I've been a little latent recently, but I my schedule seems to be clearing for mid-November. At that point, I'll have some time to implement these feature requests and fix these bugs I've been promising due for so long. Thanks for your feedback and continue use of my tool. Thanks, West.andrew.g (talk) 22:42, 1 November 2010 (UTC)
Special characters in article title's should no longer break outgoing links. See the notes about the 2010/11/28 release. Thanks, West.andrew.g (talk) 03:32, 28 November 2010 (UTC)

STiki Release -- Nov. 28 2010

Greetings STiki users, a new version was uploaded tonight! I encourage everyone to download it and begin trying it out. Per the CHANGELOG, some of the bigger changes include:

  • Rollback implemented. Rollback is now an option when reverting edits (as opposed to simple "undo"). For users who have the "rollback" permission, this was straightforward to implement. For those who do not, a more expensive "software rollback", that mimics the native permission, was encoded. To this end, a more informative revert-feedback panel now notes both the technique used, and the number of edits undone/RB'ed.
  • Per WP:MINOR, all STiki reverts are now marked as `minor'
  • The "current page" and "page history" links now handle special characters in article titles appropriately. In particular, this fixes a reported issue with ampersands.
  • User's whose last warning was a "4im" of *any* kind (not just vandalism), will now be reported to AIV if reverted by STiki. The post to AIV will reflect the unique nature of this request.

As STiki grows more complex, it becomes increasingly difficult for me to test all the corner cases. I'd appreciate if everyone continues to post their bug reports and feature requests here. Thank you, and happy reverting! West.andrew.g (talk) 03:14, 28 November 2010 (UTC)

Nice! Arthena(talk) 10:38, 28 November 2010 (UTC)

Recent performance hiccup fixed

Greeting STiki users. Something funky happened on the STiki server a couple of days ago, and as a result, new edits were not being processed in the back-end. Thus, these edits were not being added to the queue, and STiki users were only seeing old edits that had little probability of being vandalism.

I was alerted to this issue when I noticed that only 1-5% of STiki-viewed edits were being classified as vandalism (in the past several days). Typically, this percentage hovers in the 33-50% range. Rest assured that everything has been fixed, and that performance levels should once again be as expected. Thanks, West.andrew.g (talk) 21:19, 6 December 2010 (UTC)

Bug

Threaded-POST attempt (reversion) failed:
java.lang.NullPointerException
        at java.net.URLEncoder.encode(Unknown Source)
        at mediawiki_api.api_post.edit_rollback(api_post.java:149)
        at gui_support.gui_revert_and_warn.run(gui_revert_and_warn.java:173)
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
        at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
        at java.util.concurrent.FutureTask.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source
)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

Caused program to hang ungracefully and then crash. 930913 (Congratulate/Complaints) 13:14, 17 December 2010 (UTC)

Thanks, a result of the recent change to include rollback functionality. I believe I already have a fix locally, and will push it out with the next release. Thanks, West.andrew.g (talk) 15:04, 17 December 2010 (UTC)

Bug 2

Ungracefully crashed; from cmd, salvaged this error message. 

        at java.lang.reflect.Constructor.newInstance(Unknown Source)
        at com.mysql.jdbc.Util.handleNewInstance(Util.java:409)
        at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:357)
        at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:285)
        at java.sql.DriverManager.getConnection(Unknown Source)
        at java.sql.DriverManager.getConnection(Unknown Source)
        at core_objects.stiki_db_con.get_con(stiki_db_con.java:55)
        at executables.stiki_frontend_driver.reset_connection(stiki_frontend_driver.java:289)
        at gui_support.gui_ping_server.run(gui_ping_server.java:59)
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)

        ... west.andrew.g SNIPPED some 100 lines ...

Caused by: java.io.EOFException: Can not read response from server. Expected to
read 4 bytes, read 0 bytes before connection was unexpectedly lost.
        at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:2503)
        at com.mysql.jdbc.MysqlIO.readPacket(MysqlIO.java:600)
        ... 21 more

930913 (Congratulate/Complaints) 23:36, 19 December 2010 (UTC)

Per our IRC discussion (and just to formally close this for the rest of the world)... Looks like your network connection had a hiccup. STiki tried to reset and failed. Not much to be done in terms of *fixing* the issue -- but the Exception should (and will) be more gracefully caught. Likely a side-effect of your port tunnelling, which should go away if we transition to an HTTP protocol. Thanks, West.andrew.g (talk) 15:49, 24 December 2010 (UTC)

Spacebar keystroke

Two questions related to how the spacebar is functioning in STiki:

1) Spacebar seems to be advancing to a new diff but I don't know if it's using Vandalism/Innocent/Pass. I don't intend it to do any of these, but if it does, which one is it?

2) Spacebar on most browsers is a way to advance a page downward (page down). This feature would be useful in checking diffs, which don't always show up in a single pane (especially using larger font sizes). The desired functionality could be a global spacebar shortcut to advance the page down or one which only worked after clicking on the diff itself. Is something like this possible? Ocaasi (talk) 18:01, 10 December 2010 (UTC)

For those who don't use a mouse -- the ALT+mnemonics+arrows are used to navigate to buttons in a GUI -- and the spacebar is used to "fire" a button. So if you are using a mouse, clicking on a button will give that button "focus" (indicated by a light gray rectangle) -- and pressing the spacebar will fire the button that has focus. Thus, pressing space bar will repeat the last action taken. If you use the mouse to indicate "Innocent" and then press the spacebar, that second edit will also be labelled as "Innocent".
(In response to the second question) This is conventional behavior for the mouse-less crowd, so I am not inclined to install different behavior -- at least when the classification buttons have focus. However, I can enable it such that when the browser window has focus (i.e., by clicking on it), that a spacebar press fires a scroll-down action. Seems reasonable enough, look for it in the next release (within a few days). Thanks, West.andrew.g (talk) 21:28, 12 December 2010 (UTC)
Sweet. Makes sense on both counts (although I always thought that Enter fired alt+shortcuts). Anyway, clicking on the diff to enable a spacebar/page down will definitely enhance efficiency. Thanks again... Ocaasi (talk) 23:43, 12 December 2010 (UTC)
Ok, super minor and completely after-holiday worthy--the spacebar advance is only advancing a single line at a time, as opposed to a single paragraph or pane or page. I don't know if 'page down' is actually too much, but at least 5 lines, maybe 10 or 15 would be needed to make it efficient. Thanks again, Ocaasi (talk) 08:50, 27 December 2010 (UTC)
Changed in code; will be included in next release. Thanks, West.andrew.g (talk) 03:49, 29 December 2010 (UTC)
Included in the 2011/01/31 release. Just be careful with the spacebar in this fashion to make sure you don't make accidental classifications. Closed. Thanks, -AW

Dec. 26 -- STiki down for maintenance

Happy holidays, STiki users. STiki will be taken off-line for several hours today for maintenance -- I am aware of this down time. When it comes back online, old versions of the client will not work -- but there will be a new version available for download.

There is at least one big change in the new version which should make the trouble worth it -- MULTIPLE "revision queues." Whereas STiki's metadata-driven scoring system was the *only* one used in previous versions, "Cluebot" and "WikiTrust" now contribute their scores. I'll post the full CHANGELOG here when everything is online and operating. Thanks, West.andrew.g (talk) 15:54, 26 December 2010 (UTC)

New version available for download and servers are back up. Let the bug reports starting flowing (*sigh*). I'll change the main STiki page here shortly and post the CHANGELOG to the talk page. In the meantime, the documentation internal to STiki has been updated, or really, just look at the "Queues" menu. Thanks, West.andrew.g (talk) 19:46, 26 December 2010 (UTC)

STiki Release -- Dec. 26, 2010

The following is the CHANGELOG from the 12/26/2010 release: (note that a change in communication problems will require that users of previous versions update their software)

  1. QUEUES. Prior to this version, the STiki GUI had only one queue -- an internal one based on edit metadata. (Note: A queue determines which edits are displayed to a user, based on vandalism probabilities. Techniques vary on how one can arrive at a probability). Now, the "Queue" menu offers an explicit choice among several queues. In addition to previous "metadata" queue, there are two other options: (1) "Cluebot-NG" -- those edits whose score was below the threshold required for automatic bot revert (or couldn't be reverted due to 1RR or other rules). (2) "WikiTrust" -- based on the reputation system of Adler et al.. This is a significant change, and more information is available in the STiki doumentation.
  2. Minor changes to AIV reporting. Both reports and comments now have some additional detail to help the admins at AIV.
  3. If the 'diff browser' window has focus, the spacebar key will now fire the 'scroll down' action, per a feature request @ WP:STiki.
  4. Significant changes to communication primitives. All client calls to the STiki database are now handled by stored procedures, as a matter of security. Frontend GUI users should notice no change as a result. All stored procedure calls are logged and can be audited. This change will break previous versions of the client.
  5. For the backend reputation component (for users and articles), the reversions of the recently approved "Cluebot NG" bot are now included.
  6. Back-end STiki scores, rather than being on [-inf,inf], now harness the logistic nature of ADTree scoring to normalize scores onto [0,1] (making them probability-esque). This transformation maintains relative ordering so frontend users should notice no change. This transform was also applied to all historical scorings (API accessible).

Thanks, West.andrew.g (talk) 03:42, 29 December 2010 (UTC)


Archive 1Archive 2Archive 3→Archive 5
Retrieved from "https://en.wikipedia.org/w/index.php?title=Wikipedia_talk:STiki/Archive_1&oldid=1137452765"