Computing desk
< October 23	<< Sep \| October \| Nov >>	Current desk >

Welcome to the Wikipedia Computing Reference Desk Archives
The page you are currently viewing is a transcluded archive page. While you can leave answers for any questions shown below, please ask new questions on one of the current reference desk pages.

October 24

Scoring for LLM generated Wikipedia Style models.

I am part of a team at a University where we are building a LLM style model which will be given a topic and will generate different subtopics and then text in order to write an informative article. We are going to be using several different types of scoring mechanisms, but we would ideally like to have frequent wikipedia editors collaborate with scoring the articles.

Is there a specific location that I could reach out to those who are frequently editing on wikipedia? Terribilis11 (talk) 00:23, 24 October 2023 (UTC) Terribilis11 (talk) 00:22, 24 October 2023 (UTC)[reply]

Please dont use LLMs to write articles. See the guidance in the essay at Wikipedia:Large_language_models RudolfRed (talk) 01:15, 24 October 2023 (UTC)[reply]

We aren't intending to use the LLM to write articles that we will then try to publish either on Wikipedia or the wider internet. Rather we are doing research, where Wikipedia is the gold standard we are trying to reach with our LLM. Terribilis11 (talk) 00:10, 27 October 2023 (UTC)[reply]

New articles submitted as drafts are often reviewed by participants in the WikiProject Articles for creation. Perhaps some of them may be inclined in participating in your experiment, in which case the discussion page of the Wikiproject is a suitable location. I think that in any call for cooperation you should describe the objectives of the effort. --Lambiam 11:40, 24 October 2023 (UTC)[reply]

I also wondered about the objectives. I get the feeling that the OP is looking for someone to do the grunt work to check the veracity of the LLM article. Every reference will have to be checked and vetted. LLMs are known to hallucinate. Very little - if anything - in an article like that can be taken at face value. 41.23.55.195 (talk) 05:51, 25 October 2023 (UTC)[reply]

It is not clear from the original request that the intention is to produce informative articles for publication. It is also not clear the texts will contain any references. Perhaps one of their aims is to compare the effectiveness of various methods for increasing reliability. --Lambiam 12:23, 25 October 2023 (UTC)[reply]

Thank you for the response. Our goal is for educational research. We are going to be building a LLM that will be focused on writing Wikipedia style articles with citations, and different sub-points. We will also have an automatic scorer that will score the essay based on 1. Well Written, 2. Verifiable with no original research, 3. Broad in its coverage, and 4. Qualitative comments (The first three metrics for a Good Article + Qualitiative comments). We would take a subset of our articles produced and score them by actual Wikipedia editors as a way to verify our scorer is within reason. Terribilis11 (talk) 19:45, 27 October 2023 (UTC)[reply]

In answer to the location question, there is Wikipedia:Village pump (miscellaneous). But as your project maybe educational, there is also Wikipedia:Education noticeboard. Graeme Bartlett (talk) 20:57, 24 October 2023 (UTC)[reply]

"Have you your axes ready?" MinorProphet (talk) 21:55, 28 October 2023 (UTC)[reply]