User:Jonathanveedot/sandbox

Ranking and Assessment

The distribution of UGC across the Web provides a high volume data source that is accessible for analysis, and offers utility in enhancing the experiences of end users. Social science research can benefit from having access to the opinions of a population of users, and use this data to make inferences about their traits. Applications in information technology seek to mine end user data to support and improve machine-based processes, such as information retrieval and recommendation. However, processing the high volumes of data offered by UGC necessitate the ability to automatically sort and filter these data points according to their value.

Determining the value of user contributions for assessment and ranking can be difficult due to the variation in the quality and structure of this data. The quality and structure of the data provided by UGC is application-dependent, and can include items such as tags, reviews, or comments that may or may not be accompanied by useful metadata. Additionally, the value of this data depends on the specific task for which it will be utilized and the available features of the application domain. Value can ultimately be defined and assessed according to whether the application will provide service to a crowd of humans, a single end user, or a platform designer.

The variation of data and specificity of value has resulted in various approaches and methods for assessing and ranking UGC. The performance of each method essentially depends on the features and metrics that are available for analysis. Consequently, it is critical to have an understanding of the task objective and its relation to how the data is collected, structured, and represented in order to choose the most appropriate approach to utilizing it. The methods of assessment and ranking can be categorized into two classes: human-centered and machine-centered. Methods emphasizing human-centered utility consider the ranking and assessment problem in terms of the users and their interactions with the system, whereas the machine-centered method considers the problem in terms of machine learning and computation. The various methods of assessment and ranking can be classified into one of four approaches: community-based, user-based, designer-based, and hybrid.

• Community-based approaches rely on establishing ground truth based on the wisdom of the crowd regarding the content of interest. The assessments provided by the community of end users is utilized to directly rank content within the system in human-centered methods. The machine-centered method applies these community judgments in training algorithms to automatically assess and rank UGC.
• User-based approaches emphasize the differences between individual users so that ranking and assessment can interactively adapt or be personalized given the particular requirements of each user. The human-centered approach accentuates interactive interfaces where the user can define and redefine their preferences as their interests shift. On the other hand, machine-centered approaches model the individual user according to explicit and implicit knowledge that is gathered through system interactions.
• Designer-based approaches primarily use machine-centered methods to essentially maximize the diversity of content presented to users in order to avoid constraining the space of topic selections or perspectives. The diversity of content can be assessed with respect to various dimensions, such as authorship, topics, sentiments, and named entities.
• Hybrid approaches seek to combine methods from the various frameworks in order to develop a more robust approach for assessing and ranking UGC. Approaches are most often combined in one of two ways: the crowd-based approach is often used to identify Hyperlocal content for a user-based approach, or a user-based approach is used to maintain the intent of a designer-based approach.

Community-Based Approaches

The community-based approach relies on establishing ground truth based on the collection of judgments received from a crowd of humans regarding the content of interest. The assessments provided by the community of end users is utilized to directly rank content within the system in human-centered methods. The machine-centered method applies these community judgments in training algorithms to automatically assess and rank UGC.

The human-centered method is ubiquitous across applications, and utilizes the wisdom of the crowd to assess and rank the quality of content by enabling users to contribute ratings, reviews, or tags. These crowd-based assessments are then directly used to organize content for the community of end users. It is critical, however, to consider that user ratings and reviews are often grounded in a specific context under a variety of motivations, and are influenced by their awareness of previous ratings and reviews. Additionally, the voluntary participation of users in assessing content does not guarantee the accuracy or reliability of that assessment, and the contribution of user assessments may often be sparse.

Machine learning is implemented in machine-centered methods, and leverages the judgments of the community of end-users as inputs used to classify and manipulate UGC. Most machine-centered approaches implement supervised learning algorithms that rely on user ratings as input, but performance can be significantly improved by including features related to text, the user community, and user feedback. In some cases, community assessment may be influenced by various biases (e.g., rating bias, winner circle bias, early bird bias), and judgments from an external crowd are used instead. In addition to assessing and ranking the quality of UGC, some applications seek to label content according to its type. Lexical features are often used to train Bayesian classifiers or other algorithms to label content according to the types that are of interest (e.g., objective vs subjective, conversational vs informational, editorial vs news).

How quality is defined is contingent upon the application, and various dimensions of quality have become more significant in machine-centered assessments of UGC. The task of determining the usefulness of content is application-dependent, so it is important to train the selected algorithm with respect to topic-specific items rather than a more general abstraction. Semantic and topic-based features serve as significant inputs for assessing usefulness, and features representing subjective tone, sentiment polarity , and the recognition of named entities contribute significantly to classification accuracy. In some contexts, such as product reviews, assessing the usefulness of content is more straightforward, and learning algorithms perform successfully using content length, subjective and objective tone, readability (e.g., frequency of spelling errors), deviation of content rating from aggregate ratings, and author reputation. Features related to text and content are often combined with author-based features (as well as item-based features in the case of item reviews) to detect deceptive content (spam). It should be noted that rating-based features do not contribute significantly to the classification of deceptive content since ratings are also often spammed. Additionally, linguistic features alone are found insufficient for accurate classification, but n-gram features are known to have the highest impact, especially when combined with psycholinguistic features.

The rapid advancement of new media brought about interest in implementing machine-centered methods to assess UGC in terms of popularity, credibility, and relevance. Author-based and temporal features are significant predictors of the popularity of content, but the effect of temporal features varies from platform to platform. Most of the approaches for assessing the credibility of content use supervised learning based on assessments from an external crowd. Text and content features are often insubstantial, and features related to the author are required to improve the performance of models. These author-based features are often used as indicators of expertise or authority, and include personal traits, activities, history, consistency (such as the number of posts on a topic), and network structure (especially propagation paths). Assessing the relevance of content contributions is always determined in relation to a particular event or topic of interest that is often expressed as a query for information retrieval. Many approaches cluster content together according to topic similarity, and determine relevance based on measures of centrality complemented by topic-dependent and textual features. Assessing the quality and relevance of tags is more challenging since these are essentially single-word or short-phrase descriptors. Tags are often considered according to their distribution over all tag co-occurrence, and their tag frequency distribution as it relates to items with similar visual features (scale-invariant feature transform) or semantic features (latent Dirichlet allocation).

User-Based Approaches

User-based approaches emphasize the differences between individual users so that ranking and assessment can interactively adapt or be personalized given the particular requirements of each user. The human-centered approach accentuates interactive interfaces where the user can define and redefine their preferences as their interests shift. On the other hand, machine-centered approaches model the individual user according to explicit and implicit knowledge that is gathered through system interactions.

The human-centered approach aims to design interactive interfaces that allow the user to specify their preferences and enable content adaptation. This method allow users to browse UGC according to subjects that are of interest, or other criteria. Methods based on topic modeling or clustering are frequently used to support browsing UGC by topic, and some approaches even model the user as a distribution of topics. However, text may be of insufficient length or topics discussed can be unorganized and noisy, such as in microblogging or commenting, and the approach may benefit from enriching text using search engines or some other source. The most robust methods for ranking and assessing UGC according to user preferences allow for content to be organized along various dimensions other than just topics.

Personalization depends on machine-centered methods to learn the preferences of a particular user based on implicit knowledge that is derived from previous actions and participation, as well as explicit information that is provided by the individual. Information about the topics of the content that the user typically engages in, the motivations behind the user’s interactions, such as social or informational usage, and other preferences are collected and used to construct a user model. This user model serves as input for computing the assessment and ranking of content, often using many of the same machine-learning methods implemented in community-based approaches. Many approaches leverage the connections between users in their assessment method, and sometimes even use knowledge about a user’s connections to access additional information for predicting preferences. However, the performance gains offered by using user connections can differ significantly based on the application domain and the usage intent of the individual.

Design-Based Approaches

It is sometimes the case that the assessment and ranking of UGC requires an application-specific approach that enables the intent of the designer. The designer-based approach primarily uses machine-centered methods to ensure that information filtering systems rank content in a manner that minimizes the redundancy of those items. This approach essentially seeks to maximize the diversity of content presented to users in order to avoid constraining the space of topic selections or perspectives. The diversity of content can be assessed with respect to various dimensions, such as authorship, topics, sentiments, and named entities.

Enhancing the diversity of UGC browsed by users is often used to present positive and negative reviews in the domain of product reviews, a variety of topic selections on multimedia platforms, and various perspectives of political views and news. Diversity benefits users with respect to applications presenting political content and news by allowing them to access and consider various viewpoints, and produce arguments on both sides of the issue. Feedback mechanisms can be implemented within an interface to inform the user of the left-right balance of articles with the intent of creating accountability and influencing behavior. It should be noted, however, that users often still engage in selective exposure when diverse items are presented in a distributed arrangement. Additionally, ordering agreeable content first generally does not increase the overall satisfaction with a system, especially in the case of challenge-averse users.

Hybrid Approaches

Hybrid approaches seek to combine methods from the various frameworks in order to develop a more robust approach for assessing and ranking UGC. The crowd-based approach is often used to identify hyperlocal content for each individual user. Hyperlocal content includes the identification of: active events using statistical event detectors that identify and group popular features in tweets; top topics using topic modeling and frequency counts; popular places using template-based and learning-based information extractors on check-ins and content; and active people using a social graph ranking function. Crowd behaviors can also be utilized to learn models of user roles or stereotypes in order to classify end users and present content according to their user group. Some approaches seek to leverage the crowd-based approach to minimize the cost of changing filter settings for an individual user, and often use techniques based on collaborative filtering. Hybrid approaches also sometimes utilize the behaviors of individual end users to maintain the intent of designer-based methods. This approach allows the designer to diversify content with respect to each user’s interests along various dimensions. The combinations of various approaches is becoming increasingly more common across application domains as the system design seeks to leverage the benefit of one approach as a means to improving another.