In many cases, they simply want the exact answer to their questions, asked in natural language. They are online platforms where users share their knowledge and expertise on a variety of subjects.
These sites go beyond traditional keyword-based querying and retrieve information in more precise form than given by a list of documents. This fact has changed the way people search for information on the web. For instance, the abundance of information to which developers are exposed via social media is changing the way they collaborate, communicate, and learn Vasilescu and Serebrenik Other authors have also investigated the interaction of developers with SO, and reported how this exchange of questions and answers is providing a valuable knowledge base that can be leveraged during software development Treude, Barzilay and Storey These changes have produced a marked shift initially from mere websites born to provide useful answers to the question, towards large collaborative production and social computing web platforms aimed at crowdsourcing knowledge by allowing users to ask and answer questions.
The end product of such community-driven knowledge creation process is of enduring value to a broad audience; it is a large repository of valuable knowledge that helps users to solve their problems effectively Anderson et al. Library reference services have a long tradition of evaluation to establish the degree to which a service is meeting user needs. By a quality answer, we mean the one that satisfies the asker Liu et al. Answers, launched in Some sites such as Yahoo!
The fact that both Atwood and Spolsky were popular bloggers contributed to its success in the early stages of the project as they brought their two communities of readers to their new site and generated the critical mass that made it work Atwood ; Spolsky Soon, the phenomenon of SO caught the attention of investors and media.
The site employs gamification to encourage its participants to contribute Deterding Participation is rewarded by means of an elaborate reputation system that is set in motion through a rich set of actions. The basic mode of viewing content is from the question page, which lists a single question along with all its answers and their respective votes.
The vote score on an answer —the difference between the upvotes and downvotes it receives— determines the relative ordering in which it is displayed on the question page. When users vote, askers and answerers gain reputation for asking good questions and providing helpful answers, and they also obtain badges that give them more privileges on the website.
In addition, at any point, an asker can select one of the posted answers as the accepted answer, suggesting that this is the most useful response. This also makes the asker and the answerer earn reputation. The reputation score can be seen as a measure of expertise and trust, which signifies how much the community trusts a user Tian et al. This community is increasing both in size and in the amount of content it generates.
The number of questions added each month has been steadily growing since the inception of SO Ponzanelli et al. The content is heavily curated by the community; for example, duplicate questions are quickly flagged as such and merged with existing questions, and posts considered to be unhelpful unrelated answers, commentary on other answers, etc.
As a result of this self-regulation, and despite its size, content on SO tends to be of very high quality Anderson et al. The primary idea of this study is to understand the relation between a question and its accepted answer in order to predict potential accepted answers from a set of candidate answers for new questions. This problem has two main components: crowdsourcing and machine learning. The term crowdsourcing is the combination of the words crowd and outsourcing , and is the process of getting work from a large group of people, especially from online communities, rather than from employees or suppliers.
This model of contribution has been applied to a wide range of disciplines, from bioinformatics Khare et al. The former is focused on the Web as the natural environment for crowdsourcing.
The latter emphasizes the power of humans to undertake tasks that computers cannot yet do effectively. SO relies on the crowd, and some authors underlined the soundness of this user-generated content model to provide quality solutions. Therefore, a large number of answers on SO are presumably effective solutions, as they came from qualified programmers that incorporate good work practices.
Parnin, Treude and Grammel showed that companies like Google, Android, or Oracle also acknowledge the quality of contents produced on SO. The authors collected usage data using Google Code Search currently shut down , and analyzed the coverage, quality, and dynamics of the SO documentation for these APIs.
They found that the crowd is capable of generating a rich source of content with code examples and discussion that is more actively viewed and used than traditional API documentations. The second pillar of this work is machine learning , a subfield of artificial intelligence that studies how to create algorithms that can learn and improve with experience. These algorithms learn from input observations, build a model that fits the observed data, and apply the model to new observations in order to make predictions on them.
Machine learning is a transverse area used in a large number of disciplines, and with multiple applications, from computer vision to speech and handwriting recognition. A recurrent problem solved with machine learning is classification. This is the problem of identifying the category of a new observation, and belongs to the type of supervised methods, that is, the task of building a model from categorized data.
The question seems trivial but even in many branches of pure mathematics, where specialists deal with objective and universal knowledge, it can be surprisingly hard to recognize when a question has, in fact, been answered Wilf As already mentioned above, our approach is to extract features from questions, answers, and users, and apply classification to learn the relation between a question and its accepted answer.
Many authors have applied machine learning techniques to provide the correct answer to a question by pursuing different objectives. For instance, some of them focused on directly identifying the best answer.
Wang et al. Answers as relational data Wang et al. They assumed that answers are connected to their questions with various types of links that can be positive indicating high-quality answers , negative indicating incorrect answers or user-generated spam. They proposed an analogical reasoning-based approach which measures the analogy between the new question-answer linkages and those of some previous relevant knowledge that contains only positive links.
The answer that had the most analogous link to the supporting set was assumed to be the best answer. Shah and Pomerantz instead, evaluated and predicted the quality of answers, also on Yahoo! Answers Shah and Pomerantz They extracted features from questions, answers, and the users that posted them, and trained different classifiers that were able to measure the quality of the answers. The answer with the highest quality was considered the best one. This is actually a key finding in our experiment, as we will see in the results section 5.
Another approach is to redirect the question to the best source of information. For example, Singh and Shadbolt matched questions on SO to the corresponding Wikipedia articles, so that the users could find out the answer by themselves Singh and Shadbolt For this, they applied natural language processing tools to extract the keywords of the question, and then they matched them to the keywords of the pertinent Wikipedia article. Tian et al. They proposed an approach to predict the best answerer for a new question on SO.
They are systems that receive a question in natural language and return small snippets of text that contain an answer to the question Voorhees and Tice Related to these systems, Rodrigo et al.
They studied two methods, F-score and ROC, and compared both evaluation approaches. Less relevant to our work, machine learning can be applied to classify questions instead of answers. Ponzanelli et al. They investigated how to model and predict the quality of a question by considering both the contents of a post from simple textual features to more complex readability metrics and community-related aspects for example, the popularity of a user in the community as features.
Other authors analyzed the problem of question classification by topic. Miao et al. Answers, and reported the importance of incorporating user information to improve both methods Miao et al. Blooma et al. Li and Roth developed a semantic hierarchical classifier guided by an ontology of answer types Li and Roth As stated before, we formulated the idea of identifying valuable knowledge on SO as a classification problem in machine learning.
Let us have a question, q i , that has k answers, a i 1 , a i 2 , …, a ik , and none of them has been marked as accepted yet. Information unit or quan : a quan is a question-answer pair. Each answer on SO has a minimum level of accuracy with respect to its question; in any other case, it would have been removed by the community.
Therefore, every quan provides always valid information. Knowledge unit or kuan : a kuan is a particular quan formed by a question and its accepted answer. Hence, a kuan is a source of valuable knowledge for other users that could face the same problem in the future. Formally, let Q be the set of questions and n its cardinality, i. Let A be the set of answers and m its cardinality, i. And let ans be the function that returns the answers for a question and k i the number of answers for the i th question q 1 has k 1 answers, q 2 has k 2 answers, etc.
Let us group the total m answers in n subsets of answers one subset for each of the n questions. Each question may have a different number of answers with respect to the other questions:. Let QA be the set of quans i. Or equivalently, if a iji is the accepted answer for the question q i , then KA can be redefined as follows:. Let us note that the KA is a subset of QA, i. The following example illustrates the previous concepts.
Python is a programming language that allowed us to work quickly and integrate systems more effectively. Pandas is an open source, BSD-licensed library that provides high-performance, easy-to-use data structures and data analysis tools for Python.
Scikit Learn is a collection of simple and efficient tools for data mining, data analysis, and machine learning.
Our experiment was performed in several steps:. With the Ubuntu command line, we split the posts into questions and answers, and then used the Element Tree XML API to parse and extract the data from questions, answers, and users. Table 1 summarizes the information of the total data dump. In certain topic categories, however, the number of competing answers and the history of the answerer were more likely to predict answer quality.
These findings are consistent with Agichtein et al. In the same vein, to predict questioners' satisfaction with the answers presented in Yahoo! Answers , Liu et al. Among them, askers' ratings satisfaction of an answer in response to a previous question was the most salient feature to predict their satisfaction with a new answer.
On the other hand, the reputation of the answerer was much less important, suggesting that the authority of the answerer might only be important for some, but not all, information needs. Interestingly, some researchers and operators of social question and answer sites speculate that one of the reasons such sites are not regarded as reliable sources of high-quality information is the prevalence of conversational questions e. Harper et al.
Since these two question types are asked for different purposes, people may apply different sets of criteria when evaluating answers provided for each type of question. Therefore, this study categorizes question types into conversational and informational questions and examines their influence on credibility judgments. The purpose of this study is to investigate users' credibility judgments in a social questions and answers site. Recognizing credibility judgment as an ongoing process instead of a discrete activity, the study describes a sample of users' motivations to use a social questions and answers site, credibility judgments of answers, and post-search verification behaviour.
In addition, it investigates the relationships between question type and credibility judgment. The specific research questions the study addresses are as follows:. This study is part of a bigger project whose aim is to understand the information seeking and providing behaviour of questioners and answerers in a social questions and answers site.
Since the project was necessarily descriptive and exploratory in nature, interviews were conducted to investigate both questioners and answerers' experiences of the site. This study reports only on the interviews with questioners regarding their credibility judgments. E-mail, chat, and telephone interviews were held with thirty-six questioners of Yahoo! Answers and the interview transcripts were analysed using the constant-comparison method Lincoln and Guba This study selected Yahoo!
Answers as a research setting because of its dominant status among social question and answer sites. As of March , Yahoo!
Answers was the most visited question and answer site in the U. It has attracted twenty-five million users with million answers in the U. The astonishing scale of data and diversity of topics have made Yahoo!
Answers a popular setting for recent research on such sites despite its short history Agichtein et al. The process of asking and obtaining answers to a question is quite simple: a user questioner posts a question under a relevant category from twenty-five top-level topic categories and it becomes an open question. Once the question is posted, any user answerers can post answers to it.
Among all answers posted, the questioner can select the best answer or, alternatively, allow the community to vote for the best answer. When a best answer is chosen, either by the questioner or by the vote, the question becomes a resolved question and remains in the Discover section for browsing and searching. To encourage user participation and reward high quality answers, Yahoo! Answers implements a point system and, based on the points, categorizes members into different levels Yahoo!
Answers When questioners choose the best answer for themselves or vote for the best answer for others they also get points. The earned points allow everyone to recognize how active and helpful a user has been in the site.
Starting November and ending April , a solicitation e-mail was sent to Yahoo! Answers users individually for the project. Each week during the twenty-five week period, thirty users of Yahoo! Answers fifteen questioners and fifteen answerers were selected from one of twenty-five top-level topic categories in the Discover section using three criteria.
The first criterion was to select those who asked a question most recently in each topic category. Because of the heavy traffic, the selected participants had usually asked questions within the last day.
The second criterion was to select those whose Yahoo! The third criterion excluded those who explicitly stated in profiles, questions or answers that they were under In the solicitation e-mail, the participants were given four options of interviewing: telephone, e-mail, chat, and face-to-face for nearby participants only.
Given that Yahoo! Answers users are geographically dispersed and they vary considerably in terms of Internet proficiencies and writing skills, it was an appropriate choice to provide as many interview methods as possible. By allowing people to select the interview method they felt most comfortable with, the weaknesses associated with each method were expected to be reduced to a minimum. For the bigger project, two types of semi-structured interview questionnaires were prepared: one was for questioners and the other for answerers.
The participants were asked to select the type of questionnaire s they would like to complete based on their questioning and answering experience in the site.
As aforementioned, this study reports only on the data derived from the interviews with questioners. The semi-structured interviews for questioners included seven open-ended questions about:. A questioner's familiarity with the topic of the question, urgency of the information need, experience with the site, and demographic information age, sex, occupation were solicited at the end of the interview.
The Critical Incident Technique was used to help the participants focus on their most recent questions and evaluation processes. This is a popular interview technique used to identify specific incidents which participants experienced personally rather than eliciting their generalized opinions on a critical issue. Since the purpose of this study is to describe how the participants assess information, the method was useful in drawing out realistic details without observing them directly.
From the e-mails sent, thirty-six interviews resulted with questioners and forty-four interviews with answerers. A possible reason for the low participation rate is the use of Yahoo! Creating a Yahoo! People create the e-mail accounts as a means to be a member of the site, but not all of them are actually using the accounts.
This may have caused a high undeliverable rate. Among thirty-six interviews with questioners, there were 17 e-mail interviews, 10 through Internet chat Chatmaker and Yahoo!
Messenger , and 9 by telephone. Each chat or telephone interview took approximately 40 minutes to an hour and a half. During the chat and telephone interviews, some questioners pulled up their questions in the site and walked through their evaluation processes with associated answers.
The chat session transcripts were automatically recorded and the telephone interviews were audio-taped and transcribed verbatim. Five follow-up interviews were conducted with the e-mail interviewees for clarification and missing data. The data was analysed using the constant-comparison method of content analysis Lincoln and Guba The researcher read through the transcripts and classified individual statements into categories with a simultaneous comparison of other categories. Throughout the process, themes formed inductively, guided by the interview questions and patterns emerged to provide various perspectives on central issues.
To see the influence of question type on credibility judgments, the participants' questions were categorized into two groups as in Harper et al. Conversational questions are intended to spark a discussion and do a poll on a particular issue, and therefore are not expected to have one correct answer. On the other hand, informational questions are intended to call for facts, procedures, recommendations for products and services, or sources of information.
This type of question is expected to have one correct answer or appropriate recommendation. When it comes to credibility criteria, the researcher and a library science graduate student coded the mentions of criteria independently.
Through the initial coding, the two coders developed a codebook iteratively by reaching a consensus on the analysis of shared transcripts. The codebook included a list of criteria the participants used along with their definitions and examples. With the finalized codebook, the coders coded the entire transcripts again independently.
After this round of coding, inter-coder reliability was calculated using Cohen's kappa. All disagreements were resolved through discussion to reach consensus. To verify the researcher's interpretations and conclusions, a member check of the results occurred with four participants, selected based on the number of times they were quoted in the study.
A member check is regarded as the most critical method in establishing validity in a qualitative study Lincoln and Guba A preliminary draft of the results was sent to the four participants and they confirmed and agreed with all the findings presented. The participants ranged widely in age from 18 to 67 mean: 37, SD: It was assumed that the participants would be from the United States, but at least two of them were from other countries, as revealed by the questions they asked.
The participants greatly varied in their occupations including student, banker, bus driver, computer programmer, graphic designer, aerospace engineer, hair stylist, homemaker, unemployed, and more. Regarding the experience with Yahoo! Considering the participants' long experience with the site and the frequency of use, a majority of the participants were experts accustomed to using various features of the site.
The type of question the participants asked was closely tied to their motivation for using Yahoo! For those questioners who asked conversational questions, Yahoo! Answers was a natural choice because they ' could get answers from millions of real people ' Participant 11 P11 and answerers were believed to have ' an ability to answer as openly as possible ' P While three out of the thirteen questioners did a pre-search for background information, most did not feel a need to search information prior to a discussion.
Put differently, when the questioners wanted to initiate a discussion or do a poll on a specific issue, they usually went to Yahoo! Answers directly without consulting other sources. In this case, the credibility of the site was not an important consideration. Instead, the questioners sought tools that allowed them to participate in conversation, as one participant prioritized the social interaction taking place in the site over its trustworthiness:.
The other participants asked informational questions to look for solutions to problems at hand, to expand knowledge on a topic, or to find a fact.
As opposed to those who asked conversational questions, most of this group searched information before coming to Yahoo! Answers, mainly using the Web or interpersonal sources. For example, a college student working part-time in technical support services at his college was facing difficulty in accessing his local network from one computer.
After performing many Internet searches, calling the hardware manufacturer, and consulting a few of his colleagues at work, he went to Yahoo! Answers and finally found what the problem was. While this example demonstrates the use of Yahoo!
Answers as the last resort when searches fail with other sources, several questioners used the site to confirm the information they gathered from other sources, as P13 wanted to check if the information that his mechanic gave to him was real and could be trusted.
One participant illustrated the type of questions that can be better addressed by a social questions and answers site than a general Web search engine. When he found a spider in his bedroom, he took a picture of it and posted a link to the picture in the site to identify what kind of spider it was:.
In a nutshell, the questioners came to the site because of its abilities to deal with difficult questions calling for discussion, personal advice, suggestion, or other information that cannot be easily answered by a traditional Web search engine. Moreover, previous positive experience with Yahoo! Answers raised its perceived credibility and motivated the questioners to use it as an information source. Seven questioners made comments such as:.
It is evident that the perceived credibility of the site influences one's decision to use it. It is not always the first and foremost factor, however, because questions may not require credible information.
For example, the ability to compare a wide spectrum of others' opinions can come first before the credibility issue, depending on the goal of asking a question.
The next section examines the questioners' opinions on the credibility of the site in more detail. Answers' users as information providers:. They also reported that biased and hateful users were abundant particularly in politics, religion, and global warming categories where opinion was particularly divided.
Accordingly, the nature of the subject category a questioner has been active in strongly influences his evaluation of the entire site; for example, a favourable evaluation by a participant who asked a hunting question may have resulted from experience with the less controversial topic shared among like-minded people who have the same hobby in the Hunting category.
If a questioner can find a serious person who is willing to take time to research and explain his answer along with a source, the information is very likely to be credible. Users were incentivised to create unverifiable info as fast as possible. You earned two utterly meaningless points for every answer you posted, and you earned ten utterly meaningless points if your answer was voted to be the best. As you moved up the ranking system, you could attain the black belt of level seven — able to post hundreds of questions and thousands of answers per day.
Except perhaps the well-loved benign troll Ken M, but he was active on a dozen different platforms. The highest-scoring user of all time was a grandma. I asked Youtube creator JT Sexkik for his opinion. He discovered Y! A via the forum SomethingAwful when he stumbled upon a post compiling a bunch of Yahoo Answers screenshots, chief among which was the now-infamous how is babby formed.
The section on health and pregnancy had so many excellent misspellings that it gave Sexkik the idea to rejig the babby meme: how is prangent formed. The video was a smash hit, skyrocketing into the tens of millions of views, and producing possibly the most well-known piece of Yahoo Answers content of all time.
This is, I suspect, how a lot of people know and understand Yahoo Answers: a near-limitless repository of bizarre questions and even more bizarre answers. A meme factory. A way to harmlessly waste some time.
Twitter users have been sharing their favourite oft-traded screenshots from Y! You were either there because you had absolutely nowhere else to go — or you were there to take a voyage among the perpetually confused. The actual service of Y! A was the opportunity to rubberneck at the complete derangement of everyone else involved. It is not trusted. If you believe you can do anything. Nearly every service that Yahoo Answers nominally provided has been filled by some other thing.
WebMD has answers to your medical woes the answer is always cancer. The premium content rises to the top, the cringe, for the most part, vanishes.
0コメント