Breaking Bad: Understanding Behavior of Crowd Workers in Categorization Microtasks

Crowdsourcing systems are being widely used to overcome several challenges that require human intervention. While there is an increase in the adoption of the crowdsourcing paradigm as a solution, there are no established guidelines or tangible recommendations for task design with respect to key parameters such as task length, monetary incentive and time required for task completion. In this paper, we propose the tuning of these parameters based on our findings from extensive experiments and analysis of categorization tasks. We delve into the behavior of workers that consume categorization tasks to determine measures that can make task design more effective.

Authors: Ujwal Gadiraju, Patrick Siehndel, Besnik Fetahu, Ricardo Kawase

PDF: gadiraju_ht2015Download

Exploiting the Wisdom of the Crowds for Characterizing and Connecting Heterogeneous Resources

Ricardo Kawase presenting @HT2014 in Santiago, Chile (picture taken by Christoph Trattner)

Heterogeneous content is an inherent problem for cross-system search, recommendation and personalization. In this paper we investigate differences in topic coverage and the impact of topics in different kinds of Web services. We use entity extraction and categorization to create fingerprints that allow for meaningful comparison. As a basis taxonomy, we use the 23 main categories of Wikipedia Category Graph, which has been assembled over the years by the wisdom of the crowds. Following a proof of concept of our approach, we analyze differences in topic coverage and topic impact. The results show many differences between Web services like Twitter, Flickr and Delicious, which reflect users’ behavior and the usage of each system. The paper concludes with a user study that demonstrates the benefits of fingerprints over traditional textual methods for recommendations of heterogeneous resources.

Authors: Ricardo Kawase, Patrick Siehndel, Bernardo Pereira Nunes, Eelco Herder and Wolfgang Nejdl

PDF: kawase-ht2014

Online Prototype: http://twikime.l3s.uni-hannover.de

 

A Taxonomy of Microtasks on the Web

Nowadays, a substantial number of people are turning to crowdsourcing, in order to solve tasks that require human intervention. Despite a considerable amount of research done in the field of crowdsourcing, existing works fall short when it comes to classifying typically crowdsourced tasks. Understanding the dynamics of the tasks that are crowdsourced and the behaviour of workers, plays a vital role in efficient task-design. In this paper, we propose a two-level categorization scheme for tasks, based on an extensive study of 1000 workers on CrowdFlower. In addition, we present insights into certain aspects of crowd behavior; the task affinity of workers, effort exerted by workers to complete tasks of various types, and their satisfaction with the monetary incentives.

Authors: Ujwal , Ricardo Kawase and Stefan Dietze

PDF: gadiraju-ht2014

 

Hyperlink of Men

Ricardo Kawase presenting @LAWEB2012

Hand-made hyperlinks are increasingly outnumbered by automatically generated links, which are usually based on text similarity or some sort of  recommendation algorithm. In this paper we explore the current linking and appreciation of automatically generated links. To what extent do they prevail on the Web, in what forms do they appear, and do users think those generated links are just as good as human-created links? To answer these questions we first propose a model for extracting contextual information of a hyperlink. Second, we developed a hyperlink ranker to assigned relevance to each existing human generated link. With the outcomes of the hyperlink ranker, together with another two recommendation strategies, we performed a user study with over 100 participants. Results indicate that automated links are “good enough”, and even preferred in
some user contexts. Still, they do not provide the deeper knowledge as expressed by human authors.

Venue: LAWEB2012

Authors: Ricardo Kawase, Patrick Siehndel, Eelco Herder and Wolfgang Nejdl

PDF:  kawase-laweb2012

Best Paper Award: Beyond the Usual Suspects: Context-Aware Revisitation Support

(picture taken by Paul De Bra)

The paper ’Beyond the Usual Suspects: Context-Aware Revisitation Support’ has won the Engelbart Best Paper Award at the 22nd ACM Conference on Hypertext and Hypermedia.

According to the jury report, the paper deals with a classic, relevant topic with potentials beyond what is stated in the paper. The paper presents an elegant prototype and a careful two-level evaluation of the approaches

The Engelbart Best Paper Award is named after Hypertext Pioneer Douglas Engelbart (born 1925).

See our previous blogpost for a short summary of the paper (or download the pdf).

Beyond the Usual Suspects: Context-Aware Revisitation Support

Ricardo Kawase @HT2011

A considerable amount of our activities on the Web involves revisits to pages or sites. Reasons for revisiting include active monitoring of content, verification of information, regular use of online services, and reoccurring tasks. Browsers support for revisitation is mainly focused on frequently and recently visited pages. In this paper we present a dynamic browser toolbar that provides recommendations beyond these usual suspects, balancing diversity and relevance. The recommendation method used is a combination of ranking and propagation methods. Experimental outcomes show that this algorithm performs significantly better than the baseline method. Further experiments address the question whether it is more appropriate to recommend specific pages or rather (portal pages of) Web sites. We conducted two user studies with a dynamic toolbar that relies on our recommendation algorithm. In this context, the outcomes confirm that users appreciate and use the contextual recommendations provided by the toolbar.

Venue: HT2011

Authors: Ricardo Kawase, George Papadakis, Eelco Herder and Wolfgang Nejdl

Award:  Engelbart Best Paper Award (HT2011)

PDF: kawase-ht2011

The impact of bookmarks and annotations on refinding information

Refinding information has been interwoven with web activity since its early beginning. Even though all common web browsers were equipped with a history list and bookmarks early enough to facilitate this need, most users typically use search engines to refind information. However, both bookmarks and search based tools have significant limitations that impact their usability: the former are known to be hard to manage over the course of time, whereas the latter require the user to recall a specific combination of keywords or context. Most importantly, though, both are particularly inappropriate in cases where a piece of information is contained within an unstructured web page. In this paper, we present in-context annotation as a more efficient alternative to these methodologies. To verify this claim, we conducted a study in which we compare the performance of experienced users in all three approaches while revisiting specific pieces of information in the web after a long period of time. The outcomes suggest that in-context annotation clearly outperforms both traditional strategies.

Venue: HT2010

Authors: Ricardo Kawase, George Papadakis, Eelco Herder and Wolfgang Nejdl

PDF: kawase-ht2010

Leveraging multi-faceted tagging to improve search in folksonomy systems

In this paper we present ranking algorithms for folksonomy systems that exploit additional contextual information attached to tag assignments available. We evaluate the algorithms in the TagMe! system, a tagging front-end for Flickr, and show that our algorithms, which exploit categories, spatial information, and URIs describing the semantics of tag assignments, perform significantly better than the FolkRank that does not consider such contextual information.

Venue: HT2010

Authors: Fabian Abel, Ricardo Kawase and Daniel Krause

PDF: abel-ht2010