Towards Quality Assessment of Crowdworker Output Based on Behavioral Data
Journal
Proc. of The Third Human-in-the-loop Methods and Human-Machine Collaboration in BigData, IEEE Big Data
Number of pages
3
Reviewed
Reviewed
Presentation type
Oral
Date
December 2019
Research fields
Crowdsourcing, Activity Recognition, Sensing
Key
Yuasa2019 (id=1706)
Abstract
In this paper, we show preliminary results on the quality assessment of crowdworker output based on the movements of the mouse and the eyes while the task is performed. We assume that the mouse and the eyes stop longer if the quality is lower due to the lack of knowledge, or confidence, etc. Because the mouse- and eye-stopping duration follows log-normal distribution, we estimate its parameters (mean and standard deviation) to evaluate the quality. Results of preliminary experiments with 10 participants show that the parameters of correct outputs are different from those of incorrect ones. As compared to the task duration, which is often used as a feature for assessment, we have found that the mouse- and the eye-stopping duration is advantageous and complementary for the assessment.
Files
…
Export this publication
Tab-separated. Paste into the sheet starting at column A. Required fields that cannot be inferred are left blank.
Format definition
Edits apply immediately and stay in effect until you pick a style again or reset. First line is the default; add lines like "book: …" to override per type.
How to write the definition
Insert a field with braces like {authors}. A part enclosed in [ ... ] is omitted entirely when any field inside it is empty. Available fields:
One macro per line as \name{field}{field}…. A field name in braces is replaced by its (LaTeX-escaped) value; other text is kept. Rename macros or add lines freely — they appear in the macro selector. A line like master_thesis: thesis fixes the macro for a type (automatic choice: journals use paperdoi with a DOI, paperno with a number, else paper; conferences conf with pages, else confpaper; books book). {pages} comes as 119--124; {journal} falls back to the book title. Fields: