video relevance judgment guidelines v2

Prévia do material em texto

VIDEO RELEVANCE JUDGEMENT GUIDELINES 
Contents 
OVERVIEW ................................................................................................................................................... 1 
UNDERSTANDING THE QUERY / USER INTENT ..................................................................................... 2 
UNDERSTAND/EVALUATE THE QUERY AND THE RESULTS .................................................................. 2 
The most important things that you can do as a judge to be successful: .................................................... 2 
Understand the most likely intent for the query. What different kinds of video results is 
the user looking for? .......................................................................................................................... 2 
Understand the types of video results for the query and sort them into buckets of 
different levels of usefulness. ........................................................................................................ 2 
Some final notes: .................................................................................................................................... 3 
VIDEO RELEVANCE RATING SCALE .......................................................................................................... 3 
Specific Rules for Specific Query Segments ............................................................................................ 5 
Segment: Movie Titles, Movie Franchise Queries ............................................................................... 5 
Segment: Television Show Queries ...................................................................................................... 5 
Segment: Musicians and Band Name Queries .................................................................................... 5 
Segment: Song Title Queries ................................................................................................................. 6 
Segment: Sports Queries ....................................................................................................................... 6 
LANGUAGE JUDGEMENT ........................................................................................................................... 7 
 
 
22.2.2021: See new update here 
OVERVIEW 
You will be shown video search queries, and video result urls. Your task is to label how satisfied users would 
be with those video results for the query. To provide correct answers, you need to deeply understand user 
intent, as well as watch some portion of the video. Your judgments will be used to understand how well 
our search engine is meeting user needs. 
 
You cannot make accurate judgements solely based on the title of the page or video, as there are 
many cases where the video is not related to the page content. Please ensure that you do proper due 
diligence when labeling or you may be banned from the hitapp and/or UHRS. 
For each query/result pair, you will have to answer four questions: 
1. Where is the video located on the page? 
o Is the video at the top, middle, or bottom of the page? 
2. How relevant is the video to the user’s query? 
o Is it Highly Relevant, Relevant, Related, or Not Related? 
3. Does the language of the video degrade the usefulness of the results? 
o Does the language of the video match what the user would expect? Or is it in a 
secondary (less useful language), or is it in a totally unknown language for the user? 
4. What is the quality of the video experience? 
o Is the video overly pixelated? Is camera footage shaky in way that is unexpected / 
unintended? Are there issues with audio quality? 
 
 
UNDERSTANDING THE QUERY / USER INTENT 
For understanding user intent, assume that the user is searching for videos – i.e. the query was typed 
in YouTube or Bing Video /Google Videos. 
For e.g. for query “python” on Pinterest, most users are looking for images of snakes. Whereas, for 
“python” on Youtube, most users are looking for the programming language (use Google Trends in the 
query locale, do your research on Bing and Google video/ Youtube to understand user intent). 
Update: A brief description about the user intent will be given next to 
the query. You can use it to understand what kind of results are useful 
UNDERSTAND/EVALUATE THE QUERY AND THE RESULTS 
It’s critical to understand query intent before judging relevance. Some notes below: 
• Queries may contain acronyms. Understand them. 
o Example: For query “wow” the result “world of warcraft” might seem Not Related 
to query intent. However, on searching Google/Bing/YouTube it’s clear that many people 
issue this query to see World of Warcraft videos. 
• Queries may be in a foreign language (in rare cases). Should you encounter a foreign 
language query we expect you to try and provide a well-reasoned judgement. You are not being 
asked to guess the relevance label (please don’t guess) but are being asked to apply some 
thought before giving up and labeling as Cannot Judge. 
o Important Note: Most commonly you will see foreign queries of the pattern 
proper noun of a foreign language entity like “Y Tu Mamá También”, “La Vie en rose”, or 
“Akihabara”. For these, we expect you to properly label the results. 
The two most important things that you can do to be a successful judge: 
- Understand the most likely intent for the query. What different kinds of video results is the user 
looking for? 
- Understand the types of video results for the query and sort them into buckets of different 
levels of usefulness. 
o Example: For query “Never gonna give you up by Rick Astley” 
o the best possible result will be the song’s official music video– this can be “Highly 
Relevant”. 
o The next most useful results are videos of this song that aren’t the official music video/ 
or videos containing a portion of the song – these can be “Relevant”. 
o Videos that discuss the song, without having the song audio, are the next most likely 
intent – these can be “Related” as they are not about the main intents for the query, 
but might still be of interest to a user. 
file:///C:/Users/shramak/Documents/New%20Doc%202019-07-11%2007.49.01
o Nearly all other results for this query will be Not Related, as they will either have 
nothing to do with the query. 
As illustrated in above example, you should conceptualize the kinds of results which exist for a query, weigh 
them in terms of overall usefulness, and then fit them into the rating scale provided. 
Final notes: 
• Some queries will have many Highly Relevant results, and some queries won’t. 
• For song name queries, there will probably be one or two “best in class” / 
HighlyRelevant videos. 
• Some queries may be broad/vague/ ambiguous. Here’s a checklist for them: 
• Broad queries: (e.g. “China”, “tiger”, “flamingo”, “yellowstone national park” -→ 
such queries need very comprehensive results to be treated highlyrelevant. Use 
highlyrelevant for broad queries only if results are sufficiently 
comprehensive. 
• Ambiguous queries: “black panther”, “sherlock holmes”, “Apple” → for such 
queries, understand the major intent and rate results accordingly. Videos perfectly 
satisfying minor intent (e.g. Apple as a fruit for “apple”) can be relevant at best. 
Videos perfectly satisfying Rare intent (e.g Apple as name of a person for “apple”) 
can be related at best. 
• Vague queries: (e.g. “its”, “cal”, “am”, “church door”) → for vague queries, you 
are suggested to avoid extreme ratings (highlyrelevant/notrelated). 
 
VIDEO RELEVANCE RATING SCALE 
Below is our guideline for the meaning of the Relevance Ratings. 
Important Note: Below guidance applies for most types of queries, but there are some query 
segments with more specific rules. The guidelines for specific segments will immediately follow the 
rating scale definition. 
RelevanceRating Explanation 
Highly Relevant • Video answers query 100% 
• Video is categorically among best video results for query in terms of meeting 
user intent. 
• There are no significantly better videos for answering the user’s intent. 
• Does not offer anything additional that would broaden or narrow the query or 
lessen user satisfaction. 
• Video is a “best in class” answer to the query. 
Relevant • Video matches the intended subject however it contains additional less-useful 
content detracting from user intent. 
• Part of video (not entire video), matches intended subject. 
o Query is for opening scene of a movie and result is full movie. 
• Video is partial/incomplete version of intended subject. 
o Query is for a full movie and video is showing part 2 of 3 of movie. 
• Video meets/ fully satisfies a minor interpretation of query (such results can at 
best be labeled relevant) 
o Query is "Apple” and the video is discussing the fruit. 
Related • Video fully satisfies a rare interpretation of query (such results can at best, be 
labeled related) 
o e.g., Query is “Amazon” and result is about Amazon warrior women. 
• Video doesn’t match query’s intent but contains related content that may be of 
interest to user. 
o e.g., Query is looking for "how to change oil and oil filter" and result is 
"how to change oil pan gasket." 
• Query asks for specific content from a specific source and result is correct 
content but from wrong source. 
o e.g., Query is "cartoons on YouTube" and the result is "cartoons" 
on dailymotion.com. 
• Correct subject, but some qualifiers in query not matched (such as date, video 
type etc.) 
o e.g., Query asks for a concert at a specific location/date and result 
matches the location, but day is wrong by 1-2 days. 
• Satire or parody of intended subject when such parody is not explicitly 
requested. 
o e.g., Query asks for a celebrity and the result is meme of that celebrity. 
• Video is an unexpected slideshow with audio of the intended subject. 
o e.g., Query is "winter”, and the result is a slide show of winter scenes 
(do note that slideshows may be useful for education and other topics and 
useful slideshows shouldn’t be downgraded) 
• Video is a static image with audio of intended subject. 
o e.g., Query is "cat”, and the result is a static image of a cat with audio 
of a cat meowing 
Not Related • Video is unrelated to query intent. 
• Result is a spam video. 
o e.g., the video says it is a movie, but rather than showing the correct 
content the uploader tries to take you to another website to view the 
content. 
Detrimental This label indicates illegal or adult video content. A video should be rated Detrimental if any of 
following applies: 
- X-rated content: Full frontal nudity, genitals, women’s nipples, sexual intercourse, 
pornography. 
- Real footage of extreme graphic violence unsuitable for broadcast news. (e.g., a 
beheading) 
- Child Porn. 
NOTE: IF a query has adult intent the video should be rated on relevance. Detrimental should 
only be used when the query does not have adult intent, but the video contains adult content. 
Video Didn’t Play • Video does not load, either on page or by clicking URL 
• Page contains image(s) thumbnails of video players, but no actual video player. 
• Page does not contain a video. 
• Page contains an audio file without video. 
It is possible that videos of Live Events don’t play after the event is over. In such a case, use the 
other indicators like video title, source (YouTube # of subscribers, source name etc) to decide 
whether the live video was accessible at the time of search. If yes, and the video was relevant to 
the live event/topic then label the video as “Highly Relevant” instead of “Video Didn’t play”. 
Cannot Judge Video is in a foreign language and you cannot provide a well-reasoned judgement. 
The query is in a language you don’t understand, and the translation of the query doesn’t make 
sense. 
Do not mark English queries as foreign. 
 
Specific Rules for Specific Query Segments 
Below are the guidelines for several specific query segments. These guidelines supersede the general rating 
scale provided above but should mostly align with the general rating scale. 
Segment: Movie Titles, Movie Franchise Queries 
(e.g. “avengers”, “avatar”, “terminator judgment day”, “Joker movie”, “gladiator”) 
Type of Video Highly Relevant Relevant Related Not Related 
Complete Movie X 
 
Official Trailer / Commercial X 
 
Portion of complete movie 
(e.g., Part 1/3) 
 
X 
 
Movie review 
 
X 
 
Cast interview related to the movie 
 
X 
 
Behind the scenes, making of, outtakes 
from the movie’s production 
 
X 
 
Fan made content (including fan made 
movie trailers) 
 
X 
 
Wrong movie 
 
X 
 
Segment: Television Show Queries 
 (e.g. “Game of thrones”, “breaking bad”, “Indian Idol”) 
Type of Video Highly 
Relevant 
Relevant Related Not Related 
Any Complete Episode X 
 
Clip of an entire segment from the TV 
Show 
X 
 
Official trailer, commercial, or preview of 
upcoming episode 
 
X 
 
Review of TV show 
 
X 
 
Cast interview related to the production 
of the TV show 
 
X 
 
Any fan made content related to 
the show 
 
X 
 
Wrong show 
 
X 
 
Segment: Musicians and Band Name Queries 
 (e.g. “taylor swift”, “Michael jackson”, “the beatles”) 
Type of Video Highly 
Relevant 
Relevant Related Not Related 
Official music video 
where musician/band is the 
primary focus 
X 
 
Official music video where artist is 
featured, but is not the 
primary song writer 
X 
 
Live performance (when not an official 
music video) 
 
X 
 
Audio of musician with static image, 
slide show, or lyrics from an unofficial 
source 
 
X 
 
Audio of a whole album or multiple 
songs 
 
X 
 
Video only contains only a portion of a 
song by the musician/band 
 
X 
 
Remix of correct musician by a different 
artist 
 
X 
 
Segment: Song Title Queries 
(e.g. “taylor swift lover”, “hello”, “give in to me”) 
Type of Video Highly 
Relevant 
Relevant Related Not Related 
Any video of the song from official 
source 
X 
 
Live performance of the song 
 
X 
 
Audio of the song with static image, 
slideshow, or lyrics 
 
X 
 
Remix of the song by original artist 
 
X 
 
Cover of the song by an amateur 
musician 
 
X 
 
Partial song audio 
 
X 
 
Video discussing the song 
 
X 
 
Correct musician, wrong song 
 
X 
 
Segment: Sports Queries 
This should be interpreted as referring to queries for: sport names, sports team names, sports matches, 
sports leagues, and organizations. For all other sports related queries, please follow the guidelines under 
the Relevance Rating Scale. 
(e.g. “barcelona”, “liverpool”, “real madrid vs Atletico madrid”, “nba”, “nfl”, “premier league”) 
Type of Video Highly Relevant Relevant Related Not Related 
Complete game, professional level 
highlights, or professional game recaps, 
and analysis by top level experts 
X 
 
Postgame interviews 
with players and press conferences 
 
X 
 
Promotional videos for the team or for a 
specific match 
 
X 
 
Commentary from fans and other fan 
made content 
 
X 
 
Results from a video game 
 
X 
 
LANGUAGE JUDGEMENT 
Please listen to the audio of the video for a few seconds to answer this question correctly. 
The goal is to ascertain whether the video language (audio/title and captions) matches the language 
expected by user. You may need to listen to the audio to answer this correctly. A user from UK issuing 
queries like “iphone review”, “biden inauguration”, “things to do in Amsterdam” would expect English 
language results. Hindi results for such queries would not be useful for this scenario (Hindi isn’t widelyspoken in the UK). 
List of labels and their definitions: 
• Does not degrade: use this when query language matches the result 
• The query is in English and the video has English audio/text (listen to the audio 
to label correctly) 
• The query is for a foreign language, but the video has appropriate subtitles that 
make the video understood by someone who speaks the language of the hitapp. 
• e.g. Query is "Amelie film" (a French film) and the video has audio in French 
but subtitles in English 
• Query is for foreign language song, singer, band, tv show, news site, 
Youtube channel. 
o For “despacito”, user expects the results to be the Spanish language song. 
o Query for Korean TV show is expected to show results with Korean audio 
and preferably English subtitles. 
o Query for German site “prosieben” expects videos in german, query for 
“rebeka wing” (german youtuber) expects German results. 
o For query “extradicion del el mencho” in USA, results are expected in 
Spanish. 
• There is no audio/audio is not needed to understand the video. 
o e.g. Query is "Primitive Technology" (a channel that uploads videos 
without dialogue /subtitles, but neither of these are needed to understand the 
content) and result is from this channel. 
o e.g. Query is "Mimes" and result is a mime routine. 
o e.g. Query is for sports footage and result has correct content but foreign 
language commentary. 
• Partially degrades: Use this when result doesn’t match the query language but is in a 
secondary language of the region: 
o For e.g. if the query was in Spanish in USA, and the result was in English. 
o If the query was in Spanish in Spain, (or Italian in Italy) and the result was in English. 
English can be considered as secondary language in most regions. 
• Severely degrades: Use this label when the wrong language makes the results useless, 
because that language is not commonly spoken in the region. E.g. for an English query in en-
US “harden stepback review” if the results have audio in Indonesian (which is not spoken in 
USA).