Find what you want, before you get there.
Table of Contents
What is Scout?
Scout is a Chrome extension that lets you search for scenes in Youtube videos by using computer vision to determine what’s in each frame.
I made this with David Zhao at Georgetown University for HoyaHacks.
How it’s made
We used Amazon’s Rekognition image recognition service to identify objects within the frames of a video.
The preprocessing was composed of three steps:
- Download the video
- Lower resolution
- Split into frames
youtube-dl library, we combined the first two steps, choosing to download a standard definition video to cut down on both download time and processing time.
In almost all cases, it is unnecessary to check every frame, since the scene generally does not change 30 times a second. We initially tried to detect scene changes, but the extra processing overhead was not worth it, and we ended up just taking one frame from every three seconds of video.
Ultimately, the largest bottleneck for speed was downloading the videos.
We determined early on that a Chrome extension would be the best fit for this sort of project, since it was an extension to YouTube.
The backend consisted of a simple Flask server, hosted on Director. All of the frame results are stored in AWS, so each video needs only be processed once for all users.