Predicting Instructables Contest Winners With App Inventor and Web Scraping

I've broken my divination crystal sphere, so I was looking for more reliable foretelling tools one could easily build. When it comes to web scraping, it might indeed look like a sort of magic at first glance. Because it is. However you could learn some. Coding skills are not mandatory, since we are going to code with blocks in App Inventor IDE.

Web scraping means extracting data from webpage source code for further analysis. It is completely legal as long as you process publicly available data from open webpages.

Obviously we can't exactly predict any contest results. What we are going to do is to build a mobile app to analyze which contest entries are more popular than others and therefore are likely to become contest finalists. Sort of unofficial contest rating just for fun and programming teaching, so don't take it seriously. I'll use the following formula:

🤖Instructable impact = 👁×1 + ❤️×100 + ⭐×1000

That is to say, one 👁view gives you one point, one ❤️like gives 100 points, and if you are lucky enough to be ⭐featured, you'll get 1000 points to Gryffindor.

It's just my biased vision. Please share your opinion about these metrics. Is it feasible to estimate a publication impact this way? Should the number of 🗨comments or 👤subscribers be also taken into account?

Does it really work?

I've tested this approach on some finished contests and it usually was able to "predict" post factum about 60-70% of finalists, and nearly all first and second prize winners were among top 20 positions of the app's rating.

What's in it for you?

1. You may use this approach to estimate which topics are more popular and find out what people like. It finally might help you create a more impressive contest entry.
2. If you are a Computer Science educator, you may use my project to teach your students web scraping essentials with block coding. In this case you might also like other App Inventor projects I shared earlier: (1), (2).

I have attached apk file just in case you want to give it a try. However I recommend that you download the source code (.aia file) and import it to App Inventor to discover it yourself.

1. An Android device with Internet access and QR-code scanner.
2. A computer or laptop with a web browser (you could use the same device for programming and testing, but I prefer a desktop browser for this purpose).

Register on http://ai2.appinventor.mit.edu or sign in with your Google account and create new project.

Drag a VerticalArrangement onto your app's main screen and make it fill all available space. It will align all the controls vertically. Put a Label and a TextBox for URL input on the VerticalArrangement. Put a load Button and ListView for data displaying just below. Another Label will be used as a status bar.

If you wish, you could customize colors of the widgets. When you're ready, switch to Blocks mode and proceed to the further steps.

Please use this map as an overview of the following steps. The source code is available here.

Initialize the requests variable. It will count the number of unanswered GET requests.

On Button click load URL from the TextBox and perform GET request of a contest webpage.

This subroutine performs the most of web scraping "magic". It extracts the text between precedingTags and followingTags. For this purpose split function is used two times. It splits a text into a list of strings and we use only parts we need to isolate our data from surrounding text.

This code will be used to process contest entries.

In the same manner we extract the contest entries URLs and pack them into URLList. Afterwards we perform GET request for each item of the list.

Now we process each Instructable webpage to collect the number of 👁views, ❤️likes and find out whether it was ⭐featured.

First we build ListView items to be displayed. Note that \n means new line character which is used for formatting. Then the list is being sorted according to our rating. Please note that App Inventor sort function allows you to implement your own comparison function. This powerful feature is used here.

Now you may download your final app to your smartphone and test it. You might have noticed that I had not implemented failed request handling and a lot of other checks. This is an educational project, not an ultimate app, so I tried to keep it as simple as possible. Please feel free to improve it or use the described ideas in your projects.