This week, the last of the term, served as a wrap up. I made some modifications to how the website presented data, which I will discuss in more detail later in the post. I feel like I have learned a lot over the course of the past three months, and I would like to offer a brief reflection on each component of this IP.
Although I expected it to be the most challenging part of this project, I think that Django was ultimately the portion with the fewest hiccups. Although it took a significant amount of time for me to get used to the framework and learn its “ins and outs,” once I started actually coding my blog, everything went quite smoothly. I was lucky enough to have some time developing with Django before the IP began, which simplified things.
This process has given me a new appreciation for several aspects of Django. One is its organizational structure. Each Django project is separated into distinct “apps” which each perform a separate function in the final program. This term, to create mlanalysis.herokuapp.com, I created two apps: Blog (where I hosted each of my posts) and Tracker (which contains all the data visualizations). Although Tracker won’t have much use outside of this context, the fact that each app works independently means that I, and others, can reuse the Blog app in future projects.
Although my website was relatively small, I can see how in larger projects Django would be incredibly useful as well. One of the most important rules in computer science is to never duplicate code. Templates can access each other’s code, which makes creating a basic background layout for your site incredibly easy, as every specific page can reuse the code that is written once in the base template.
The website building process also helped me gain insight into the decision making that goes into data visualization. Initially, I had a stacked bar which showed the cumulative karma gained by all bots. However, I had to consider the value of such a graph. What was its purpose? Through “stacking” karma I prioritized the total, instead of the initial focus of the experiment - to compare machine learning and non machine learning models. I have since replaced this graph with a more appropriate overlaid of the data.
Reddit and Bots
I spent a far greater amount of time developing and deploying the bots for this project than I expected. It seemed as though I ran into a setback at every point in the process. Because each bot required interacting with several external systems (PRAW for each of them, OED for the dictionary bot, etc.) I encountered a number of external errors. For example, if Reddit’s server when down I would get an error when trying to search, or if my bot had just recently commented I would hit a frame limit error. I could only catch each of these bugs after they happened once, and as some of there were quite rare, it took me over a week to handle each of them.
Beyond this, I hit several unforeseen obstructions in the bot’s performance. Reddit allows certain communities (subreddits) to ban users. As the machine learning versions of each bot started off with extremely poor performance, they were frequently banned before they had a chance to learn everything. Furthermore, since each sub had a new associated model, the bots essentially had to start from scratch whenever they started someone new. This was something I did not consider. JargonBot, a far more prolific poster then RepostBot, got banned from a huge number of subreddits. Unfortunately, RepostBot was banned from /r/jokes, the source of nearly all the reposts it could detect. I think if I could perform this experiment again, it would be smart to begin with some supervised learning of each model so that they reached a certain degree of competency, and only then release them to the public.
It was interesting to see the public response to each bot. At the moment I write this, JargonBot has accrued 431 karma, and RepostBot has accrued 2056. Here is a sample of comments each received.
- “was waiting for a bot like this”
- “Same title, different joke ya dumb boy”
- “I don't know what's worse. That we now have an actual repost bot or that it doesn't work”
- “No bot, your designer is a fool.”
- “What a useless bot.”
- “Worst bot ever. You are bad and you should feel bad.”
- “The definition is correct lil buddy but I’m not too sure that’s a difficult word. Keep trucking tho!”
- “My thoughts are you’re worse than the goddam mexicans. Stop trying to take our jobs you damned machines!”
It was interesting to see that although the majority of votes were positive (as both bots finished with positive karma scores), the majority of replies were negative. I also got a number of personal message on my main account from the most frustrated of users.
When I first started this IP, I expected that the non-machine learning algorithms would initially be more successful than the machine-learning counterparts, but that this would shift over time. The reality is entirely different. The machine-learning bots gained more karma consistently throughout the entire experiment. However, I don’t think this is because they made better posts - rather, they posted much more. The restrictions governing the condition to post for both the RepostBot and JargonBot were much more limiting for the non machine-learning versions (they would only post if they knew with near certainty it was a good idea). In contrast, the machine learning bots posted a huge amount. Many of those posts resulted in negative karma scores, but they were outweighed by the sheer volume of positive results. It looks like, at least on Reddit, quantity is greater than quality.