August 6, 2017

Sexist Programmers Are Awful Engineers

Men and women are fundamentally different. So are white people and black people and autistic people and gay people and transgender people and conservatives and liberals and every other human being along every imaginable axis of discrimination. Some of these differences are cultural. Others are genetic. Others depend on environmental factors. These differences mean that some of us are inherently better at certain tasks than others. On average, men are better at spatial temporal reasoning, women are better at reading comprehension and writing ability, and psychopaths can sometimes be excellent CEOs.

Whenever I meet a programmer who insists on doing everything a certain way, I make a mental note to never hire them, ever. Just as object-oriented programming didn't fix everything, neither will functional programming, or data-oriented programming or array-based programming or any other language. They are different tools that allow you to attack a problem from different directions, much like we have different classes of algorithms to attack certain classes of problems. Greedy algorithms, lazy evaluation, dynamic programming, recursive-descent, maximum flow, all of these are different ways to approach a problem. They represent looking at a problem from different perspectives. A problem that is difficult from one angle might be trivial when examined from a different angle.

When I stumbled upon this anti-diversity memo written by a Google employee, I wonder just how dysfunctional of an engineer that person is. Problems are never solved by being closed-minded. They are solved by opening ourselves to new possibilities and exploring the problem space as an infinitely-dimensional fabric of possible configurations. You do not find possible edge-cases by being closed-minded. You find them by probing the outer edges of your solution, trying to find singularities and inflection points that hint at unusual behavior.

You cannot build a great company by hiring people who are good at the same things you are. Attempting to maximize diversity only comes at a perceived cost of aptitude if you are measuring the wrong things. If your concept of what makes a "good programmer" is an extremely narrow set of skills, then you will inevitably select towards a specific ethnicity, culture, or sex, because the tiny statistical differences will be grossly magnified by the extremely narrow job requirements. Demand that all your programmers invert a binary tree on a whiteboard and you'll filter out the guy who wrote the software 90% of your company uses.

If you think the field of computer science is really this narrow, you're a terrible programmer. Turing completeness is a fundamental property of the universe, and we are only just beginning to explore the full implications of information theory, the foundations of type theory, NP-completeness, and the nature of computation itself. Disregarding other people because they can't do something without ever considering what they can do will only hurt your team, and your company. Diversity inclusion programs shouldn't try to hire more women and ethnic groups because they're the same, they should be trying to hire them because they are different.

When hiring someone to complete a job, you should hire whoever is the best fit for the job. In a vacuum where there is a single task that needs to be completed, gender and ethnicity should be ignored in favor of a purely meritocratic assessment. However, if you have a company that must respond to a changing world, diversity can reveal solutions you never even knew existed. An established company like Google must actively seek to increase diversity so that it can explore new perspectives that may give it an edge over its rivals. They cannot select on a purely meritocratic basis, because all measures of merit would be based on what the company is already good at, not what it could be good at. You cannot explore new opportunities by hiring the same people.

Intelligent people value feedback from people who think differently than them. This is why many executives will deliberately hire people they disagree with so they can have someone challenge their views. This helps avoid creating an echo-chamber, which is the ultimate irony of a memo that's called "Google’s Ideological Echo Chamber", because scrapping the diversity inclusion programs as the memo suggests would itself create a new echo-chamber. You can't remove an echo-chamber by removing diversity - the author's premise is self-defeating. If they had stuck with only claiming that conservative ideologies should not be discriminated against, they would have been correct. Unfortunately, telling everyone they shouldn't discriminate against your perspective, which itself involves discriminating against other perspectives, is by definition a contradiction.

We aren't going to write better programs by doing the same thing we've been doing for the past 10 years. To improve is to change, and those who seek to become better software engineers must themselves embrace change, or they will be left behind to rot in the sewers of forgotten programs, maintaining rancid enterprise code for the rest of their lives. If we are unwilling to change who is writing the programs, we'll be stuck making the same thing over and over again. A business that thinks nothing needs to change is one ripe for disruption. If you really think only hiring white males who correctly answer all your questions about graph theory and B-trees will help your business in the long-term, you're an idiot.

July 30, 2017

Why I Never Built My SoundCloud Killer

While the news of SoundCloud imploding are fairly recent, musicians and producers have had a beef with the music uploading site's direction for years. I still remember when SoundCloud only gave you a paltry 2 hours worth of free upload time and managed to convert my high quality lossless WAV files to the shittiest 128 kbps MP3 I've ever heard in my life. What really pissed me off was that they demanded a ridiculous $7 a month just to double your upload time. This is in contrast to Newgrounds, a tiny website run by a dozen people with an audio portal built almost as an afterthought that still manages to be superior to every single other offering. It gives you unlimited space, for free, and lets you upload your own MP3, which allows me to upload properly encoded joint-stereo 128 kbps MP3 files, or much higher quality MP3s for songs I'm giving out for free.

Obviously, Newgrounds is only able to offer unlimited free uploads because the audio portal just piggybacks on the rest of the site. However, I was so pissed off at SoundCloud's disgusting subscription offering that I actually ran the numbers in terms of what it would cost to store lossless FLAC encodings of songs using Amazon S3. These calculations are now out of date, so I've redone them for the purposes of this blog.

The average size of an FLAC encoded song is around 60 MB, but we'll assume it's 80 MB as an upper-bound, and to include the cost of storing the joint-stereo 128 kbps streaming MP3, which is usually less than 10% the size (using OPUS would reduce this even more, but it is not supported on all browsers yet). Amazon offers the first 50 TB of storage at $0.023 per gigabyte, per month. This comes down to about $0.00184 per month, per song, in order to store the full uncompressed version. Now, obviously, we must also stream the song, but we're only streaming the low-quality version, which is 10% the size, which is about 7 MB in our example (7 MB + 70 MB is about 80 MB for storage). The vast majority of music producers on the website have almost no following, and most will be lucky to get a single viral hit. As an example, after being on SoundCloud for over 7 years, I have managed to amass a mere 100000 views total. If I somehow got 20000 views of my songs every single month, the total cost of streaming 140 GB from Amazon S3 at $0.05 per GB would be $7 per month. That's how much SoundCloud is charging just to double my storage space!

This makes even less sense when you calculate that 6 hours of FLAC would be 4.7 GB, or about 5 GB including the 128 kbps streaming MP3s. 5 GB of storage space costs a pathetic $0.12 cents a month to store on Amazon S3! All of the costs come down to bandwidth, which is relatively fixed by how many people are listening to songs, not how many songs there are. This means, if I'm paying any music service for the right to store music on their servers, I should get near unlimited storage space (maybe put in a sanity check of 10 GB max per month to limit abuse). I will point out that Clyp.it actually does this properly, giving you 6 hours of storage space for free and unlimited storage space if you pay them $6 a month.

Unfortunately, Clyp.it does not try to be SoundCloud as it has no comments, no reshares, and well, not much of anything, really. It's like a giant pastebin for sounds where you can follow people or favorite things. It's also probably screwed.

Even though I had a name and a website design, I never launched it because even if I could find a way to identify a copyrighted song via some sort of ContentID system, I couldn't make it work without the record industry's cooperation. The problem is that the system has to know what songs are illegal in order to block them in the first place. Otherwise, people could upload Justin Bieber songs with impunity and I'd still get sued out of existence. The hard part about making a site like SoundCloud isn't actually making the website, it's dealing with the insane, litigation-happy oligarchs that own the entire music industry.

SoundCloud's ordeal is mentioned in this article. Surprisingly, it took until 2012 for them to realize they had to start making deals with the major music labels. It took until 2014 for many of those deals to actually happen, and they were not in SoundCloud's favor. A deal with Warner Music Group, closed in 2014, gave Warner a 3-5% stake in the company and an undisclosed cut of ad-revenue, just so SoundCloud could have the privilege of not being sued out of existence. This wasn't even an investment round, it was just so SoundCloud could have Warner Music Group's catalog on the site and not get sued!

At this point, you have to be either very naive or very rich to go up against an industry that can and will send an army of lawyers after you. The legal system is not in your favor. It will be used to crush you like a bug and there is nothing you can do about it, because of one fundamental problem: You can't detect copyright infringement without access to the original copy.

Because of this, the music industry holds the entire world hostage with a kind of Catch-22: They demand you take down all copyright infringing material, but in order to figure out if something is copyright infringement, you need access to their songs, which they only give out on their terms, which are never in your favor.

July 24, 2017

Integrating LuaJIT and Autogenerating C Bindings In Visual Studio

Lua is a popular scripting language due to its tight integration with C. LuaJIT is an extremely fast JIT compiler for Lua that can be integrated into your game, which also provides an FFI Library that directly interfaces with C functions, eliminating most overhead. However, the FFI library only accepts a subset of the C standard. Specifically, "C declarations are not passed through a C pre-processor, yet. No pre-processor tokens are allowed, except for #pragma pack." The website suggests running the header file through a preprocesser stage, but I have yet to find a LuaJIT tutorial that actually explains how to do this. Instead, all the examples simply copy+paste the function prototype into the Lua file itself. Doing this with makefiles and GCC is trivial, because you just have to add a compile step using the -E option, but integrating this with Visual Studio is more difficult. In addition, I'll show you how to properly load scripts and modify the PATH lookup variable so your game can have a proper scripts folder instead of dumping everything in bin.

Compilation

To begin, we need to download LuaJIT and get it to actually compile. Doing this manually isn't too difficult, simply open an x64 Native Tools Command Prompt (or x86 Native Tools if you want 32-bit), navigate to src/msvcbuild.bat and run msvcbuild.bat. The default options will build an x64 or x86 dll with dynamic linking to the CRT. If you want a static lib file, you need to run it with the static option. If you want static linking to the CRT so you don't have to deal with that annoying Visual Studio Runtime Library crap, you'll have to modify the .bat file directly. Specifically, you need to find %LJCOMPILE% /MD and change it to %LJCOMPILE% /MT. This will then compile the static lib or dll with static CRT linking to match your other projects.

This is a bit of a pain, and recently I've been trying to automate my build process and dependencies using vcpkg to act as a C++ package manager. A port of LuaJIT is included in the latest update of vcpkg, but if you want one that always statically links to the CRT, you can get it here.

An important note: the build instructions for LuaJIT state that you should copy the lua scripts contained in src/jit to your application folder. What it doesn't mention is that this is optional - those scripts contain debugging instructions for the JIT engine, which you probably don't need. It will work just fine without them.

Once you have LuaJIT built, you should add it's library file to your project. This library file is called lua51.lib (and the dll is lua51.dll), because LuaJIT is designed as a drop-in replacement for the default Lua runtime. Now we need to actually load Lua in our program and integrate it with our code. To do this, use lua_open(), which returns a lua_State* pointer. You will need that lua_State* pointer for everything else you do, so store it somewhere easy to get to. If you are building a game using an Entity Component System, it makes sense to build a LuaSystem that stores your lua_State* pointer.

Initialization

The next step is to load in all the standard Lua libraries using luaL_openlibs(L). Normally, you shouldn't do this if you need script sandboxing for player-created scripts. However, LuaJIT's FFI library is inherently unsafe. Any script with access to the FFI library can call any kernel API it wants, so you should be extremely careful about using LuaJIT if this is a use-case for your game. We can also register any C functions we want to the old-fashioned way via lua_register, but this is only useful for functions that don't have C analogues (due to having multiple return values, etc).

There is one function in particular that you probably want to overload, and that is the print() function. By default, Lua will simply print to standard out, but if you aren't redirecting standard out to your in-game console, you probably have your own std::ostream (or even a custom stream class) that is sent all log messages. By overloading print(), we can have our Lua scripts automatically write to both our log file and our in-game console, which is extremely useful. Here is a complete re-implementation of print that outputs to an arbitrary std::ostream& object:
int PrintOut(lua_State *L, std::ostream& out)
{
  int n = lua_gettop(L);  /* number of arguments */
  if(!n)
    return 0;
  int i;
  lua_getglobal(L, "tostring");
  for(i = 1; i <= n; i++)
  {
    const char *s;
    lua_pushvalue(L, -1);  /* function to be called */
    lua_pushvalue(L, i);   /* value to print */
    lua_call(L, 1, 1);
    s = lua_tostring(L, -1);  /* get result */
    if(s == NULL)
      return luaL_error(L, LUA_QL("tostring") " must return a string to "
        LUA_QL("print"));
    if(i > 1) out << "\t";
    out << s;
    lua_pop(L, 1);  /* pop result */
  }
  out << std::endl;
  return 0;
}
To overwrite the existing print function, we need to first define a Lua compatible shim function. In this example, I pass std::cout as the target stream:

int lua_Print(lua_State *L)
{
  return PrintOut(L, std::cout);
}
Now we simply register our lua_Print function using lua_register(L, "print", &lua_Print). If we were doing this in a LuaSystem object, our constructor would look like this:

LuaSystem::LuaSystem()
{
  L = lua_open();
  luaL_openlibs(L);
  lua_register(L, "print", &lua_Print);
}
To clean up our Lua instance, we need to both trigger a final GC iteration to clean up any dangling memory, and then we call lua_close(L), so our destructor would look like this:
LuaSystem::~LuaSystem()
{
  lua_gc(L, LUA_GCCOLLECT, 0);
  lua_close(L);
  L = 0;
}

Loadings Scripts via Require

At this point most tutorials skip to the part where you load a Lua script and write "Hello World", but we aren't done yet. Integrating Lua into your game means loading scripts and/or arbitrary strings as Lua code while properly resolving dependencies. If you don't do this, any one of your scripts that relies on another script will have to do require("full/path/to/script.lua"). We also face another problem - if we want to have a scripts folder where we simply automatically load every single script into our workspace, simply loading them all can cause duplicated code, because luaL_loadfile does not have any knowledge of require. You can solve this by simply loading a single bootstrap.lua script which then loads all your game's scripts via require, but we're going to build a much more robust solution.

First, we need to modify Lua's PATH variable, or the variable that controls where it looks up scripts relative to our current directory. This function will append a path (which should be of the form "path/to/scripts/?.lua") to the beginning of the PATH variable, giving it highest priority, which you can then use to add as many script directories as you want in your game, and any lua script from any of those folders will then be able to require() a script from any other folder in PATH without a problem. Obviously, you should probably only add one or two folders, because you don't want to deal with potential name conflicts in your script files.
int AppendPath(lua_State *L, const char* path)
{
  lua_getglobal(L, "package");
  lua_getfield(L, -1, "path"); // get field "path" from table at top of stack (-1)
  std::string npath = path;
  npath.append(";");
  npath.append(lua_tostring(L, -1)); // grab path string from top of stack
  lua_pop(L, 1);
  lua_pushstring(L, npath.c_str());
  lua_setfield(L, -2, "path"); // set the field "path" in table at -2 with value at top of stack
  lua_pop(L, 1); // get rid of package table from top of stack
  return 0;
}
Next, we need a way to load all of our scripts using require() so that Lua properly resolves the dependencies. To do this, we create a function in C that literally calls the require() function for us:
int Require(lua_State *L,const char *name)
{
  lua_getglobal(L, "require");
  lua_pushstring(L, name);
  int r = lua_pcall(L, 1, 1, 0);
  lua_pop(L, 1);
  WriteError(L, r, std::cout);
  return r;
}
By using this to load all our scripts, we don't have to worry about loading them in any particular order - require will ensure everything gets loaded correctly. An important note here is WriteError(), which is a generic error handling function that processes Lua errors and writes them to a log. All errors in lua will return a nonzero error code, and will usually push a string containing the error message to the stack, which must then be popped off, or it'll mess things up later.
void WriteError(lua_State *L, int r, std::ostream& out)
{
  if(!r)
    return;
  if(!lua_isnil(L, -1)) // Check if a string was pushed
  {
    const char* m = lua_tostring(L, -1);
    out << "Error " << r << ": " << m << std::endl;
    lua_pop(L, 1);
  }
  else
    out << "Error " << r << std::endl;
}

Automatic C Binding Generation

Fantastic, now we're all set to load up our scripts, but we still need to somehow define a header file and also load that header file into LuaJIT's FFI library so our scripts have direct access to our program's exposed C functions. One way to do this is to just copy+paste your C function definitions into a Lua file in your scripts folder that is then automatically loaded. This, however, is a pain in the butt and is error-prone. We want to have a single source of truth for our function definitions, which means defining our entire LuaJIT C API in a single header file, which is then loaded directly into LuaJIT. Predictably, we will accomplish this by abusing the C preprocessor:

#ifndef __LUA_API_H__
#define __LUA_API_H__

#ifndef LUA_EXPORTS
#define LUAFUNC(ret, name, ...) ffi.cdef[[ ret lua_##name(__VA_ARGS__); ]]; name = ffi.C.lua_##name
local ffi = require("ffi")
ffi.cdef[[ // Initial struct definitions
#else
#define LUAFUNC(ret, name, ...) ret __declspec(dllexport) lua_##name(__VA_ARGS__)
extern "C" { // Ensure C linkage is being used
#endif

struct GameInfo
{
  uint64_t DashTail;
  uint64_t MaxDash;
};

typedef const char* CSTRING; // VC++ complains about having const char* in macros, so we typedef it here

#ifndef LUA_EXPORTS
]] // End struct definitions
#endif

  LUAFUNC(CSTRING, GetGameName);
  LUAFUNC(CSTRING, IntToString, int);
  LUAFUNC(void, setdeadzone, float);

#ifdef Everglade_EXPORTS
}
#endif

#endif
The key idea here is to use macros such that, when we pass this through the preprocessor without any predefined constants, it will magically turn into a valid Lua script. However, when we compile it in our C++ project, our project defines LUA_EXPORTS, and the result is a valid C header. Our C LUAFUNC is set up so that we're using C linkage for our structs and functions, and that we're exporting the function via __declspec(dllexport). This obviously only works for Visual Studio so you'll want to set up a macro for the GCC version, but I will warn you that VC++ got really cranky when i tried to use a macro for that in my code, so you may end up having to redefine the entire LUAFUNC macro for each compiler.

At this point, we have a bit of a choice to make. It's more convenient to have the C functions available in the global namespace, which is what this script does, because this simplifies calling them from an interactive console. However, using ffi.C.FunctionName is significantly faster. Technically the fastest way is declaring local C = ffi.C at the top of a file and then calling the functions via C.FunctionName. Luckily, importing the functions into the global namespace does not preclude us from using the "fast" way of calling them, so our script here imports them into the global namespace for ease of use, but in our scripts we can use the C.FunctionName method instead. Thus, when outputting our Lua script, our LUAFUNC macro wraps our function definition in a LuaJIT ffi.cdef block, and then runs a second Lua statement that brings the function into the global namespace. This is why we have an initial ffi.cdef code block for the structs up top, so we can include that second lua statement after each function definition.

Now we need to set up our compilation so that Visual Studio generates this file without any predefined constants and outputs the resulting lua script to our scripts folder, where our other in-game scripts can automatically load it from. We can accomplish this using a Post-Build Event (under Configuration Properties -> Build Events -> Post-Build Event), which then runs the following code:
CL LuaAPI.h /P /EP /u
COPY "LuaAPI.i" "../bin/your/script/folder/LuaAPI.lua" /Y
Visual Studio can sometimes be finicky about that newline, but if you put in two statements on two separate lines, it should run both commands sequentially. You may have to edit the project file directly to convince it to actually do this. The key line here is CL LuaAPI.h /P /EP /u, which tells the compiler to preprocess the file and output it to a *.i file. There is no option to configure the output file, it will always be the exact same file but with a .i extension, so we have to copy and rename it ourselves to our scripts folder using the COPY command.

Loading and Calling Lua Code

We are now set to load all our lua scripts in our script folder via Require, but what if we want an interactive Lua console? There are lua functions that read strings, but to make this simpler, I will provide a function that loads a lua script from an arbitrary std::istream and outputs to an arbitrary std::ostream:
const char* _luaStreamReader(lua_State *L, void *data, size_t *size)
{
  static char buf[CHUNKSIZE];
  reinterpret_cast(data)->read(buf, CHUNKSIZE);
  *size = reinterpret_cast(data)->gcount();
  return buf;
}

int Load(lua_State *L, std::istream& s, std::ostream& out)
{
  int r = lua_load(L, &_luaStreamReader, &s, 0);

  if(!r)
  {
    r = lua_pcall(L, 0, LUA_MULTRET, 0);
    if(!r)
      PrintOut(L, out);
  }

  WriteError(L, r, out);
  return r;
}

Of course, the other question is how to call Lua functions from our C++ code directly. There are many, many different implementations of this available, of varying amounts of safety and completeness, but to get you started, here is a very simple implementation in C++ using templates. Note that this does not handle errors - you can change it to use lua_pcall and check the return code, but handling arbitrary Lua errors is nontrivial.
template<class T, int N>
struct LuaStack;

template<class T> // Integers
struct LuaStack<T, 1>
{
  static inline void Push(lua_State *L, T i) { lua_pushinteger(L, static_cast<lua_Integer>(i)); }
  static inline T Pop(lua_State *L) { T r = (T)lua_tointeger(L, -1); lua_pop(L, 1); return r; }
};
template<class T> // Pointers
struct LuaStack<T, 2>
{
  static inline void Push(lua_State *L, T p) { lua_pushlightuserdata(L, (void*)p); }
  static inline T Pop(lua_State *L) { T r = (T)lua_touserdata(L, -1); lua_pop(L, 1); return r; }
};
template<class T> // Floats
struct LuaStack<T, 3>
{
  static inline void Push(lua_State *L, T n) { lua_pushnumber(L, static_cast<lua_Number>(n)); }
  static inline T Pop(lua_State *L) { T r = static_cast<T>(lua_touserdata(L, -1)); lua_pop(L, 1); return r; }
};
template<> // Strings
struct LuaStack<std::string, 0>
{
  static inline void Push(lua_State *L, std::string s) { lua_pushlstring(L, s.c_str(), s.size()); }
  static inline std::string Pop(lua_State *L) { size_t sz; const char* s = lua_tolstring(L, -1, &sz); std::string r(s, sz); lua_pop(L, 1); return r; }
};
template<> // Boolean
struct LuaStack<bool, 1>
{
  static inline void Push(lua_State *L, bool b) { lua_pushboolean(L, b); }
  static inline bool Pop(lua_State *L) { bool r = lua_toboolean(L, -1); lua_pop(L, 1); return r; }
};
template<> // Void return type
struct LuaStack<void, 0> { static inline void Pop(lua_State *L) { } };

template<typename T>
struct LS : std::integral_constant<int, 
  std::is_integral<T>::value + 
  (std::is_pointer<T>::value * 2) + 
  (std::is_floating_point<T>::value * 3)>
{};

template<typename R, int N, typename Arg, typename... Args>
inline R _callLua(const char* function, Arg arg, Args... args)
{
  LuaStack<Arg, LS<Arg>::value>::Push(_l, arg);
  return _callLua<R, N, Args...>(function, args...);
}
template<typename R, int N>
inline R _callLua(const char* function)
{
  lua_call(_l, N, std::is_void<R>::value ? 0 : 1);
  return LuaStack<R, LS<R>::value>::Pop(_l);
}

template<typename R, typename... Args>
inline R CallLua(lua_State *L, const char* function, Args... args)
{
  lua_getglobal(L, function);
  return _callLua<R, sizeof...(Args), Args...>(L, function, args...);
}
Now you have everything you need for an extensible Lua scripting implementation for your game engine, and even an interactive Lua console, all using LuaJIT. Good Luck!

June 24, 2017

Discord: Rise Of The Bot Wars

The most surreal experience I ever had on discord was when someone PMed me to complain that my anti-spam bot wasn't working against a 200+ bot raid. I pointed out that it was never designed for large-scale attacks, and that discord's own rate-limiting would likely make it useless. He revealed he was selling spambot accounts at a rate of about $1 for 100 unique accounts and that he was being attacked by a rival spammer. My anti-spam bot had been dragged into a turf war between two spambot networks. We discussed possible mitigation strategies for worst-case scenarios, but agreed that most of them would involve false-positives and that discord showed no interest in fixing how exploitable their API was. I hoped that I would never have to implement such extreme measures into my bot.

Yesterday, our server was attacked by over 40 spambots, and after discord's astonishingly useless "customer service" response, I was forced to do exactly that.

A Brief History of Discord Bots

Discord is built on a REST API, which was reverse engineered by late 2015 and used to make unofficial bots. To test out their bots, they would hunt for servers to "raid", invite their bots to the server, then spam so many messages it would softlock the client, because discord still didn't have any rate limiting. Naturally, as the designated punching bags of the internet, furries/bronies/Twilight fans/slash fiction writers/etc. were among the first targets. The attack on our server was so severe it took us almost 5 minutes of wrestling with an unresponsive client to ban them. Ironically, a few of the more popular bots today, such as "BooBot", are banned as a result of that attack, because the first thing the bot creator did was use it to raid our server.

I immediately went to work building an anti-spam bot that muted anyone sending more than 4 messages per second. Building a program in a hostile environment like this is much different from writing a desktop app or a game, because the bot had to be bulletproof - it had to rate-limit itself and could not be allowed to crash, ever. Any bug that allowed a user to crash the bot was treated as P0, because it could be used by an attacker to cripple the server. Despite using a very simplistic spam detection algorithm, this turned out to be highly effective. Of course, back then, discord didn't have rate limiting, or verification, or role hierarchies, or searching chat logs, or even a way to look up where your last ping was, so most spammers were probably not accustomed to having to deal with any kind of anti-spam system.

I added raid detection, autosilence, an isolation channel, and join alerts, but eventually we were targeted by a group from 4chan's /pol/ board. Because this was a sustained attack, they began crafting spam attacks timed just below the anti-spam threshold. This forced me to implement a much more sophisticated anti-spam system, using a heat algorithm with a linear decay rate, which is still in use today. This improved anti-spam system eventually made the /pol/ group give up entirely. I'm honestly amazed the simplistic "X messages in Y seconds" approach worked as long as it did.

Of course, none of this can defend against a large scale attack. As I learned by my chance encounter with an actual spammer, it was getting easier and easier to amass an army of spambots to assault a channel instead of just using one or two.

Anatomy Of A Modern Spambot Attack

At peak times (usually during summer break), our server gets raided 1-2 times per day. These minor raids are often just 2-3 tweens who either attempt to troll the chat, or use a basic user script to spam an offensive message. Roughly 60-70% of these raids are either painfully obvious or immediately trigger the anti-spam bot. About 20% of the raids involve slightly intelligent attempts to troll the chat by being annoying without breaking the rules, which usually take about 5-10 minutes to be "exposed". About 5-10% of the raids are large, involving 8 or more people, but they are also very obvious and can be easily confined to an isolation channel. Problems arise, however, with large spambot raids. Below is a timeline of the recent spambot attack on our server:

messages
19:41:25
19:41:45
19:42:05
19:42:25
19:42:45

This was a botched raid, but the bots that actually worked started spamming within 5 seconds of joining, giving the moderators a very narrow window to respond. The real problem, however, is that so many of them joined, the bot's API calls to add a role to silence them were rate-limited. They also sent messages once every 0.9 seconds, which is designed to get around Discord's rate limiting. This amounted to 33 messages sent every second, but it was difficult for the anti-spam to detect. Had the spambots reduced their spam cadence to 3 seconds or more, this attack could have bypassed the anti-spam detection entirely. My bot now instigates a lockdown by raising the verification level when a raid is detected, but it simply can't silence users fast enough to deal with hundreds of spambots, so at some point the moderators must use a mass ban function. Of course, banning is restricted by the global rate limit, because Discord has no mass ban API endpoint, but luckily the global rate limit is something like 50 requests per second, so if you're only banning people, you're probably okay.

However, a hostile attacker could sneak bots in one-by-one every 10 minutes or so, avoiding setting off the raid alarm, and then activate them all at once. 500 bots sending randomized messages chosen from an English dictionary once every 5 seconds after sneaking them in over a 48 hour period is the ultimate attack, and one that is almost impossible to defend against, because this also bypasses the 10-minute verification level. As a weapon of last resort, I added a command that immediately bans all users that sent their first message within the past two minutes, but, again, banning is subject to the global rate limit! In fact, the rate limits can change at any time, and while message deletion has a higher rate limit for bots, bans don't.

The only other option is to disable the @everyone role from being able to speak on any channel, but you have to do this on a per channel basis, because Discord ignores you if you attempt to globally disable sending message permissions for @everyone. Even then, creating an "approved" role doesn't work because any automated assignment could be defeated by adding bots one by one. The only defense a small Discord server has is to require moderator approval for every single new user, which isn't a solution - you've just given up having a public Discord server. It's only a matter of time until any angry 13-year-old can buy a sophisticated attack with a week's allowance. What will happen to public Discord servers then? Do we simply throw up our hands and admit that humanity is so awful we can't even have public communities anymore?

The Discord API Hates You

The rate-limits imposed on Discord API endpoints are exacerbated by temporary failures, and that's excluding network issues. Thus, if I attempt to set a silence role on a spammer that just joined, the API will repeatedly claim they do not exist. In fact, 3 separate API endpoints consistently fail to operate properly during a raid: A "member joined" event won't show up for several seconds, but if I fall back to calling GetMember(), this also claims the member doesn't exist, which means adding the role also fails! So I have to attempt to silence the user with every message they send until Discord actually adds the role, even though the API failures are also counted against the rate limit! This gets completely absurd once someone assaults your server with 1000 spambots, because this triggers all sorts of bottlenecks that normally aren't a problem. The alert telling you a user has joined? Rate limited. It'll take your bot 5-10 minutes to get through just telling you such a gigantic spambot army joined, unless you include code specifically designed to detect these situations and reduce the number of alerts. Because of this, a single user can trigger something like 5-6 API requests, all of which are counted against your global rate limit and can severely cripple a bot.

The general advice that is usually given here is "just ban them", which is terrible advice because Discord's own awful message handling makes it incredibly easy to trigger a false positive. If a message fails to send, the client simply sends a completely new message, with it's own ID, and will continue re-sending the message until an Ack is received, at which point the user has probably send 3 or 4 copies of the same message, each of which have the same content, but completely unique IDs and timestamps, which looks completely identical to a spam attack.

Technically speaking, this is done because Discord assigns snowflake IDs server-side, so each message attempt sent by the client must have a unique snowflake assigned after it is sent. However, it can also be trivially fixed by adding an optional "client ID" field to the message, with a client-generated ID that stays the same if the message is resent due to a network failure. That way, the server (or the other clients) can simply drop any duplicate messages with identical client IDs while still ensuring all messages have unique IDs across their distributed cluster. This would single-handedly fix all duplicate messages across the entire platform, and eliminate almost every single false-positive I've seen in my anti-spam bot.

Discord Doesn't Care

Sadly, Discord doesn't seem to care. The general advice in response to "how do I defend against a large scale spam attack" is "just report them to us", so we did exactly that, and then got what has to be one of the dumbest customer service e-mails I've ever seen in my life:

Discord Being Stupid

Excuse me, WHAT?! Sorry about somebody spamming your service with horrifying gore images, but please don't delete them! What happens if the spammers just delete the messages themselves? What happens if they send child porn? "Sorry guys, please ignore the images that are literally illegal to even look at, but we can't delete them because Discord is fucking stupid." Does Discord understand the concept of marking messages for deletion so they are viewable for a short time as evidence for law enforcement?! My anti-spam bot's database currently has more information than Discord's own servers! If this had involved child porn, the FBI would have had to ask me for my records because Discord would have deleted them all!

Obviously, we're not going to leave 500+ gore messages sitting in the chatroom while Discord's ass-backwards abuse team analyzes them. I just have to hope my own nuclear option can ban them quickly enough, or simply give up the entire concept of having a public Discord server.

The problem is that the armies of spambots that were once reserved for the big servers are now so easy and so trivial to make that they're beginning to target smaller servers, servers that don't have the resources or the means to deal with that kind of large scale DDoS attack. So instead, I have to fight the growing swarm alone, armed with only a crippled, rate-limited bot of my own, and hope the dragons flying overhead don't notice.

What the fuck, Discord.


June 1, 2017

Programmers Should Take Linguistics

The older I get, the more I realize that 90% of all disagreements or social drama results from a miscommunication of some kind. Every time I wind up having to resolve a dispute, I'll try to get both sides of the story, only to realize that they're the same story, and both parties were actually either in agreement or fighting over a perceived insult that never actually existed. Unsurprisingly, a disproportionate amount of this miscommunication often involves programmers being far too strict with their interpretations of what certain words mean.

A linguistics class teaches you about what a language actually is - a bunch of sounds that we mutually agree mean certain things. Language is, intrinsically, a social construct. The correct definition of a word is whatever the majority of your primary social circle thinks it is. However, this also means that if you interact with a secondary social circle, and they all think the word means something else, then whenever you interact with them, it really does mean something else. Language is inherently contextual, and the possible meanings of a word can change based on who is saying it and to whom they're saying it to. If everyone else on Earth has decided that 'literally' can also mean 'figuratively', then it does, even if the dictionary says otherwise. It also means most people don't actually care if you say jif or gif, they'll just say whatever pronunciation gets you to shut up about it.

It's important to realize that a word's meaning is not defined by a dictionary, but rather by how people use it. The dictionary is simply a reflection of it's usage, and is generally a few years out of date. Just as the pronunciation of a word can vary by dialect, so can the potential meanings of a word. Meanings can be invented by regional subdialects and spread outward from there, which is the origin of many slang terms. Sometimes we invent entirely new words, like "dubstep", but young words may have fuzzy definitions. In some dialects of electronic music listeners, "dubstep" is not actually a a specific genre, but instead refers to all electronic music. Using dubstep to refer to any electronic song is currently incorrect if used in general parlance, because most people think it is referring to a very specific kind of music. However, if this usage of the word continues to be popularized, eventually the meaning of the word will change into a synonym for electronica, and the dictionaries will be updated to reflect this.

The fluid nature of language is why prescriptive grammar is almost always unnecessary, unless you are deliberately conforming to a grammar standard for a specific medium, such as writing a story. In almost any other context, so long as everyone in your social group understands your 'dialect' of English, then it is valid grammar. However, if you attempt to use this dialect outside of your social circle with people who are not familiar with it, you will once again be in the wrong, as they will have no idea what you're talking about. This, however, does not mean there are no mandatory grammar rules, it's just that most of the rules that are actually necessary to speak the language properly are usually so ingrained that you don't even think about them.

A fantastic example of this is a little known rule in English where all adjectives must come in a very specific order: opinion-size-age-shape-color-origin-material-purpose Noun. So you can have a lovely little old rectangular green French silver whittling knife, but if you switch the order of any of those adjectives you'll sound like a maniac. Conversely, you can never have a green great dragon. Despite the fact that this grammar rule is basically never talked about in any prescriptive grammar book, it is mandatory because if you don't follow it you won't be speaking proper English and people will have difficulty understanding you. True grammar rules are ones that, if not followed, result in nonsensical sentences that are difficult or impossible to parse correctly.

However, this does not mean all sentences that are difficult to understand have incorrect grammar. In fact, even some words are completely ambiguous by default. If I say I'm "dusting an object", the meaning of the phrase is completely dependent on what the object is. If it's a cake, I'm probably dusting it with something. If it's a shelf, I'm probably dusting it to get rid of the dust.

Programmers tend to be very literal minded people, and often like to think that language is a set of strict rules defined by their English class. In reality, language is a fluid, dynamic, ambiguous, constantly changing enigma that exists entirely because we all agree on what a bunch of sounds mean. We need to recognize this, and when we communicate to other people, we need to be on the lookout for potential misinterpretations of what we say, so we can provide clarifications when possible. If someone says something that seems ridiculous, ask them to clarify. I'm tired of resolving disagreements that exist only because nobody stopped to ask the other side to clarify what they meant.

Stop demanding that everyone explain things in a way you'll understand. That's impossible, because everyone understands language slightly differently. Instead, ask for clarification if someone seems to be saying something unusual or before you debate a point they made. Maybe then we can keep the debates to actual disagreements, instead of arguing over communication failures.