Tetris Clone in C and SDL2 – Post C++ Rewrite C Refactoring

This is the third retrospective document about this project. The first one was about my actual specs for the program. The second was a post analysis of the project’s implementation. This one is the result of a refactor due to a code review I had to do while doing the C++ re-write of this project, which actually also yield a document as well.

It all started with a tetris clone I wrote in C and SDL2.

Notice that unless said differently, “object” has its C meaning, which is pretty much a memory region.

The Major Change – Error Handling

As was mentioned several times when an error happens, the program quits with an
error message.

One of the things I’ve noticed in my code review is that the error handling
mechanism had a too much complicated clean up scheme. It depended on series of
labels whose cascading would lead to the right clean up. This scheme had an
advantage. It was local. It also required a lot of those clean up code
everywhere, and sometimes even duplication of clean up code (one under the
error related labels, and another in a destroy function). For example, while
loading a screen, if it fails, init must clean up, but the screen abstract
object must also contain a destroy function will cleans up the screen.

So the first change in error handling is that although local clean up is still
allowed (and done in some places), it’s no longer necessary. All modules will
have their destroy functions called at the end, and they should be able to
do something reasonable even if the module hasn’t been initialized yet.

For example, if initializing the assets module fails, the program shuts down by
calling all the clean up procedures available (more on this later), reporting
the error, and exiting. One of the available modules is the music module. The
music module is only initialized after the assets module as it happens to be
right now (that is not to be relied upon, but it is how it is). So its destroy
procedure gets called even though it wasn’t initialized. It should be, and is,
able to deal with such cases.

In general, what this first change means is that if a clean up function gets called, it should be able to deal with an “uninitialized situation” in the case of a module. This last point should be clear: “in the case of a module”. Although there is a module implementing the data type TextImage, an instance of TextImage itself isn’t a module. This first change isn’t saying that destroy_text_image has to work on an uninitialized TextImage object.

The way modules can deal with this first change is by using module local variables. It’s no big deal, but it makes cleanup much easier.

The second change has to do with reporting the error. Back then, when an error happened, what one would see is the error message from a direct call to one of (SDL|IMG|TTF|Mix)_GetError. There were problems with this approach. I never faced those problems, but I thought it’d be interesting to fix them anyway.

For example, once an error is identified, errorFn, which doesn’t exist anymore, was set to the appropriated function pointer ((SDL|IMG|TTF|Mix)_GetError for example) and a goto statement to the appropriated clean up label was executed. During the clean up SDL functions would be called to clean up previously set up state. The issue is that some SDL functions set the error message even though they don’t fail. So after the clean up, the error message set internally in the SDL and helper libraries could be completely different from what it was when the error happened.

This means that I should be saving the contents of the message at the point of failure instead of saving the function pointer.

Another issue with error handling was that there was no trace. I don’t mean a full stack trace, but there is a sequence of calls which lead to failure, and I’d like to know what is that sequence. My attempt to do that led me to complete a simple, but pretty useful as it seems, error reporting system. Here is an example:

[tetris] (master)$ ./main
Error! Aborting program.
Bread Crumbs:
=============
text_image.c: init_text_image: L30: (missing message)
FAILED CHECK: !stext
menu.c: init_menu: L71: (missing message)
FAILED CHECK: init_text_image(&title, title_font, "Tetris", g_rend, &DEFAULT_FG_COLOR)
main.c: init_screens: L51: (missing message)
FAILED CHECK: init_menu(rend, &screen_size)
main.c: init: L67: (missing message)
FAILED CHECK: init_screens()
main.c: main: L182: (missing message)
FAILED CHECK: init()

SDL_GetError() => (empty)
IMG_GetError() => (empty)
TTF_GetError() => (empty)
Mix_GetError() => (empty)

This is a report of what gets collected when an error happens. It points that there was a sequence of calls: main -> init -> init_screens -> init_menu -> init_text_image. It points out which checks failed, the line of code in which the check was being performed, and also the file name. The message, which is missing in all of the entries is something custom (at error checking point, it has to be provided, or it’ll be missing). The error message is used only for the case in which I want to copy the error message contents from (SDL|IMG|TTF|Mix)_GetError. The 4 last lines tell the current messages reported by the (SDL|IMG|TTF|Mix)_GetError functions at error reporting time. This could be useful, but I’m not so sure.

Here is an example of when there is an error message to set:

[tetris] (master)$ ./main
Error! Aborting program.
Bread Crumbs:
=============
assets.c: init_assets: L44: Couldn't open arcade.sttf
FAILED CODE: large_font
main.c: init: L65: (missing message)
FAILED CODE: init_assets(rend)
main.c: main: L173: (missing message)
FAILED CODE: init()

SDL_GetError() => Couldn't open arcade.sttf
IMG_GetError() => Couldn't open arcade.sttf
TTF_GetError() => Couldn't open arcade.sttf
Mix_GetError() => Couldn't open arcade.sttf

This is all using standard C. I imagine that going OS specific, something better can be done.

The way this is done is by changing how to error macros work. Doing something like

COND_ERROR_SET(error_check, label, function_pointer);

would expand to pretty much into

do {
  if (!error_check) {
    errorFn = function_pointer;
    goto label;
  }
} while (0)

Now what happens is that there are other macros. This one (COND_ERROR_SET) and its friend (COND_ERROR) are gone. What exists now is for example, COND_ERET_LT0. Here is how you’d use it:

COND_ERET_LT0(SDL_Init(SDL_INIT_VIDEO), SDL_GetMessage());

It’s defined as follows:

#define COND_ERET_LT0(EXPR, MSG) do { \
    int ERROR_H__err = (EXPR); \
    if (ERROR_H__err < 0) { \
      link_error((MSG), __LINE__, __FILE__, __func__, # EXPR); \
      return ERROR_H__err; \
    } \
  } while (0)

The call to link_error adds an entry to the error module, which maintains an internal structure of error entries. Each entry you add is chained together with the existing ones. At any point you can ask for a copy of the error chain (which you’d have to free yourself later on through a call to free_error).

There are macros for:

  • return and gotos (RET, GOTO);
  • true conditionals, false conditionals and less than 0 results (no suffix, IF0, LT0);
  • when there is a message, and when there isn’t (E, P).

So for example, COND_PGOTO_LT0 will goto the given label if the given expression is less than 0, and also it’ll provide no error message.

Error messages are always the last parameter to the macro application, and the error check always the first. Some times, three parameters are needed, and the one in the middle has custom semantics. For example, COND_ERET_LT0 will return whatever the failing expression will return in the case of an error, but COND_ERET_IF0 needs a parameter which indicates which is the return value:

COND_ERET_IF0(sdl_renderer, -1, SDL_GetError());

Will return -1 if sdl_renderer is false-ish. It’ll also add the error message returned by SDL_GetError in the case of an error.  The downside of this approach is that it fills the code with macro applications, which were already in for the old error handling approach (they were just different macros), so that didn’t make the code any worse. Another downside is the use of that ERROR_H__err variable inside a do..while block. I don’t think that’s going to be an issue. The error module relies on the ability to declare variables whose identifiers are prefixed with ERROR_H inside those do..while blocks.

As a final, and less important downside, the error module relies heavily on dynamic allocation, which I was hoping to avoid in this program, but since it’s only for the error gathering/reporting mechanism, which is only used when the program is about the shut down, I don’t think it’ll be an issue.

A benefit is that it’s general. It doesn’t depend on this tetris program at all.

The struct ErrorInfo

These are the two structs that power error handling.

struct BasicString {
  char *data;
  int len;
};

struct ErrorInfo {
  struct BasicString msg, file_name, func_name, code;
  int line;
  struct ErrorInfo *next;
};

This is a basic linked list structure. Each link has information abouts its error entry and a pointer to the call which led to it. There are them 3 functions for error handling:

int
link_error(const char *msg,
           int line,
           const char *file_name,
           const char *func_name,
           const char *code);

struct ErrorInfo*
get_error(void);

void
free_error(struct ErrorInfo *err);

Notice that link_error, alone, is very clumsy to use. It’s not meant to be used directly, even though you can. It’s meant to be used through the error handling macros.

When you ask for error information with get_error, you get a copy of the internal error structure, which you should free once you’re done with it through the free_error function. If you pass null to free_error, it’s going to free the internall error structure. This is useful for program shutdown. It may be a bad feature to have because it means you have check for null before calling it if you don’t want to free the internet structure.

Freeing the internal structure isn’t a problem if you don’t care about losing it. In fact, if you want to recover from an error, what you should do is free the internal structure so it gets out of the way of newer error messages after the recovery.

Error recovery, though, isn’t much supported by this module. As I said before, error handling in the program is pretty much: clean up, report, and quit. However, it wouldn’t be hard to add more information to the ErrorInfo structure so that it can record error problems. For example:

  • It could have an error code (int) and context memory (void*) that is fully user manipulated.
  • Users could create their own special macros that work in terms of ones provided by the error module. These macros would call the underlying error.h macros with the right error code and context creation code.
  • Convention such as ERROR_H prefixed identifiers could be used to avoid major identifier clashing.

It wouldn’t be the best or a pleasant system to work with. Error handling in C isn’t something I know how to deal with in a way that I find enjoyable. But it would be useful and allow for the kinds of operations I want to work with: error issuing, reporting, handling/recovering.

Bugs

Doing the C++ rewrite showed me that it was useful to have a SDL wrapper for raising exceptions whenever SDL failed underneath. This made the C++ code for error issuing and reporting much more convenient to deal with than the C version.

As a side note, the C++ error issuing/reporting code is still more convenient to work with in my opinion, even after the C refactoring. However, the current C version of the error handling code is much more useful. A similar mechanism would also be possible in C++, but it’d take away its convenience.

During the creation of that SDL wrapper, I had to look at the docs of many SDL functions and found out I didn’t know SDL as well as I thought. I hadn’t realized how many SDL calls can actually fail. As of now, writing this document, the bugs I can remember all had to do with that: making calls to SDL functions without realising they could fail.

Single Threading

The system is currently single threaded. The error handling system can be made multi threaded simply by making use of thread local variables. Each thread needs its own error handling subsystem. Their data aren’t supposed to be shared.

For the other system, something message passing-wise seems appropriated, but I haven’t thought of anything yet. The games I have in mind are too simple to need any of the complications of multi-threaded code.

The Quake Games

The Quake games (by ID software) were all done mostly in C. Maybe I’ll look how error handling was done in there when I get to more complicated games.

Tetris Clone Rewrite – in C++ and SDL2

Tags

, , , , , ,

Some weeks ago, I’ve written a tetris clone in C. I decided then to work a little more on the project to experiment with some ideas. One of which is to see what I could do differently, if not better, by rewriting the code in C++. The result is in this github repository for the C++ rewrite. The C version is on github at http://github.com/phao/tetris. I’ve decided to rewrite the code in C++ to see if I could improve it somehow. I’ve rewritten, and this document is a post-analysis of the whole process and end results.

Code Size

The C++ the line count is higher.

  • C is 1566 LOC
  • C++ is 1927 LOC

This isn’t to say that the C++ version was worse in any way. The increase in LOC has to do with the following:

  • Larger header files to contain class definitions. The C version has separate modules for different functionalities while the C++ code has different classes.
  • The C++ version has a wrapper on the SDL library while the C version doesn’t. This wrapper alone is 397 LOC.

These LOC values include comments, blank lines and so forth. There was no attempt to make it “more meaningful”, whatever that means to you.

C Modules and C++ Classes

The solution/implementation involved many concepts, including:

  • 3 “alike” screens;
  • an assets gateway;
  • a music player;
  • a text image GUI widget.

In the C version, these items mapped into modules. The last item also involves a non-opaque data type.

Everyone knows that C doesn’t really have a modules system. In here, I just mean a separate compilation unit in a C file, with an accompanying header file describing its interface.

A module can implement a data type which you can instantiate and whose instances you can manipulate through functions. That is the case for the text image GUI widget.

A module can also encapsulate access to something through functions and other language features. This “something” is usually a set of data structures, but it can be anything (a collection of constants for example, or even a set of other functions). The first three items were implemented through modules like this. This is what is called an abstract object (definition from end of section 19.1 of C Programming – A Modern Approach Second Edition). If you’ve never heard of this before image it being an ADT, but with only one instance of it.

For example, the assets gateway is an abstract object. It has basically 3 kinds of functions: init, shutdown, and assets accessors. Clients of this module only need to know the functions to access the assets they need. Internally, the assets module knows which state to maintain and which functions to call in order to be an assets gateway.

The three screens are also abstract objects. However, they’re not manipulated directly. The screens are “alike” because they’re all manipulated in the same way. As explained in my older documents, whatever is the screen currently active, it has to handle events, do updates, rendering, and so forth. It doesn’t matter which of the three screens is currently active, the screen manipulation logic is the same.

This led to the following design. Each screen is an abstract object, but the only function directly exposed is the init function, which sets up the internal data structures used by the screen, and also indirectly exposes its other operations through a pack of function pointers. Indirect exposure also uses a function from the main module called register_screen.

An ID is conventionally assigned to each screen, and during its setup function, each screen will register its pack of function pointers giving its ID. Like this:

register_screen(MENU_SCREEN, my_operations);

This way, the main module can receive and store the packs, and manipulate screens by keeping a pointer to the pack of the currently active screen. To manipulate a different screen, it can simply point at a different pack of function pointers.

Here is game.h (header file for the game screen module). Notice how it only exposes its init function.

#ifndef GAME_H
#define GAME_H

#include "screens.h"

int
init_game(const struct GameContext *gx);

#endif

Here is the header file for the assets module.

#ifndef ASSETS_H
#define ASSETS_H

#include <SDL2/SDL.h>
#include <SDL2/SDL_ttf.h>

static const struct SDL_Color DEFAULT_BG_COLOR = {25, 25, 25, 255};
static const struct SDL_Color DEFAULT_FG_COLOR = {225, 225, 225, 255};
static const int LARGE_FONT_SIZE = 46;
static const int MEDIUM_FONT_SIZE = 34;
static const int SMALL_FONT_SIZE = 22;
static const int BLOCK_WIDTH = 30;
static const int BLOCK_HEIGHT = 30;
static const int NUM_SONGS = 5;

int
init_assets(SDL_Renderer *r);

TTF_Font*
get_large_font(void);

TTF_Font*
get_medium_font(void);

TTF_Font*
get_small_font(void);

void
destroy_assets(void);

SDL_Texture*
get_tetris_block_img(void);

SDL_Texture*
get_bg_img(void);

#endif

This is one of the most complicated header files I had, in pair with screens.h. Here is music.h (much simpler):

#ifndef MUSIC_H
#define MUSIC_H

int
init_music(void);

int
play_new(void);

int
destroy_music(void);

#endif

The music player module also is for an abstract object. Internally, a Mix_Music* is used to play the tetris background music.

An advantage of having abstract objects represented through modules is how simple things get. Main is also an abstract object whose purpose is to start the program, control setup and shutdown of other abstract objects, and implement/run the game loop.

Main overall:

  • init
  • game loop
  • cleanup

An evident problem with this approach is that if I ever wanted two assets gateways, or seven (for whatever reason), I’d be screwed. In this case, a tetris clone, I knew I would only need one. So there was no harm in making it an abstract object. The same is true for the music player, for each of the screens, and so forth.

C++ makes it seem like a small leap to turn modules like those into classes. So I took the bait.

The class feature in C++ is a general mechanism to represent in your program arbitrary concepts of your solution. In C++, you can use a class to do OOP, but that’s not all there is to it. OOP is one kind of solution, and classes are really a general solution-representation mechanism.

OOP, as Alan Kay said, involves message passing (which today we call method [or member function in C++] calling so it’s not confused with the concurrent programming concept of message passing), late binding (which in C++ is achieved through virtual member functions) and state retention (which in C++ is sort of difficult to achieve, but can be with discipline and the right programming practices).

In the C++ version of the tetris clone, OOP isn’t practiced very much (not any more than in the C version). The rewrite wasn’t so that I could make the code more OO. I don’t think much benefit, if any at all, will come from that.

State retention is there, but not quite. Clients of classes generally have no choice but to rely on private aspects of a class, such as which private members are there, simply because the header files exposes those aspects. This means that a client depends in the structure of the class’ private members, and thus will have to be recompiled in the case of a change in those aspects (the recompilation isn’t always necessary by the way).

Although things are not simple as I made them sound, it is true that a client will end up eventually in need of a recompilation because of a change in an implementation aspect of a class it merely has a pointer to. This is the kind of thing state retention attempts to avoid (dependency of any kind on implementation detail so that change in implementation details won’t affect client code).

C++ lets you enforce state retention to its full utility, but I’m not looking for that much in this program.

Notice that in the C code the natural way of programming modules enforced state retention well enough because I was dealing mostly with abstract objects. With the text image GUI widget, which isn’t an abstract object, similar state retention issues as in C++ appear. This way of programming (with abstract objects) can also be done in C++, but I didn’t choose it because I wanted to see how much of an improvement using classes would have caused.

I need to say that the PIMPL pattern isn’t an alternative here. The PIMPL pattern involves some form of allocation scheme because you hold a pointer to the implementation class type, which will have to be set up properly later. In all the examples I’ve seen, that is through some form of dynamic allocation with the new operator or some custom allocator that relies on new behind the scenes (but does so more efficiently). The C version doesn’t have any dynamic allocation in it (this was an implementation constraint I wanted to satisfy from the beginning), and I wanted the C++ version also like that. So, unless I learn how to apply the PIMPL pattern without dynamic allocation, it won’t be applicable in here.

Another reason why OOP isn’t much present in this program has to do with late binding. Very little of the program is actually late bound. Most calls are statically bound.

OOP isn’t fully absent though. It’s present in the game loop function, in the screen classes, and in main’s cleanup function. The game loop function in main uses screen objects through late bound method calls. Although state retention isn’t quite there in the program as a whole, that game loop function makes no assumption about the implementation details of the screen objects. This isn’t a special property of the C++ version of the program, this was also present in the C version.

So it should be clear that C++ wasn’t used for its special OOP support. The C++ version actually involves no more OOP concepts than the C version.

In C++, instead of implementing abstract objects, what I did was to have a class instead and leave to the client code to decide where to allocate the instances. Taking away decision making from classes and letting client code provide context made things more flexible. It’s a weird benefit though because it seems useful, but nowhere in the program it’s used. For example:

  • Instead of depending on a fixed single assets module, a screen receives a pointer to an Assets instance, and then could work with any Assets instance. So if you have three assets instances and want each screen to use a different on, in the C++ version, you can easily do that. Nowhere in the code this is done.
  • Client code can decide to dynamically allocate a music player (or an assets instance, or a menu screen, …) instead of using automatic storage or static storage. Nowhere in the code this is done. The old C code used static storage, and the C++ code also does the same. However, the C++ code could have used automatic storage for example.

The transition from C modules to C++ classes was the major one. It was the most time consuming one, and the one most difficult to implement in comparison. The benefits from doing so seemed pretty small for this program in particular. The benefits of being able to have many instances instead of only one is real, but not here it seems.

Error Handling

The major reason why a SDL wrapper was written has to do with error handling. A huge number of SDL calls can fail. It’s easy to forget which calls can fail and which can’t. Also, with a SDL call failing, error handling is basically:

  • clean up
  • print error message
  • exit

In the C version, I made a mess out of error handling (I’ll fix that), but even without my mess I reckon the C++ version of error handling is way more convenient to deal with. The SDL wrapper makes the underlying call, check for errors and throw an exception if there were any. A single try/catch statement exists in the main function and that’s it. In the case of an error, the catch block in main knows what to cleanup, report and how to exit.

Figuring out which functions throw and which functions don’t throw so I could annotate them with noexcept appropriately was annoying and seemed highly error prone. It was a downside. Maybe I shouldn’t be annotating functions with noexcept. I know it’s important to do so for constructors (default, copy, move, …) of types you intend to use with the standard library (e.g. you want to have a vector<T> where T is a custom type of yours). For other functions that don’t throw, it seems logical that I should annotate them with noexcept, but I’m not sure (again, highly annoying and it seems extremely error prone).

The error proneness of annotating with noexcept, for all I can tell, comes from the fact that you need to keep in sync the noexcept specifier and the implementation of the function. If the implementation code changes, you may have to change the noexcept specifier. Even though you can make your noexcept specifier dependent on arbitrary conditionals, even involving the noexcept-ness of other code, the error proneness is still there. The noexcept-ness of a function is part of its interface, which means this whole noexcept conditional specifier will have to be exposed as part of the function interface (in a header file), which means that if you depend on some call to be noexcept or not in the implementation, you might have to place that call code in the function’s declaration in the header file for noexcept’s sake. I’m not much familiar with C++’s noexcept, so I may be wrong here, but this seems a whole new level of mess. Until I learn more, I’ll stick to simpler noexcept specifier (mostly those that don’t depend on anything else, or on very simple and harmless conditionals).

Discipline in using exceptions is needed though. In all the cases, all I can do when an error happens is what I told before (clean, print, exit), so exceptions seem to apply effortlessly. In so many cases I see people using exceptions for more complicated decision making, and they don’t seem fit for that. If you have ever written Java code before, maybe you’ve came across code trying to parse integers and floats, and maybe you’ve seen how much of a mess it is dealing with errors in such code exactly because of exceptions.

In a more elaborate program, I won’t just print something to the console. The exception instances could be enriched with values known at the point of failure. Some report could be generated in main’s catch block and maybe an email could be sent to communicate the error to the development team’s. Maybe the email could be archived for later if no internet connection is available. C++’s exceptions support that approach as well.

In C, dealing with errors is less nice. It involves a combination of error codes, global variables (errno and friends), and even non-local jumps in some cases. In my C version of this tetris clone, error handling was done with:

  • error (return) codes
  • helper macros
  • a single global variable for error information

C++ is a clear winner here as far as I can tell. Exceptions didn’t make the code any harder to write. Even though the SDL wrapper was needed, it’s a one time thing, meaning I’d have to write it once and simply use it the next times. The code also wasn’t any harder to read because of exceptions. On the contrary actually the code was easier to read because all the code needed to propagate the error and perform cleanup became much simpler using exceptions.

It’s true that my error handling C code is much more complicated than it needs to be, but even if it wasn’t the C++ version would be still simpler.

RAII

The big thing I hear about C++ is RAII. It’s how class type instances can be used to control init and shutdown of various resources automatically through static scoping rules. This is a very attractive idea, and even though I make use of RAII, I don’t think it made this particular code any better (or any worse).

Operator Overloading

There are two uses of operator overloading that I found particularly interesting. One is operator == and operator != for the SDL_Color type. The other is operator= for the std::array type. Mainly the second one.

This second one replaced a few calls to memcpy that existed in the code. It made the code significantly more readable in those particular sections (i.e. the functions that used to call memcpy and now simply run an assignment).

Copying and Moving

C++ introduces language mechanisms supporting moving and copying. They are pretty straightforward to deal with those for simpler types (e.g. Scores), but for some cases I’ve found much easier to simply use C++11’s delete feature to disable moving and/or copying. For all I can tell, this is perfectly fine. Not all types are supposed to be copyable, and moving seems more like an optimization possibility for copyable types, but it can be used for non-copyable types (and it is useful to do so in some cases).

Deleting these seems specially fine for types which represent IO concepts like classes for the screens, for the music player, for the assets gateway. Moving can make sense for those. For example, it’d allow for a function which returns a new instance of a class, and I have implemented moving for some types, but it was mostly as a curiosity thing. Generally, I don’t make much use of copying and moving for the custom types I’ve created.

In the process of deleting the copy constructors and assignment operators, I was generally fine with the decision of doing so. However, for moving, it was clear to me that in most cases (if not all) there was a plausible implementation for move. Implementing it though would have been a waste of time since I didn’t see the need to use it anywhere in the program. This was apparent based on the C implementation, and it became clearer as I dug deeper into the C++ rewrite.

Copy and move aren’t completely absent though. For example, the Scores type is pretty much a traditional value type “like int”, and it behaves so nicely that the defaults (copy, move, and destruct) work just fine for it. However, it’s good to point out that this doesn’t contradict what was said earlier: move and copy constructor/assignment isn’t used in this program for the custom types I’ve implemented. Even though the Scores type have it, it isn’t used by this program.

The Manual “Hack”

There is a class template called Manual. It serves to let you manually call the destructor/constructor in case you want to delay when the constructor is called or hurry up when the destructor gets called. This template is used throughout the program.

In many cases, I found myself wanting to delay initialization (see the Scores class), keep the storage after deletion for reutilizing later for a newer instance (also see the Scores class). In some other cases, my problem was that I couldn’t find a way to use initialization lists to get the members to be initialized. The screen classes also use the Manual template.

The reason why I’m calling it a hack is because I can’t help myself but to think this is all because I’m not structuring my solution into classes that play well with the C++’s way of doing things.

I feel like it’s important to point out that using Manual<T> instead of just T didn’t make the code any more difficult to understand as far as I could tell.

In many cases (if not all), dynamic allocation could have been used instead of the Manual template. Reutilization of storage could be done with placement new, and deletion without deallocation could be done by calling the destructor directly. There are two reasons why I didn’t do this:

  • I wanted to avoid dynamic allocation as an implementation constraint I imposed upon this project.
  • It seems like a much more error prone approach than using Manual.

In C, life cycle is always manually controlled.

Looking Back and Ahead

Looking back from now, what I’d like to see is what would happen if I only used C++ for error handling support, which was where there was a big win. Instead of trying that however, what I’ll probably do is adjust the C version.

During the C++ rewrite, I’ve found places in which the C version was way more complicated than what it needed to be. I also found bugs in it. My next step is dealing with those issues. Then, I’ll polish this project’s documents, and then move to another game.

In general, I think the code became better. I think it could become even better if I knew more about C++. But in many cases, advantages are clear. For example, one thing I didn’t mention much before was the use of virtual functions and purely abstract base classes. In large projects, these seem much better tools than structs of function pointers, specially because packs of function pointers won’t always cut it.

C++ seem to introduce many interesting ideas through language features, and although the ideas are interesting, the language features aren’t nearly as so much in my opinion.

The Major Benefit of It All

The major benefit wasn’t the rewrite. It was the review needed for the rewrite. During this review I’ve found bugs in the program, unnecessary complexity in my error handling code, design flaws and so many other little things to fix.

For example, there were many SDL functions that I thought couldn’t fail that actually can. In the middle of the C++ rewrite, I decided to take a look at those (for other reasons besides error handling) and noticed that SDL_RenderCopy can fail. This was when it hit me that maybe I wasn’t checking for errors in all my SDL calls. This led to extending my SDL wrapper to add many more entries than it had, and also led to a reminder that I should fix my C code.

In the end, the benefit wasn’t rewriting it in C++. It was taking a second look at the code. As someone else said out there, “Writing is rewriting.” and I think this quote is highly applicable here.

Address and Thread Sanitizer in GCC

Tags

, , ,

I didn’t know, but GCC has address and thread sanitizers.

Check out this video:

And also this post http://developerblog.redhat.com/2014/12/02/address-and-thread-sanitizers-gcc/.

The GCC options are -fsanitize=address and -fsanitize=thread (https://gcc.gnu.org/onlinedocs/gcc-4.8.2/gcc/Debugging-Options.html). I had to link libasan (-lasan flag) to use it.

Guess what? I’ve just found a bug in my code.

Tetris in C and SDL2 – Retrospective – Implementation

This is a follow up on Tetris in C and SDL2 – Retrospective – Specs, but here I talk about implementation aspects.

Implementation Aspects

The implementation of the game wasn’t that complicated. It’s a C program using the SDL2 library. Being a C program usually means you’ll have to worry about details you usually don’t worry about by programming in another language.

SDL2

SDL2 makes a ton of things really convenient. For example, the following items were trivial to be dealt with because of how SDL2 deals with:

  • Keyboard input
  • Rendering
  • Timing

These items above weren’t complicated according to the games spec, and SDL2 made it so that they weren’t complicated in implementation either. That’s a big win. SDL2 didn’t make the system more complicated than it seemed it would have to be. Using OpenGL for example would have made rendering be a bit more complicated than what it seems it had to be because of how much setup code that ends up getting written with OpenGL. In general, SDL2 got out of the way and it let me do what I wanted to do, except in one case, which I’ll talk about later.

Modules

These are the major modules in the game:

  • main
  • menu screen
  • scores screen
  • game screen
  • assets
  • music

The key to understanding the overall modules structure is the screens structure. Each screen should have these functions: init, focus, update, handle_event, render, and destroy. Main calls the init function of every screen module. This allows the module to tell main what are its other 5 functions. Main doesn’t know those out of the box, it has to be told what they are. This is done through a function exposed by the main module called register_screen. Here is an example of calling register_screen from the init_game function:

self = (struct ScreenObject) {
  .focus = focus,
  .render = render,
  .update = update,
  .handle_event = handle_event,
  .destroy = destroy
};
register_screen(GAME_SCREEN, &self);

Main knows there are only 3 screens (3 is actually a program constant), and that each has an integer ID (also program constants). Calling register_screen tells main which is the bag of (the 5) function pointers for a given screen ID. The type struct ScreenObject serves to pack together pointers to screen functions, except for the screen’s init function, because it isn’t needed during game play. Before calling the init function of each screen module (init_game, init_scores, init_menu), main will setup the other parts of the program: SDL2, SDL2_ttf, SDL2_image, SDL2_mixer, assets module, music module, a window, a renderer, etc. By the time a screen init function is called, all of those other parts were already initialized. After all the setup is done, if no error occured, the main module enters into the game loop. The game loop goes pretty much like this:

  • Initially, current screen is MENU.
  • Call current screen’s focus function.
  • LOOP:
    • Get events.
    • Send events to current screen’s handler function.
    • Transition to another screen if needed (call focus on transition).
    • Call current screen’s update function.
    • Transition to another screen if needed (call focus on transition).
    • Call current screen’s render function.

Transitioning to another screen happens when handle_event or update tells main (through a sentinel return value) to leave itself and go to another screen. When a transition happens, the new screen’s focus function is called. The game loop function follows. It relies on a bunch of module variables (static file scope variables in main.c) that main uses to do its job. For more on it, check main.c in the source tree.

static int
game_loop(void) {
  SDL_Event e;

  xassert(current);

  current->focus(&gx);
  play_new();

  enum ScreenId s = SELF;
  int err = 0;

  for (;;) {
    // Events
    while (SDL_PollEvent(&e)) {
      if (e.type == SDL_QUIT) {
        return 0;
      }
      s = current->handle_event(&gx, &e);
      if (s == ERROR) {
        return -1;
      }
      else if (s != SELF) {
        break;
      }
    }

    // Screen Update
    if (s == SELF) {
      s = current->update(&gx);
      if (s == ERROR) {
        return -1;
      }
    }
    
    // Screen Change?
    if (s != SELF) {
      xassert(s < (int)NUM_SCREENS);
      xassert(s >= 0);
      current = all_screens[s];
      err = current->focus(&gx);
      if (err < 0) {
        return err;
      }
      s = SELF;
      play_new();
    }

    // Render.
    err = render();
    if (err < 0) {
      return err;
    }
  }
  return 0;
}

It’s also interesting to take a look at main’s cleanup and render functions because they also make use of screen functions.

static void
cleanup(void) {
  int i;

  for (i = 0; i < NUM_SCREENS; i++) {
    if (all_screens[i]->destroy(&gx) < 0) {
      fprintf(stderr, "Error while destroying screen: %d.\n", i);
      report_error();
    }
  }
  destroy_video();
  TTF_Quit();
  IMG_Quit();
  SDL_Quit();
}

static int
render(void) {
  SDL_Texture *bg = get_bg_img();
  SDL_RenderCopy(gx.r, bg, 0, 0);
  int err = current->render(&gx);
  if (err < 0) {
    return err;
  }
  SDL_RenderPresent(gx.r);
  return 0;
}

Main has a helper procedure to initialize screens. It has the init functions hard coded in it. It also makes use of the screen functions, but it’s in a specialized manner for setup/shutdown.

static int
init_screens(void) {
  COND_ERROR(init_menu(&gx) == 0, e_bad_menu);
  COND_ERROR(init_game(&gx) == 0, e_bad_game);
  COND_ERROR(init_scores(&gx) == 0, e_bad_scores);

  current = all_screens[MENU_SCREEN];
  return 0;

  e_bad_scores:
  all_screens[GAME_SCREEN]->destroy(&gx);
  e_bad_game:
  all_screens[MENU_SCREEN]->destroy(&gx);
  e_bad_menu:
  return -1;
}

This function exposes a little of error handling in the program, which is the next topic. But before that, here is an example of a screen init function and of a focus funcion. From game.c, focus:

static int
focus(const struct GameContext *gx) {
  (void) gx;

  last_update_ms = SDL_GetTicks();
  fall_delay_ms = INIT_FALL_DELAY_MS;

  // On focus, a new game should be started.
  empty_blocks();
  reset_piece();
  score.points = 0;
  int err = refresh_points_text(gx->r);
  if (err < 0) {
    return err;
  }
  update_next_piece();
  return 0;
}

From menu.c, init_menu:

int
init_menu(const struct GameContext *gx) {
  TTF_Font *title_font, *button_font;

  menu.within = gx->dim;
  title_font = get_large_font();
  button_font = get_medium_font();

  COND_ERROR(
    init_text_image(&menu.title, title_font, "Tetris", gx->r) == 0,
    e_bad_title_text);
  menu.title.pos.x = hor_center_within(&menu.title.dim, &menu.within);
  menu.title.pos.y = TOP_TITLE_OFFSET;

  COND_ERROR(
    init_text_image(&menu.new_game, button_font, "New Game", gx->r) == 0,
    e_bad_new_game_text);
  menu.new_game.pos.x = hor_center_within(&menu.new_game.dim, &menu.within);
  menu.new_game.pos.y = TOP_MENU_OFFSET;

  COND_ERROR(
    init_text_image(&menu.scores, button_font, "High Scores", gx->r) == 0,
    e_bad_scores_text);
  menu.scores.pos.x = hor_center_within(&menu.scores.dim, &menu.within);
  menu.scores.pos.y = TOP_MENU_OFFSET + menu.new_game.dim.h;

  self = (struct ScreenObject) {
  .destroy = destroy,
  .handle_event = handle_event,
  .update = update,
  .render = render,
  .focus = focus
  };
  register_screen(MENU_SCREEN, &self);

  return 0;

  e_bad_scores_text:
  SDL_DestroyTexture(menu.new_game.image);
  e_bad_new_game_text:
  SDL_DestroyTexture(menu.title.image);
  e_bad_title_text:
  return -1;
}

Error Handling

Right away, one major topics in doing a C programming is error handling. The reaction to an error in this game is basically print an error message and quit. Errors were usually from procedures outside the control of the game, like initializing SDL or loading an image. Here is an example of error handling reasoning done in the game. If the tetris image for a block cannot be loaded, then the game will quit with an error. The error can be avoided by having the block image be rendered through a procedure that takes an SDL_Renderer* and draws the block. This approach is harmful for many reasons:

  • It means reprogramming a function whenever the looks of the tetris block is changed.
  • It also means that visualizing off game the looks of the image would be more difficult.
  • It also means experimenting with different visuals for the image would be considerably more work.
  • There would be no instant feedback of how the image looks as you implement it.

So even though the error of not being able to load an image could have been fully avoided, it wasn’t. The C language provides very little support for error handling. So what ends up happening is that you come up with your own techniques for it. In this program, error handling is basically done through these means:

  • COND_ERROR macro
  • COND_ERROR_SET macro
  • errorFn global function pointer
  • returning sentinel values indicating errors
  • sequences of labels marking statements that cascade for clean up

Here is a code (from main.c) which involves all of those:

static int
init(void) {
  gx.dim.w = WIN_WIDTH;
  gx.dim.h = WIN_HEIGHT;

  COND_ERROR_SET(SDL_Init(SDL_INIT_VIDEO | SDL_INIT_AUDIO) == 0, e_bad_sdl,
    SDL_GetError);
  COND_ERROR(init_video() == 0, e_bad_video);
  COND_ERROR_SET(TTF_Init() == 0, e_bad_ttf, TTF_GetError);
  COND_ERROR_SET((IMG_Init(IMG_INIT_PNG) & IMG_INIT_PNG) == IMG_INIT_PNG,
    e_bad_img, IMG_GetError);
  COND_ERROR(init_assets(gx.r) == 0, e_bad_assets);
  COND_ERROR(init_music() == 0, e_bad_music);
  COND_ERROR(init_screens() == 0, e_bad_screens);

  return 0;

  e_bad_screens:
  destroy_music();
  e_bad_music:
  destroy_assets();
  e_bad_assets:
  IMG_Quit();
  e_bad_img:
  TTF_Quit();
  e_bad_ttf:
  destroy_video();
  e_bad_video:
  SDL_Quit();
  e_bad_sdl:
  return -1;
}

The COND_ERROR macro will check if a condition that is given to it is true. If it isn’t, then it’ll go to the label given as the second parameter. The COND_ERROR_SET macro will do the same, but it’ll set errorFn to its third parameter right before going to the given label. Functions usually return a negative int for error and 0 for success. Sometimes null for error and not null for success. The labels are setup in such a way that cascading makes for proper clean up in reverse order in which the items were initialized. For example, SDL_Quit is called last and SDL_Init is called first. This ordering allows for setups and shutdowns that depend on each other. The combination of these features allows for local error handling somewhat conveniently. It’s problematic mostly because the reverse ordering isn’t enforced and because the clean up code is in a different section of the function, and not right after its test condition. In general, it’s a very error prone mechanism. As it turned out, error handling for initialization code wasn’t an issue. It was generally pretty straightforward to do so. What was mostly annoying was error handling in the middle of game logic. For example, to update the scores text when the player scored, a new image is created. This operation can fail, and as a result of that, every update of the game can potentially fail.

Assets and Music

The decision to have a module dedicated to assets loading and providing made things very easy. The assets module simply has an initialization function that loads up all the assets, constructs the fonts and so forth. It then provides a bunch of functions for other modules to get those assets. For the music module, it works as a very simple player that can initialize, shutdown, and replay the tetris theme song.

Modules

The modules structure turned out to work pretty well. It could be argued that the game screen module does too much. Splitting it into pieces could scatter the core game logic into different places, which I don’t think is very helpful for something small like tetris, but there are splits that could have been done which wouldn’t cause that problem. For example, there could be a module for manipulating a blocks panel. It could have operations like, clear_full_lines, add_block, has_block_at, render_block_panel, and stuff like that. Such module would simplify the game screen module without scattering the core tetris game logic into multiple places. Besides this issue with game.c, the modules structure seems fine. It’s surprising how effective file scope static names can be. Leaving dynamic dispatch on the side, a module can be though of as behaving like a single object with local aids (the module local names implemented through static file scope things).

C and C++

The game was developed fully in C. I wonder how much simpler (or complex) it would become by having it in C++. I’ll probably re-write the game in C++ to the best of my knowledge to see if I can make it better.

Tetris in C and SDL2 – Retrospective – Specs

Tags

,

Last weekend, I finished a first version of a tetris clone I was writing.

Before writing any code, I wrote a small document called little_specs.txt. It was something like a small requirements document telling some (very small number) of characteristics the game should have:

- Three screens: game, scores, menu.
- Menu screen leads you to one of the other two screens.
- Scores screen tells you the scores or that there aren't any.
There will also be an arrow for going back.
- Game screen is for playing the game. The game shouldn't start
right away.
- Before starting, game asks player to "click the screen" and then
game will start.
- There will be the main panel for the tetris game and a 'next
piece' on the side.
- Pieces will come at random.
- <space> flips a piece.
- up, down, left, right are used to move a piece.
- Two fast paced <down> hits will drop the piece (make it go as
fast as possible downwards).
- On top of the tetris panel, there will be your current points
and a max score on the side for you to compare with.
- A score is a combination of in-game points and time to complete
those points.
- Data structures are organized per screen. There are MainScreen,
GameScreen and ScoreScreen. They should be self contained in the
sense that all computing going on while in the score screen should
only use data in the DS ScoreScreen.

Note: This is a rough spec about the code and the game that was
written before any code. It's being done done in half an hour or
so and it won't be changed after now. This will serve as a
comparison point for the end of this first stage of the game. For
more information, see the code.

Point-by-Point Analysis of little_specs.txt

The game has three screens as planned, and indeed the menu screen leads you to the other two screens. The scores screen behaves differently than what was predicted. When no scores were recorded, then it’s pretty much a ‘blank’ panel. To go back, you don’t have an arrow image, you hit ESC. The fact that there is no text for when there are no recorded scores is a missing-feature bug.

The game screen is for playing the game. The game does start right away, which is another missing-feature bug. The next piece on the side was implemented and it works just fine. Pieces do come at random, but it’s the up arrow key that flips the piece, which seems way better than having the space bar do it. Moving the piece up was a bug in the specification. That’s not something you can do in tetris games.

The two fast hits to the down arrow key doesn’t make it drop. Holding the down arrow key will make it go faster though. This seems like a better feature to have than having the drop. But having the drop instead would make the game harder it seems. I’ll leave this as is.

The game screen doesn’t show you the max score for you to compare with. That is a bug (missing-feature).

The specification had something to say about the data structures used in the game. This is another bug in the specs. It shouldn’t be talking about such matters. It should only talk about characteristics perceived by the player.

Current Specs

While little_specs.txt was a rough requirements document. This, however is a report on what characteristics the game actually has. It’s a document built by analyzing what was really built.

  • There are three screens: MENU, SCORES and GAME.
  • At the beginning of the game or right after the transition from a screen to another, the tetris music should re-start.
  • Music plays in an infinite loop.
  • Theme song is the classic tetris theme song.
  • All screens have a static simple black and white pixelart-ish stars background.
  • The game starts at the MENU screen.
  • To quit the game, you hit the X button on your window. Some underlying shortcut can also be used (like ALT+F4 on linux/xfce4). There is no in-game sure-fire way to close the game (like hitting ESC on the MENU screen).
  • The MENU screen has a title and two buttons to lead each to the other two screens.
  • The SCORES screen shows the recorded high scores of the session (from game beginning to closing the game).
  • Scores aren’t remembered through game sessions.
  • When there aren’t any scores, the SCORES screen will only show the background, a title and the help text telling the player how to go back.
  • Hitting ESC goes back form the SCORES screen to the MENU screen.
  • Only the top 5 scores are shown.
  • During the game play, pieces do seen to come randomly. Sometimes it feels like the game is against you, but it also feels like it’s helping you some other times.
  • There is nothing before hitting “new game” and the game starting. The player doesn’t have to click anywhere or do anything. The game starts right after the player going into the GAME screen.
  • When the player loses, the game just goes back to the MENU screen. There is no “Game Over” or no indication that your score is now in the top 5, or that it’s a new global record.
  • There is no animation or sound effect when a line is cleared during the game.
  • You can control the piece with up, down, left and right. Using the up arrow key will flip the piece. Using the down arrow key will make it move faster downwards, but it won’t drop it right away.
  • If you clear more lines at once, there are more points to be made. If you clear those lines higher in the pieces panel, then yet more points are to be made: clearing 1 and the first line makes you 1 point; clearing 4 lines above at once will give you 96 points.
    • Clearing N lines: N*2^(N-1)*P where P is the position modifier.
    • For bottom, P is 1; middle, P=2; above, P=3.
    • So for N=3 in the middle: 3*2^2*2 = 3*8 = 24.

Some of the emphasis on this list is bringing into the foreground things I think were a mistake and things I think were done well. For example:

Mistakes

  • No animation or sound effect when clearing lines.
  • Static background. It should be animated.
  • Game starts without prompting the player. No indication of how the particular play affected your scores.
  • Nothing but going back to the MENU screen happens when you lose.
  • Scores aren’t persisted.
  • Soundtrack restarts at the beginning of every screen.
  • No levels. The game doesn’t get more difficult in any way.

Rights

  • Points system. It encourages you to try going for 4 lines at once above the middle ground. It makes the game riskier (more interesting then).
  • Down button doesn’t drop the piece right away. This is a deviation from little_specs.txt, but I think this turned out to be a better thing to do.

The rights also include all the places in which the actual spec was in sync with what a tetris game should do and with what was expected from the game in the first place (see little_specs.txt). For example, it’s a right that when you have a filled row, it disappears and you score. But it’s not useful specifying all of those cases because then this list would be huge without adding any value to this document.

Things I’m not sure If They were Rights or Wrongs

  • In no point, the scoring system is explained to the player. This gives an advantage to the more observant player (that sees way more points being made by clearing more rows). It also adds a “secondary puzzle”, which is breaking down the rules for scoring.
  • Only 5, and not some other number of top scores, like 10, or 3.
  • No speeding up as the game progress.

Difficulty in Playing Tetris

One of the ways in which I said I made a mistake was:

“No levels. The game doesn’t get more difficult in any way.”

Also, a characteristic which I don’t know if I did it right or not was:

“No speeding up as the game progress.”

These may seem conflicting, but they’re actually not. There are different ways to make tetris more difficult:

  • Restricting how many times a piece may be flipped (3 for most difficulty).
  • Increasing the speed in which a piece falls.
  • Increasing the “cooldown” of a flip action (as of now, you can flip as much and as fast as you can hit the up arrow key).
  • Slowing how fast you can move the piece horizontally.
  • Increasing how fast the piece falls when you hit the downwards arrow key.
  • Having an AI that picks pieces that will purposely hurt your game.
  • Dropping blocks which can’t be cleared.

And many more…

Tetris Clone in C and SDL2

Tags

, , ,

Two weeks ago, I’ve posted about a rough version of a tetris clone I was building. Last week I finished it:

The code is on github and there is a youtube video of it.

Here is the video (same as in the tweet):

These updates happen weekly because I only work on my C and SDL2 game development during the weekends. In the last weekend, I’ve finished the fist version of the tetris clone. This weekend, I decided to write some retrospective text to help me thing think about what I did (this almost sound like I’m grounded =D). It’ll probably span many posts, but I’ll link them all in here:

The retrospective texts are still incomplete. I’ll keep tweaking them and posting new ones next weekend.

Microbenchmark for Cache Friendliness Benefits

Tags

, , , , ,

I’ve always heard of how being (CPU) cache friendly pays off. It always made sense to me, which isn’t a surprise given it’s a somewhat easy concept to grasp in theory (applying it in practice tends to be a little bit more complicated though). However, I’ve never had the chance to compare a cache friendly version of a code with a cache unfriendly version to see the differences. This is mostly due to the fact of me not being really someone who works on performance intensive code.

I decided to write a microbenchmark to see this in practice. I basically go through an array looking for an element. I have a cache friendly version of it and a cache unfriendly one.

This code uses the SDL2 library for timing functions so I can measure how long things take. Here is the C++ code.

http://paste.ubuntu.com/9052389/

Running it often gives me running times which are about 20 times faster for the cache friendly version:

FIND1: 0.0142819 secs.
FIND2: 0.302028 secs.
RATIO (delta1/delta2): 0.0472868.
RATIO (delta2/delta1): 21.1476.

Interestingly, the difference is also visible in javascript code:

http://paste.ubuntu.com/9052441/

Populate: 0.757 secs.
D1: 0.026 secs.
D2: 0.137 secs.
Ration D1/D2: 0.1897810218978102
Ration D2/D1: 5.269230769230769

In there though I only get ~5 times faster for the cache friendly version. Using a Float64Array instead of play Array makes population times way faster (~0.03s) but the other numbers remain somewhat similar. Here is a result I got from changing Array to Float64Array:

Populate: 0.033 secs.
D1: 0.026 secs.
D2: 0.127 secs.
Ration D1/D2: 0.2047244094488189
Ration D2/D1: 4.884615384615385

Tetris clone in SDL2 and C

Tags

, , ,

I’ve started developing a tetris clone in SDL2 and C to practice both technologies and also game development.

The idea is to get some rough version going and then enhance with sounds and the particles-ish simulation I did last week.

Check it out, it’s on github:

What is in there, I did from sunday night to about monday 5 am, so don’t expect much. It currently only opens the game and shows the menu screen.

After finishing the rough version, I’ll write some sort of report talking about the development. It seems like it’ll be an interesting experience.

jsrinth – Labyrinth Generator for JS

Tags

, , ,

I’ve written a simple labyrinth generator for JS.

https://github.com/phao/jsrinth

It allows you to specify the width and height of the labyrinth, and also a callback to be called for each wall that is going to be removed in the construction of the labyrinth. If you imagine the labyrinth being constructed by removing walls from a grid of square-ish cells, the callback will be called for each removed wall with the cell x,y position and an identifier for the removed wall (‘left’, ‘right’, ‘up’ or ‘down’).

Some example of labyrinths generated by it.

The code isn’t much fast. It takes 1.021s in my machine to generate a 100×100 labyrinth using nodejs v0.10.25, but it’s basically Prim’s algorithm for computing minimum spanning trees so it can be optimized as such or be replaced with anything roughly equivalent. Maybe I’ll try to improve its performance later, but I won’t need to for all I can tell. My intention is to use this for simple games, in which case I’ll be generating the labyrinth in the beginning of a game stage (for example) and I could actually put a web worker to do it in order to hide the latency.

Learning SDL – Second weekend

Tags

, , , , , , ,

I recently started studying SDL2 (in C) on the weekends. On the first weekend, I started reading lazyfoo’s tutorials and didn’t do much more than just following through what was there.

In the second week, as suggested by Icculus (Ryan Gordon) I did a simple pong clone on Saturday, and on Sunday I decided to do something I did back when I was learning a bit of OpenGL, which was a simple particle-ish thing.

Both projects are on github:

Icculus actually twitted about my pong clone (=D), which got me some stars on github.

Some videos of the two apps:

If you don’t know who Ryan Gordon is (Icculus), you should look for his talks on youtube (https://www.youtube.com/results?search_query=icculus and also https://www.youtube.com/results?search_query=ryan+gordon). They’re very (very!) interesting to watch.

Follow

Get every new post delivered to your Inbox.