Any way to speed up the execution of the game?

[smnai]

Hi,
I'm importing the game as a custom environment for training an agent, but the game execution is at human speed, and the _MAX_SPEED parameter cannot be changed to speed up the game. Is there any way to speed up the execution of the game in order to reduce training times?

Thanks for releasing this amazing code!
Saman

[dmely]

Hi Saman,

Thank you for your kind comment.

Just to make sure we are on the same page: _MAX_SPEED is the maximum allowed speed of the ball, and is surrounded by a lot of warnings not to change it because the game engine was developed (including heuristics that speed up the resolution of game dynamics) assuming the ball moves at a speed equal to, or slower than, its current value (2).

As for reducing the execution time, are you currently displaying each step as an image, or getting the resulting state in sparse form (setting argument return_state_as_image to False for BreakoutEngine)? The former might affect performance adversely.

Best,
David

[smnai]

Thanks for your reply David!

Actually I'm planning on doing both: training both a DQN agent on game images, so setting return_state_as_image to True, and another agent that uses the sparse form representation of the game states. I modified the code to register it as a custom openAI gym environment.

So far I only started training the DQN agent: I disabled all rendering but kept return_state_as_image=True, so the agent takes in game frames as input. However I noticed that training is much slower than for the Breakout-v0 environment in openAI gym. It seems like the game is actually being played at human-level speed. I'm wondering if there is a way to speed up the velocity of the ball and paddle so that the agent can take more steps in a given amount of time, which would speed up training. As you mentioned, _MAX_SPEED, which controls the speed of the ball, can't be changed, so I'm wondering if there is some other way to speed up execution.

Thanks again for your reply, this is really helpful,
Saman

[dmely]

Hi Saman, no problem.

I see. You could try increasing the ball speed (via _MAX_SPEED) and paddle speed as well, but you would have to revisit places in the game engine where heuristics were made, that possibly assumed _MAX_SPEED = 2, and change them accordingly.

As a disclaimer, I have not looked at this code in a while, but I can point you to a few places (probably an incomplete list):

BreakoutEngine._resolve_ball_physics

The way collision checking works is by looking at the tentative position of the ball at the next time step given its current position and velocity: x+dx, y+dy (not the actual variable names, just convenience notation for the purpose of this message), and checking if there's anything there.

The problem with that of course is that we do not check if the ball's trajectory intersects with any obstacle, just whether the end point of its trajectory does (thus allowing the ball to "tunnel through" an obstacle if it is thin enough).

This is an OK heuristic if all objects are thicker than the maximal amount of distance a ball can travel per time step, or, in equivalent terms, if the ball is slow enough that the largest distance it can travel in a time step is smaller than the thinnest obstacle in the game. Thus, if the maximal speed of the ball increases, you would either

1. Have to make all objects thicker for the heuristic to remain valid, or
2. Change collision resolution yourself to check whether any part of the tentative trajectory of the ball (not just the end point) intersects with an obstacle, and resolve the event accordingly.

BreakoutEngine.index_to_velocity

This is used mostly for randomizing the ball velocity in stochastic games.

BreakoutEngine.update_paddle_position

This should be fine after you change the paddle velocity, but worth double-checking.

Good luck!
David

[smnai]

Many thanks for this, this is really helpful!

I suspected something similar to what you suggested, but this definitely helps clarify how the whole thing works.
I looked at the code again and did some tests, and turns out that the reason for training being so slow is actually the time it takes to run a single step in the game, rather than the speed of the ball and paddle. It seems that the environment is much slower than gym's own Breakout-v0 implementation: on my machine, one call to env.step(env.action_space.sample()) in Breakout-v0 takes on average 0.0006 seconds to execute, while the same call on the modified Schema Breakout takes on average 0.029 seconds (both are averages over 300 steps). Disabling return_state_as_image and returning the game states as objects instead speeds up execution, but not by much (0.023 seconds per step).

Any idea why this environment might be so much slower, and if anything could be done to speed it up?

Thanks again for taking the time to write the answer above!
Saman

Edit: actually it seems like a lot of the speed loss is due to the _resolve_ball_physics function in core.BreakoutEngine (by commenting that out each step takes only 0.0045 on average). I guess that's due to Python's speed limitation's vs ALE/Stella's C++ and that not much can be done about it...