Ran into an issue which I posted about first over here:
http://drupal.stackexchange.com/questions/177519/route-parameter-upcasting-and-route-access
Finally realized that having the path as a quoted string was not working. But now, why is it that File Entity and now my custom module has to have non quoted strings as paths for parameter upcasting to work? Is this a core problem or did I miss something? The node module has all of its paths quoted, with parameters. If there is a convention or reason to it I would like to know.