No obvious way to await till idle

[Dreamsorcerer]

I'm completely lost as to how I would write await/async code within a Gtk Application. The example in the README is too short to be of any use, and those in examples/ are also somewhat lacking and very out-of-date.

Here's roughly what I have so far:

async def fetch(session, id):
    async with session.get(URL.format(id)) as resp:
        return ET.fromstring(await resp.text())

async def update_things(callback):
    async with aiohttp.ClientSession() as session:
        for r in asyncio.as_completed(tuple(fetch(session, id) for id in THINGS)):
            xml = await r

            # Process data...

            game_callback(ratio)

class AppWindow(Gtk.ApplicationWindow):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

        self.progressbar = Gtk.ProgressBar()
        self.progressbar.show()
        self.add(self.progressbar)

        await update_things(self.update)
        self.progressbar.hide()

    def update(self, fraction):
        self.progressbar.set_fraction(fraction)

class Application(Gtk.Application):
    window = None

    def __init__(self, *args, **kwargs):
        super().__init__(*args, application_id="foo", **kwargs)

    def do_activate(self):
        if not self.window:
            self.window = AppWindow(application=self, title="Foo")
        self.window.present()

if __name__ == "__main__":
    gbulb.install(gtk=True)
    loop = asyncio.get_event_loop()
    loop.run_forever(Application(), sys.argv)

But, I can't use the await update_things() line, as the init function is not async. The closest thing I've been able to find, is changing that to asyncio.ensure_future(update_things()), but the issue is that there doesn't seem to be a way to pause the current code like await does, in order to run the progressbar.hide() line after that coroutine has completed.

I've also tried loop.run_until_complete() and similar functions, but it doesn't seem possible to run another coroutine that way either.

[nhoad]

How much experience do you have with vanilla asyncio?

Also, if you could tell me how the examples are out of date or not useful, that would be good.

[Dreamsorcerer]

Not a huge amount, enough to get my head around the basic concepts, but struggling to understand how this works when a GtkApplication is already taking the main coroutine.

I'm also seeing some potential unresponsiveness, for example if the # Process data... part takes a little while, which I've tested by adding a blocking time.sleep(1), then the GTK application doesn't seem to refresh/update until that entire loop has finished. I expected the await line at the top of the loop to give the application a chance to update the window during processing.
i.e. It seems like if there's work to do, it prioritises that work immediately over updating GTK. I would expect that if the last GTK update was >100ms or similar, then it would prioritise a GTK update before returning to the workload.
If that block were to take 0.1 seconds, but it was iterating over 1000 objects, then the application is left completely unresponsive for 100 seconds with the current behaviour, whereas I would expect it to never be unresponsive for more than 0.1-0.2 seconds.

[stuaxo]

You need to use asyncio.sleep instead of time.sleep, so that the loop can run.

[Dreamsorcerer]

@stuaxo I think you misunderstood. I'm using the time.sleep() to simulate a workload. If there was a real workload, it would not allow the loop to run in the middle of the work.

My point is that if I add time.sleep(1) into that block, and have 100 items in the loop, then the GTK window won't get updated for the next 100 seconds. I would expect it to get updated roughly every second, as there is an await line at the top of the loop being run before each time.sleep() call. My understanding is that an await line returns control to the loop and gives it chance to run some other code.

A GTK event loop should surely prioritise GUI responsiveness over background work, but it seems the opposite is true.

[Dreamsorcerer]

OK, I've dug into it a little more. Seems like I was wrong about await returning control to the event loop, it seems that is only done after a yield in a coroutine.

After looking at some of the code, I've managed to create a function based off asyncio.sleep():

async def async_break():
    loop = asyncio.get_event_loop()
    future = loop.create_future()
    h = loop.call_later(0, asyncio.futures._set_result_unless_cancelled,
                        future, None)
    try:
        return await future
    finally:
        h.cancel()

Then adding an await async_break() into the block returns control to the event loop and schedules it to run when idle again (because of the 0 delay).

However, it doesn't work if you do a bare yield, which is what asyncio.sleep() does if you specify a delay of 0. The equivalent code would look like:

@types.coroutine
def __sleep0():
    yield

async def async_break():
    await __sleep0()

So, when this yield happens which should return control to the event loop, it's not waiting till idle to run it again. If it was possible to have the event loop reschedule until idle when yielded, then users can do a simple await asyncio.sleep(0) in order to ensure an application remains responsive.

[Dreamsorcerer]

So, I'm not sure how the previous issue could be fixed, as I don't understand how the Gio.Application.run() method works within Python.

One other minor problem I see is the call_soon() method doesn't wait until idle, potentially resulting in some similar unresponsiveness. This is because the code changes the GLib.Idle priority. I see no reason for the priority to be changed (and even if that was the desired behaviour, why not use GLib.Timeout instead?). Removing this line would fix that minor issue: https://github.com/beeware/gbulb/blob/master/src/gbulb/glib_events.py#L865

[nhoad]

I'm sorry, but I'm not maintaining gbulb anymore. I don't have time to work on it anymore due to my personal life.

If you'd like to discuss ownership, please refer to #32.

[ldo]

It seems to me your __init__ method should create a task on the event loop to run the update_things() method, and return without waiting for it to complete. update_things() can also take care of calling progressbar.hide().

One little caveat is you need to keep a strong reference to the created task somewhere until it completes. asyncio itself only keeps weak references to tasks.

I think I have something similar to what you are trying to do in my animsort example here.

[Dreamsorcerer]

Actually I slipped up in that example, as __init__() is not async. I do create a new task as you mentioned in my actual implementation.

This makes no difference, as the event loop will not update the widgets when you do a simple yield, as mentioned in my more recent comment: #44 (comment)

So, doing await asyncio.sleep(0) yields back to the event loop, and the event loop then immediately jumps back into the same task, without updating any widgets. I've had to hack around this by using the function at the beginning of that comment (which is basically what asyncio.sleep() does with a non-zero value.

[stuaxo]

Is it not updating widgets because there are no Gtk events pending ?

[Dreamsorcerer]

I'm not sure what you mean, the only difference between the code is whether it yields or awaits on a call_later() in a busy loop.

Here's a complete reproducer. If you run this, you'll see a working progress bar. If you switch to one of the sleep 0 lines which are commented out, then you'll find the progress bar is not displayed/updated until the loop has completed.

import asyncio
import gbulb
import time
import types
from gi.repository import Gtk

async def async_break():
    """Simulate what asyncio.sleep() does with non-zero value, but without adding a delay."""
    loop = asyncio.get_event_loop()
    future = loop.create_future()
    h = loop.call_later(0, asyncio.futures._set_result_unless_cancelled,
                        future, None)
    try:
        return await future
    finally:
        h.cancel()

@types.coroutine
def __sleep0():
    yield

async def async_break_0():
    """Simulate what asyncio.sleep(0) does."""
    await __sleep0()

class AppWindow(Gtk.ApplicationWindow):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

        self.progressbar = Gtk.ProgressBar()
        self.progressbar.show()
        self.add(self.progressbar)

        asyncio.create_task(self.update_things())

    async def update_things(self):
        for i in range(100):
            time.sleep(.1)  # Simulate work
            self.progressbar.set_fraction(i/100)
            await async_break()  # Works
            # await async_break_0()  # Broken
            # await asyncio.sleep(0)  # Broken
            # await asyncio.sleep(.1)  # Works, but adds unnessary delay.
        

class Application(Gtk.Application):
    def do_activate(self):
        self.window = AppWindow(application=self, title="Foo")
        self.window.present()

if __name__ == "__main__":
    gbulb.install(gtk=True)
    loop = asyncio.get_event_loop()
    loop.run_forever(Application())

[Dreamsorcerer]

@freakboy3742 I think this is more than just documentation.

If you try out the previous program and switch between the different commented out methods, I believe all of them should work. But, currently doing await asyncio.sleep(0), which would only be used by someone to yield back to the event loop to allow other tasks to run, does not result in the GUI updating.

I'm not familiar with the implementation, but I think that when yielded back to the event loop like this, GTK should complete all pending redraws/event handling before returning to other tasks.

I'd also like to rehighlight #44 (comment)
I can make a separate issue if you like, but it looks to me that just removing that line would fix it.