Assistant design improvements + V2 API

[mpivchev]

Prerequisite: nextcloud/NextcloudKit#124

1. Implemented [Assistant_iOS] Improve Assistant on mobile screen #3241 + Assistant improvements #3226.
   Some points @Hyeyoung346:

• Tasks created in the same day show up as "Today" in the timeframe.
• I think it's good to keep the status in the task detail, as just icons is not enough information IMO. In that case in the detail the status is shown like this:

• "Pending" does not show in the list and only shows on the detail, as that is where we get more information.

2. Fixed [Assistant] A list view with scrolling issue #3201
3. Made Assistant compatible with API V1 (<NC30) and API V2 (>=NC30)
4. Added UI tests
   • @i2h3 @marinofaggiana You can run the UI tests to test V2 functionality. You need a server instance with assistant and testing app enabled. Server.sh now includes the commands to enable that, so you can start a server using that script.
   • For testing V1, please change the useV2 flag to false and test manually.

[Hyeyoung346]

@mpivchev I think AI conversations are rarely “completed”. People often want to ask follow-up questions. This aligns better with the natural flow of human reasoning (where follow-up questions are common), therefore, should avoid “completed”mark.

• Tasks created in the same day show up as "Today" in the timeframe.
  => I think "Today" is possible, but how to display the time frame after today in the list view?

• I think it's good to keep the status in the task detail, as just icons is not enough information IMO.

• "Pending" does not show in the list and only shows on the detail, as that is where we get more information.
  => I think it would be useful to show the status (timeframe, pending) in the list view rather than in the task view.
  Maybe if it’s possible to implement follow-up questions with input filed inside the task detail later, probably would be useful to add good/bad feedback or report legal issue, modify response etc.

What is the difference between API V1 (<NC30) and API V2 (>=NC30)? Is there any improvements or difference from API V1?

[mpivchev]

@Hyeyoung346

1. We can put the completion status on the left side, and the date on the right side?
   I think in general having the icon on the list items is good enough for information, and it's not too cluttered. If you do want more information you can then open the details.

• I do think in general if we are going to have a message-like input and output structure we will have to redesign a lot of this. For example instead of straight up showing the status we can do it like a message saying "I could not generate a response, try again" or something similar.

For now, however, since we just have input-output prompts, this is informative.

2. You didn't comment on this, so not sure if you agree.

3. As mentioned above, i think it becomes too cluttered.

4. Difference between API 1 and 2 is that we can have different input and output types for different task types (text input, uploading image, uploading audio, transcribing, translating etc). Currently for mobile apps only simple text input is supported.

• V2 also sends us more information: progress, last updated, scheduled at and ended at. V1 only provides completed at.

At the moment it looks like this:

[Hyeyoung346]

@mpivchev

1. You mean icon + date in the list view? I think current list view screen looks good & informative. Perhaps the date could be displayed shorter => 25.Feb 2025.
   I mean icon + date + completed mark in the task details screen looks still questionable. I noticed that output responses sometimes also ask the following questions like “Would you like to know more about ~?”, so it looked strange to show the “completed” mark. Not sure icon + date + completed mark would be informative in the task detail screen, so I want to suggest to remove this part. But, as you said, we just have simple input-output prompts, so can iterate later further.

2. You mean => Fixed [Assistant] A list view with scrolling issue #3201
   Yeah if you fixed that scrolling issue in the list view, that would be awesome!

Thanks for the detailed info! It looks better and more useful now, with more functionalities.

[mpivchev]

Hi @Hyeyoung346 the scrolling issue is already fixed in this PR. It's just open until this PR is merged :)

[mpivchev]

Regarding 1. I'll see what I can do on Monday. Have a nice weekend 😀

[mpivchev]

@Hyeyoung346 modified the date to show in a more human readable way
- Less than a minute: Returns "Less than a minute ago".
- Less than an hour: Returns the number of minutes (e.g., "5 minutes ago").
- Less than a day: Returns the number of hours (e.g., "2 hours ago").
- Less than a month: Returns the number of days (e.g., "3 days ago").
- More than a month: Returns the full formatted date (e.g., "Jan 10, 2025").

I mean icon + date + completed mark in the task details screen looks still questionable. I noticed that output responses sometimes also ask the following questions like “Would you like to know more about ~?”, so it looked strange to show the “completed” mark. Not sure icon + date + completed mark would be informative in the task detail screen, so I want to suggest to remove this part. But, as you said, we just have simple input-output prompts, so can iterate later further

I agree ultimately we should remove the status altogether and mimic a chat-like behavior. But for now, since it's just a prompt-style, we should keep it this way IMO.

[mpivchev]

@jancborchardt can you quickly go over this :)

[mpivchev]

@mpivchev Seems do not use all width and the font seems a bit too big

This doesnt happen for me

[i2h3]

I tried to test again but I cannot on our daily server due to nextcloud/server#51248

[jancborchardt]

@mpivchev can’t see your comment about the vertical spacing anymore, but if you think it’s too close together, let’s at least go for a padding value inbetween the current state and your screenshot. :)

[mpivchev]

@mpivchev can’t see your comment about the vertical spacing anymore, but if you think it’s too close together, let’s at least go for a padding value inbetween the current state and your screenshot. :)

Hi, added equal padding so it looks better now :) @jancborchardt

[jancborchardt]

Looking good, thank you! :)

[i2h3]

@mpivchev Is fixing the translation form out of scope? There is an obvious mismatch between the web version and what the app offers with its single text field and no hint what to enter there.

[mpivchev]

@mpivchev Is fixing the translation form out of scope? There is an obvious mismatch between the web version and what the app offers with its single text field and no hint what to enter there.

This is out of scope for now, currently iOS and Android only support certain tasks, and Translate is not one of them. In normal instances (not the one you use), I believe the API only returns the available tasks.

[i2h3]

UI tests are fragile and fail for me. Mostly it is the login and not your fault. By now I think we should replace the fragile web view interaction by just injecting an app token through an app launch argument so the whole login can be skipped and tests run much faster to the point.

The tests did not work on the daily server, but tech preview somewhat better.

Also it was strange that I first had to remove all the unnecessary framework references from the Xcode UI test target. Xcode was complaining about RealmSwift not being found when linking. It is not even supposed to be in the UI test target. 😵‍💫

I stop at this point now, I could spend the rest of the day making the tests work.

[i2h3]

The subscript is invalid. Issuing p app.navigationBars.allElementsBoundByAccessibilityElement.map(\.identifier) in the debugger prints:

([String]) 2 values {
  [0] = "More"
  [1] = "New ContextAgent task"
}

This resembles the navigation bar title "New ContextAgent task" in the sheet of the assistant.

[i2h3]

Interestingly, this is different when testing against the server on localhost:

([String]) 2 values {
  [0] = "More"
  [1] = "New Free text to text prompt task"
}

[i2h3]

This fails for me when testing against tech preview because "an error occurred" when creating the task, so there is no cell to find.

[i2h3]

When running against localhost, a short delay is needed in between, otherwise pull to refresh will fail because the cell has not appeared yet in the visible area.

[mpivchev]

UI tests are fragile and fail for me. Mostly it is the login and not your fault. By now I think we should replace the fragile web view interaction by just injecting an app token through an app launch argument so the whole login can be skipped and tests run much faster to the point.

The tests did not work on the daily server, but tech preview somewhat better.

Also it was strange that I first had to remove all the unnecessary framework references from the Xcode UI test target. Xcode was complaining about RealmSwift not being found when linking. It is not even supposed to be in the UI test target. 😵‍💫

I stop at this point now, I could spend the rest of the day making the tests work.

The tests were made to be ran with Server.sh just like the download limit tests. Using any other server can cause the tests to fail as they may have different tasks enabled and so on. Please always use the Server.sh instance for these tests.

Regarding injecting the app token, i think we did that before, let me check.

[i2h3]

The tests were made to be ran with Server.sh just like the download limit tests.

I got confused because you recommended other servers. 😄 I see now, for manual testing only.

Regarding injecting the app token, i think we did that before, let me check.

I think Talk team does it.

[i2h3]

The UI tests now ran through now. 🤷🏻‍♀️ Both on iPhone and iPad simulator. With the requested changes it is fine merge then.

[mpivchev]

@marinofaggiana @i2h3 please make sure to also test V1, with christine or daily nc instance.

[i2h3]

@i2h3 please make sure to also test V1, with christine or daily nc instance.

I did so. With the changes requested above I can approve this.

[mpivchev]

@i2h3 all PR suggestions added