[REG-2202] [REG-2205] [REG-2228] Exploratory bot actions + mouse key moment recording #366
Conversation
…r of real objects
…n on click hold broken right now !!!
…click not triggering right though
nAmKcAz
changed the title
nAmKcAz
changed the title
| // if we're working on the first entry in the list is the only time we do actions | ||
| if (nextBotSegmentIndex == 0) | ||
| { | ||
| ProcessBotSegmentAction(nextBotSegment, transformStatuses, entityStatuses); |
There was a problem hiding this comment.
This line is the only real change of note in this method... the rest of it shows changed because I got tired of trying to find it in this class and moved it to the top
| } | ||
| } | ||
|
|
||
| /** |
There was a problem hiding this comment.
This is where the exploration magic happens on playback / the algorithm from Update -> Update is documented
| * | ||
| * The current implementation chooses one of the previous N successful IKeyMomentExploration actions to try on each update pass. | ||
| */ | ||
| public void PerformExploratoryAction(int segmentNumber, Dictionary<long, ObjectStatus> currentTransforms, Dictionary<long, ObjectStatus> currentEntities, out string error) |
There was a problem hiding this comment.
This is where exploratory actions are driven from. We will hook in other exploratory action types to this class as we create them
| }; | ||
|
|
||
| [CanBeNull] | ||
| public BotSegment EvaluateKeyMoment(long tickNumber, out long keyMomentNumber) |
There was a problem hiding this comment.
This is where KeyMoment recording happens.
We will hook in other types beyond mouse as noted in the comments in this file.
| } | ||
|
|
||
| private List<ObjectStatus> FindObjectsAtPosition(Vector2 position, IEnumerable<ObjectStatus> statefulObjects, out float maxZDepth) | ||
| public static List<ObjectStatus> FindObjectsAtPosition(Vector2 position, IEnumerable<ObjectStatus> statefulObjects) |
There was a problem hiding this comment.
This method now actually works as you would expect and returns properly sorted results based on depth from camera as a click would be processed.
nAmKcAz
changed the title
|
Update post Loom video.. Commit e74ffb4 [REG-2228] fixes the world space object click misses |
abeizer
left a comment
abeizer
left a comment
There was a problem hiding this comment.
Wow, many changes and very dense. I appreciate the decision to document so much in-code, it was easy to understand how exploration is woven into the replay and how clicks + unclicks are tracked and applied to objects in the scene.
I'm hesitant to dig too deeply into reviewing line-by-line because I expect details to change frequently as we add exploration support to more actions and figure out how they should/shouldn't interact with each other. At a high level, I'm on board with this exploration pattern and think we should keep pushing forward.
…h to avoid causing unity to treat the input as a drag
…considerations in shrinking bounds
addisonbgross
left a comment
addisonbgross
left a comment
There was a problem hiding this comment.
I've reviewed this PR over 2 days, due to its size, and everything looks reasonable. Your Looms and PR/Code comments made it much easier to understand. Like Abby, I don't really have any commentary on the line-by-line changes. I look forward to testing this out!
vontell
left a comment
vontell
left a comment
There was a problem hiding this comment.
Just passing by because I finally had a chance to look at this today. Very excited to try this out, and good to see that these key moments and exploration is saved as an additional file to keep the previous approach intact.
I'll start to think about new exploratory approaches... got some initial ideas.
The intent for this first pass was to get 'close' to parity with botsegment mouse recordings while building out the framework for exploratory actions to expand from here. There are some quirks/weirdnesses from time to time, but for bossroom and steelline, this does some things we couldn't do before with recording, while still coming close to being as deterministic in its action choices.
Loom Demo on Bossroom of Record + Playback
Creates the framework for allowing bots to explore actions to take when a recorded action errors
Adds a new recording of 'key moments' alongside the existing bot segments
clickedObjectNormalizedPathsof the muse dataRe-writes how we detect objects at a mouse position (honestly the old was so not good I'm not sure how we go this far with it)
Sample KeyMoment Mouse Action file
{ "name":"KeyMoment: 2, Tick: 2 - Mouse Action Segment", "description":null, "sessionId":null, "apiVersion":24, "endCriteria":[ {"type":"ActionComplete","apiVersion":7,"transient":false,"data":{"apiVersion":11}} ], "botAction":{"type":"KeyMoment_MouseAction","apiVersion":1,"data":{"apiVersion":28,"mouseActions":[ {"apiVersion":1,"startTime":17.0340247,"leftButton":false,"middleButton":false,"rightButton":false,"forwardButton":false,"backButton":false,"scroll":{"x":0,"y":0},"screenSize":{"x":1502,"y":796},"position":{"x":765,"y":587},"worldPosition":null,"clickedObjectNormalizedPaths":[]}, {"apiVersion":1,"startTime":17.0433465,"leftButton":true,"middleButton":false,"rightButton":false,"forwardButton":false,"backButton":false,"scroll":{"x":0,"y":0},"screenSize":{"x":1502,"y":796},"position":{"x":765,"y":587},"worldPosition":null,"clickedObjectNormalizedPaths":["UI Canvas/ProfilePopup/ProfilesPanel/Viewport/ProfileList/ProfileListItemUI Prototype/Select Profile Button","UI Canvas/ProfilePopup/ProfilesPanel/Viewport","UI Canvas/ProfilePopup","UI Canvas/Title","UI Canvas/ProfilePopup/ProfilesPanel/Viewport/ProfileList/ProfileListItemUI Prototype/Select Profile Button/ProfileNameLabel","UI Canvas/ProfilePopup/ProfilesPanel/Viewport/ProfileList/ProfileListItemUI Prototype","UI Canvas/ProfilePopup/Image"]}, {"apiVersion":1,"startTime":17.1590328,"leftButton":false,"middleButton":false,"rightButton":false,"forwardButton":false,"backButton":false,"scroll":{"x":0,"y":0},"screenSize":{"x":1502,"y":796},"position":{"x":765,"y":587},"worldPosition":null,"clickedObjectNormalizedPaths":["UI Canvas/Title"]} ]}}}Find the pull request instructions here
Every reviewer and the owner of the PR should consider these points in their request (feel free to copy this checklist so you can fill it out yourself in the overall PR comment)