ACT record from .csv not working from Release .exe

[mno-93]

Hey there,
I tried to work with ACT from the built version (.exe) on Windows but I couldn't make it display facial expressions from an OpenFace .csv. I specified the action with a path in the configuration and it get's accepted (validated). However, in the "record" part, I can only see Lou's base face and the control panel and queue is empty. Nothing happens when I click on the "play" controller. There are no additional buttons.

Compiling it on my own, the same issue persists.

[Pierre-LucM]

Hello @mno-93 !

Thanks for your feedback !

I took a look directly in the editor, and the configuration part works without any issues—I can see a facial expression being played correctly.

In the configuration scene, make sure to enter the absolute path to the action CSV file.

If the CSV file is correct, the "Record" scene should look like the image below, where "test" is the name of the action entered in the configuration section.

Capture.video.du.2025-02-07.22-21-44.mp4

You can also check if it works with the CSV files located in Unity Side/Assets/Data/CSV/.

Could you also let me know which version of Unity you are using?

The tests were conducted with Unity version 2022.2.21f1.

Feel free to reach out if you have any other questions!

Best regards,

Pierre-Luc

[felixdollack]

I could run Wizard-of-Oz. Tested with Unity 2022.3.9f1 and Assets/Data/CSV/pexels-edmond-dantès-4426909-3840x2160-50fps.csv

However, if I use a single object in the queue there is an error message at the end in the terminal:

If I use a queue of 2 objects only the first one get's played and the second one won't start. No error is displayed.

Screen.Recording.2025-02-10.at.22.35.48.mp4

Also, while playing the file the server keeps printing fail messages of the following type:

fail: act_server.Controller.WoZRoomController.WoZRoomController[0]
      Error processing avatar blendshape move request.
      Newtonsoft.Json.JsonSerializationException: Error converting value "{"Expressions_abdomExpansion_max":0.0,"Expressions_abdomExpansion_min":0.0,"Expressions_browOutVertL_max":0.7649999999999999,"Expressions_browOutVertL_min":0.0,"Expressions_browOutVertR_max":0.7649999999999999,"Expressions_browOutVertR_min":0.0,"Expressions_browSqueezeL_max":0.0,"Expressions_browSqueezeL_min":0.85,"Expressions_browSqueezeR_max":0.0,"Expressions_browSqueezeR_min":0.85,"Expressions_browsMidVert_max":2.58,"Expressions_browsMidVert_min":0.0,"Expressions_cheekSneerL_max":0.0,"Expressions_cheekSneerR_max":0.0,"Expressions_chestExpansion_max":0.0,"Expressions_chestExpansion_min":0.0,"Expressions_eyeClosedL_max":0.0,"Expressions_deglutition_max":0.0,"Expressions_deglutition_min":0.0,"Expressions_eyeClosedL_min":0.0,"Expressions_eyeClosedPressureL_max":0.0,"Expressions_eyeClosedPressureL_min":0.0,"Expressions_eyeClosedPressureR_max":0.0,"Expressions_eyeClosedPressureR_min":0.0,"Expressions_eyeClosedR_max":0.0,"Expressions_eyeClosedR_min":0.0,"Expressions_eyeSquintL_max":0.0,"Expressions_eyeSquintL_min":0.0,"Expressions_eyeSquintR_max":0.0,"Expressions_eyeSquintR_min":0.0,"Expressions_eyesHoriz_max":0.0,"Expressions_eyesHoriz_min":0.0,"Expressions_eyesVert_max":0.0,"Expressions_eyesVert_min":0.0,"Expressions_jawHoriz_max":0.0,"Expressions_jawHoriz_min":0.0,"Expressions_jawOut_max":0.0,"Expressions_jawOut_min":0.0,"Expressions_mouthBite_max":0.0,"Expressions_mouthBite_min":0.0,"Expressions_mouthChew_max":0.0,"Expressions_mouthChew_min":0.0,"Expressions_mouthClosed_max":0.09,"Expressions_mouthClosed_min":0.0,"Expressions_mouthHoriz_max":0.0,"Expressions_mouthHoriz_min":0.0,"Expressions_mouthInflated_max":0.0,"Expressions_mouthInflated_min":0.0,"Expressions_mouthLowerOut_max":0.0,"Expressions_mouthLowerOut_min":0.077999999999999986,"Expressions_mouthOpenAggr_max":0.0,"Expressions_mouthOpenAggr_min":0.0,"Expressions_mouthOpenHalf_max":0.0,"Expressions_mouthOpenLarge_max":0.0,"Expressions_mouthOpenLarge_min":0.0,"Expressions_mouthOpenO_max":0.0,"Expressions_mouthOpenO_min":0.0,"Expressions_mouthOpenTeethClosed_max":0.0,"Expressions_mouthOpenTeethClosed_min":0.0,"Expressions_mouthOpen_max":0.0,"Expressions_mouthOpen_min":0.0,"Expressions_mouthSmileL_max":0.054000000000000006,"Expressions_mouthSmileOpen2_max":0.0,"Expressions_mouthSmileOpen2_min":0.23399999999999999,"Expressions_mouthSmileOpen_max":0.0,"Expressions_mouthSmileOpen_min":0.0,"Expressions_mouthSmileR_max":0.054000000000000006,"Expressions_mouthSmile_max":0.51,"Expressions_mouthSmile_min":0.12,"Expressions_nostrilsExpansion_max":0.0,"Expressions_nostrilsExpansion_min":0.0,"Expressions_pupilsDilatation_max":0.0,"Expressions_pupilsDilatation_min":0.0,"Expressions_tongueHoriz_max":0.0,"Expressions_tongueHoriz_min":0.0,"Expressions_tongueOutPressure_max":0.0,"Expressions_tongueOut_max":0.0,"Expressions_tongueOut_min":0.0,"Expressions_tongueTipUp_max":0.0,"Expressions_tongueVert_max":0.0,"Expressions_tongueVert_min":0.0}" to type 'System.Collections.Generic.Dictionary`2[System.String,System.Double]'. Path 'BlendshapeDict', line 3, position 3114.
       ---> System.ArgumentException: Could not cast or convert from System.String to System.Collections.Generic.Dictionary`2[System.String,System.Double].
         at Newtonsoft.Json.Utilities.ConvertUtils.ConvertOrCast(Object initialValue, CultureInfo culture, Type targetType)
         at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.EnsureType(JsonReader reader, Object value, CultureInfo culture, JsonContract contract, Type targetType)
         --- End of inner exception stack trace ---
         at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.EnsureType(JsonReader reader, Object value, CultureInfo culture, JsonContract contract, Type targetType)
         at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
         at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ResolvePropertyAndCreatorValues(JsonObjectContract contract, JsonProperty containerProperty, JsonReader reader, Type objectType)
         at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObjectUsingCreatorWithParameters(JsonReader reader, JsonObjectContract contract, JsonProperty containerProperty, ObjectConstructor`1 creator, String id)
         at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
         at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
         at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
         at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
         at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)
         at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings)
         at act_server.Controller.WoZRoomController.WoZRoomController.OnRequestAvatarBlendshapeMove(String data, String clientId) in .../ACT/ServerSide/act_server/Controller/WoZRoomController/WoZRoomController.cs:line 249

[Pierre-LucM]

Hello! @felixdollack

This issue was already present during my internship, but due to time constraints, we weren’t able to investigate it properly. At the time, we noticed the error but couldn’t determine its root cause.

It is referenced in the README file on the act_experimental branch:

Unfortunately, I won’t have time to look into it this week, but I’ll try to check as soon as possible.

In the meantime, if you have the opportunity, feel free to investigate on your side—any insights you find could be helpful.

I’ll reach out once I make any progress or find relevant information.

[mno-93]

Hey - from my side:

• The record function works when using the test file.
• It doesn't work when using my "custom" OpenFace extracted features file. Indeed, as soon as I specify the path to this file, also the test file won't appear anymore in the control panel. I think, this is another issue mentioned before. As soon as there is more than one action in the actions editor, they won't appear in the panel within the record mode anymore.
• Looking at the test-file .csv and comparing it to the actual OpenFace output, there are some differences that I suspect, lead to not being processed:

ACT test file columns: ['frame', ' face_id', ' timestamp', ' confidence', ' success', ' pose_Tx', ' pose_Ty', ' pose_Tz', ' pose_Rx', ' pose_Ry', ' pose_Rz', ' AU01_r', ' AU02_r', ' AU04_r', ' AU05_r', ' AU06_r', ' AU07_r', ' AU09_r', ' AU10_r', ' AU12_r', ' AU14_r', ' AU15_r', ' AU17_r', ' AU20_r', ' AU23_r', ' AU25_r', ' AU26_r', ' AU45_r', ' AU01_c', ' AU02_c', ' AU04_c', ' AU05_c', ' AU06_c', ' AU07_c', ' AU09_c', ' AU10_c', ' AU12_c', ' AU14_c', ' AU15_c', ' AU17_c', ' AU20_c', ' AU23_c', ' AU25_c', ' AU26_c', ' AU28_c', ' AU45_c']

OpenFace Output columns: ['frame', 'face_id', 'timestamp', 'confidence', 'success', 'gaze_0_x', 'gaze_0_y', 'gaze_0_z', 'gaze_1_x', 'gaze_1_y', 'gaze_1_z', 'gaze_angle_x', 'gaze_angle_y', 'eye_lmk_x_0', 'eye_lmk_x_1', 'eye_lmk_x_2', 'eye_lmk_x_3', 'eye_lmk_x_4', 'eye_lmk_x_5', 'eye_lmk_x_6', 'eye_lmk_x_7', 'eye_lmk_x_8', 'eye_lmk_x_9', 'eye_lmk_x_10', 'eye_lmk_x_11', 'eye_lmk_x_12', 'eye_lmk_x_13', 'eye_lmk_x_14', 'eye_lmk_x_15', 'eye_lmk_x_16', 'eye_lmk_x_17', 'eye_lmk_x_18', 'eye_lmk_x_19', 'eye_lmk_x_20', 'eye_lmk_x_21', 'eye_lmk_x_22', 'eye_lmk_x_23', 'eye_lmk_x_24', 'eye_lmk_x_25', 'eye_lmk_x_26', 'eye_lmk_x_27', 'eye_lmk_x_28', 'eye_lmk_x_29', 'eye_lmk_x_30', 'eye_lmk_x_31', 'eye_lmk_x_32', 'eye_lmk_x_33', 'eye_lmk_x_34', 'eye_lmk_x_35', 'eye_lmk_x_36', 'eye_lmk_x_37', 'eye_lmk_x_38', 'eye_lmk_x_39', 'eye_lmk_x_40', 'eye_lmk_x_41', 'eye_lmk_x_42', 'eye_lmk_x_43', 'eye_lmk_x_44', 'eye_lmk_x_45', 'eye_lmk_x_46', 'eye_lmk_x_47', 'eye_lmk_x_48', 'eye_lmk_x_49', 'eye_lmk_x_50', 'eye_lmk_x_51', 'eye_lmk_x_52', 'eye_lmk_x_53', 'eye_lmk_x_54', 'eye_lmk_x_55', 'eye_lmk_y_0', 'eye_lmk_y_1', 'eye_lmk_y_2', 'eye_lmk_y_3', 'eye_lmk_y_4', 'eye_lmk_y_5', 'eye_lmk_y_6', 'eye_lmk_y_7', 'eye_lmk_y_8', 'eye_lmk_y_9', 'eye_lmk_y_10', 'eye_lmk_y_11', 'eye_lmk_y_12', 'eye_lmk_y_13', 'eye_lmk_y_14', 'eye_lmk_y_15', 'eye_lmk_y_16', 'eye_lmk_y_17', 'eye_lmk_y_18', 'eye_lmk_y_19', 'eye_lmk_y_20', 'eye_lmk_y_21', 'eye_lmk_y_22', 'eye_lmk_y_23', 'eye_lmk_y_24', 'eye_lmk_y_25', 'eye_lmk_y_26', 'eye_lmk_y_27', 'eye_lmk_y_28', 'eye_lmk_y_29', 'eye_lmk_y_30', 'eye_lmk_y_31', 'eye_lmk_y_32', 'eye_lmk_y_33', 'eye_lmk_y_34', 'eye_lmk_y_35', 'eye_lmk_y_36', 'eye_lmk_y_37', 'eye_lmk_y_38', 'eye_lmk_y_39', 'eye_lmk_y_40', 'eye_lmk_y_41', 'eye_lmk_y_42', 'eye_lmk_y_43', 'eye_lmk_y_44', 'eye_lmk_y_45', 'eye_lmk_y_46', 'eye_lmk_y_47', 'eye_lmk_y_48', 'eye_lmk_y_49', 'eye_lmk_y_50', 'eye_lmk_y_51', 'eye_lmk_y_52', 'eye_lmk_y_53', 'eye_lmk_y_54', 'eye_lmk_y_55', 'eye_lmk_X_0', 'eye_lmk_X_1', 'eye_lmk_X_2', 'eye_lmk_X_3', 'eye_lmk_X_4', 'eye_lmk_X_5', 'eye_lmk_X_6', 'eye_lmk_X_7', 'eye_lmk_X_8', 'eye_lmk_X_9', 'eye_lmk_X_10', 'eye_lmk_X_11', 'eye_lmk_X_12', 'eye_lmk_X_13', 'eye_lmk_X_14', 'eye_lmk_X_15', 'eye_lmk_X_16', 'eye_lmk_X_17', 'eye_lmk_X_18', 'eye_lmk_X_19', 'eye_lmk_X_20', 'eye_lmk_X_21', 'eye_lmk_X_22', 'eye_lmk_X_23', 'eye_lmk_X_24', 'eye_lmk_X_25', 'eye_lmk_X_26', 'eye_lmk_X_27', 'eye_lmk_X_28', 'eye_lmk_X_29', 'eye_lmk_X_30', 'eye_lmk_X_31', 'eye_lmk_X_32', 'eye_lmk_X_33', 'eye_lmk_X_34', 'eye_lmk_X_35', 'eye_lmk_X_36', 'eye_lmk_X_37', 'eye_lmk_X_38', 'eye_lmk_X_39', 'eye_lmk_X_40', 'eye_lmk_X_41', 'eye_lmk_X_42', 'eye_lmk_X_43', 'eye_lmk_X_44', 'eye_lmk_X_45', 'eye_lmk_X_46', 'eye_lmk_X_47', 'eye_lmk_X_48', 'eye_lmk_X_49', 'eye_lmk_X_50', 'eye_lmk_X_51', 'eye_lmk_X_52', 'eye_lmk_X_53', 'eye_lmk_X_54', 'eye_lmk_X_55', 'eye_lmk_Y_0', 'eye_lmk_Y_1', 'eye_lmk_Y_2', 'eye_lmk_Y_3', 'eye_lmk_Y_4', 'eye_lmk_Y_5', 'eye_lmk_Y_6', 'eye_lmk_Y_7', 'eye_lmk_Y_8', 'eye_lmk_Y_9', 'eye_lmk_Y_10', 'eye_lmk_Y_11', 'eye_lmk_Y_12', 'eye_lmk_Y_13', 'eye_lmk_Y_14', 'eye_lmk_Y_15', 'eye_lmk_Y_16', 'eye_lmk_Y_17', 'eye_lmk_Y_18', 'eye_lmk_Y_19', 'eye_lmk_Y_20', 'eye_lmk_Y_21', 'eye_lmk_Y_22', 'eye_lmk_Y_23', 'eye_lmk_Y_24', 'eye_lmk_Y_25', 'eye_lmk_Y_26', 'eye_lmk_Y_27', 'eye_lmk_Y_28', 'eye_lmk_Y_29', 'eye_lmk_Y_30', 'eye_lmk_Y_31', 'eye_lmk_Y_32', 'eye_lmk_Y_33', 'eye_lmk_Y_34', 'eye_lmk_Y_35', 'eye_lmk_Y_36', 'eye_lmk_Y_37', 'eye_lmk_Y_38', 'eye_lmk_Y_39', 'eye_lmk_Y_40', 'eye_lmk_Y_41', 'eye_lmk_Y_42', 'eye_lmk_Y_43', 'eye_lmk_Y_44', 'eye_lmk_Y_45', 'eye_lmk_Y_46', 'eye_lmk_Y_47', 'eye_lmk_Y_48', 'eye_lmk_Y_49', 'eye_lmk_Y_50', 'eye_lmk_Y_51', 'eye_lmk_Y_52', 'eye_lmk_Y_53', 'eye_lmk_Y_54', 'eye_lmk_Y_55', 'eye_lmk_Z_0', 'eye_lmk_Z_1', 'eye_lmk_Z_2', 'eye_lmk_Z_3', 'eye_lmk_Z_4', 'eye_lmk_Z_5', 'eye_lmk_Z_6', 'eye_lmk_Z_7', 'eye_lmk_Z_8', 'eye_lmk_Z_9', 'eye_lmk_Z_10', 'eye_lmk_Z_11', 'eye_lmk_Z_12', 'eye_lmk_Z_13', 'eye_lmk_Z_14', 'eye_lmk_Z_15', 'eye_lmk_Z_16', 'eye_lmk_Z_17', 'eye_lmk_Z_18', 'eye_lmk_Z_19', 'eye_lmk_Z_20', 'eye_lmk_Z_21', 'eye_lmk_Z_22', 'eye_lmk_Z_23', 'eye_lmk_Z_24', 'eye_lmk_Z_25', 'eye_lmk_Z_26', 'eye_lmk_Z_27', 'eye_lmk_Z_28', 'eye_lmk_Z_29', 'eye_lmk_Z_30', 'eye_lmk_Z_31', 'eye_lmk_Z_32', 'eye_lmk_Z_33', 'eye_lmk_Z_34', 'eye_lmk_Z_35', 'eye_lmk_Z_36', 'eye_lmk_Z_37', 'eye_lmk_Z_38', 'eye_lmk_Z_39', 'eye_lmk_Z_40', 'eye_lmk_Z_41', 'eye_lmk_Z_42', 'eye_lmk_Z_43', 'eye_lmk_Z_44', 'eye_lmk_Z_45', 'eye_lmk_Z_46', 'eye_lmk_Z_47', 'eye_lmk_Z_48', 'eye_lmk_Z_49', 'eye_lmk_Z_50', 'eye_lmk_Z_51', 'eye_lmk_Z_52', 'eye_lmk_Z_53', 'eye_lmk_Z_54', 'eye_lmk_Z_55', 'pose_Tx', 'pose_Ty', 'pose_Tz', 'pose_Rx', 'pose_Ry', 'pose_Rz', 'AU01_r', 'AU02_r', 'AU04_r', 'AU05_r', 'AU06_r', 'AU07_r', 'AU09_r', 'AU10_r', 'AU12_r', 'AU14_r', 'AU15_r', 'AU17_r', 'AU20_r', 'AU23_r', 'AU25_r', 'AU26_r', 'AU45_r', 'AU01_c', 'AU02_c', 'AU04_c', 'AU05_c', 'AU06_c', 'AU07_c', 'AU09_c', 'AU10_c', 'AU12_c', 'AU14_c', 'AU15_c', 'AU17_c', 'AU20_c', 'AU23_c', 'AU25_c', 'AU26_c', 'AU28_c', 'AU45_c']

• The test file column names seem to have a white space before the actual column name (in all but the "frame" columns).
• There are obviously more columns available in the original OpenFace output but the namings seem consistent besides the white spaces.
• I tried to only extract the "required" columns and also adapt their names to match the sample file's (white spaces) but couldn't make it work yet. I'll paste an OpenFace output sample in the issue.

It would be nice to know what the issue with the .csv file formatting is and to know what exactly ACT expects in order to visualize it (i.e., range of AU units; how are missings handled; etc.)

EDIT: I just tried to create a copy of the working sample .csv using pandas read_csv and to_csv(index = False). However, this already leads to the copy not working with ACT. I checked the dtypes, too but couldn't figure out yet what's the issue when reading/writing the .csv with pandas....

OpenFace_output_example.csv

[felixdollack]

@Pierre-LucM I just checked and saw it is not only the second but every even file (2,4,...)
Seems like an indexing error to me. But so far I don't know where the problem is in code.

@mno-93 I tried the above with the OpenFace_output_example file you attached.
It does work and moves the avatar in Wizard-of-Oz mode. Regarding the recording mode I see the action with the OpenFace file and can select it, but pressing record only removes the menu and then does nothing.