Missing files

[senthilpalanisamy]

The training part requires preprocessing - mpii_gaze_preprocessing - which requires a matlab file - 6 points-based face model.mat. Can this file be provided, since its not possible to train without having access to this. Thank you for open sourcing this work. Its really helpful.

[kimou6055]

i'm facing the same issue

[amandacolibridev]

i'm facing the same issue

were you able to solve it? i have the same problem.

[Ghenazidan]

Any updates? I have the same problem

[amandacolibridev]

I solved it by making my own file. Honestly idk if this is right but at least i can run the code.

def extract_model(image_path):
    image = cv2.imread(image_path)
    rgb_image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
    results = face_mesh.process(rgb_image)


    if results.multi_face_landmarks:
        face_landmarks = results.multi_face_landmarks[0]
        # Example: Extracting the 3D coordinates for specific facial landmarks (like eyes and mouth corners)
        left_eye_left_corner = face_landmarks.landmark[LEFT_EYE_CORNER_LEFT]  # MediaPipe's index for left eye corner
        left_eye_right_corner = face_landmarks.landmark[LEFT_EYE_CORNER_RIGHT]
        right_eye_left_corner = face_landmarks.landmark[RIGHT_EYE_CORNER_LEFT]
        right_eye_right_corner = face_landmarks.landmark[RIGHT_EYE_CORNER_RIGHT]
        mouth_left_corner = face_landmarks.landmark[MOUTH_CORNER_LEFT]
        mouth_right_corner = face_landmarks.landmark[MOUTH_CORNER_RIGHT]

        # Store the 3D points (x, y, z) in an array
        model_pointsR = np.array([
            [left_eye_left_corner.x, left_eye_left_corner.y, left_eye_left_corner.z],
            [left_eye_right_corner.x, left_eye_right_corner.y, left_eye_right_corner.z],
            [right_eye_left_corner.x, right_eye_left_corner.y, right_eye_left_corner.z],
            [right_eye_right_corner.x, right_eye_right_corner.y, right_eye_right_corner.z],
            [mouth_left_corner.x, mouth_left_corner.y, mouth_left_corner.z],
            [mouth_right_corner.x, mouth_right_corner.y, mouth_right_corner.z]
        ])

        return model_pointsR
    return None

Then i save model_pointsR in a .txt file and then manually convert it to .mat.

[cycid]

I solved it by making my own file. Honestly idk if this is right but at least i can run the code.

def extract_model(image_path):
    image = cv2.imread(image_path)
    rgb_image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
    results = face_mesh.process(rgb_image)


    if results.multi_face_landmarks:
        face_landmarks = results.multi_face_landmarks[0]
        # Example: Extracting the 3D coordinates for specific facial landmarks (like eyes and mouth corners)
        left_eye_left_corner = face_landmarks.landmark[LEFT_EYE_CORNER_LEFT]  # MediaPipe's index for left eye corner
        left_eye_right_corner = face_landmarks.landmark[LEFT_EYE_CORNER_RIGHT]
        right_eye_left_corner = face_landmarks.landmark[RIGHT_EYE_CORNER_LEFT]
        right_eye_right_corner = face_landmarks.landmark[RIGHT_EYE_CORNER_RIGHT]
        mouth_left_corner = face_landmarks.landmark[MOUTH_CORNER_LEFT]
        mouth_right_corner = face_landmarks.landmark[MOUTH_CORNER_RIGHT]

        # Store the 3D points (x, y, z) in an array
        model_pointsR = np.array([
            [left_eye_left_corner.x, left_eye_left_corner.y, left_eye_left_corner.z],
            [left_eye_right_corner.x, left_eye_right_corner.y, left_eye_right_corner.z],
            [right_eye_left_corner.x, right_eye_left_corner.y, right_eye_left_corner.z],
            [right_eye_right_corner.x, right_eye_right_corner.y, right_eye_right_corner.z],
            [mouth_left_corner.x, mouth_left_corner.y, mouth_left_corner.z],
            [mouth_right_corner.x, mouth_right_corner.y, mouth_right_corner.z]
        ])

        return model_pointsR
    return None

Then i save model_pointsR in a .txt file and then manually convert it to .mat.

Hi, did you train the model and ran it? It works well?

[amandacolibridev]

Yes, I did. I can say the results make sense, but for me it didn't work so well. Maybe because of the setup (different camera, computer screen, etc).

[cycid]

Yes, I did. I can say the results make sense, but for me it didn't work so well. Maybe because of the setup (different camera, computer screen, etc).

About the .mat file, I think it's supposed to be mediapipes canonical_face_model, the author uses in his gaze_tracking_pipeline repo.
Did you create your own dataset with your data with the h5 file? And how did you use calibration parameters, I'v noticed the code is writing it to tensorboard log, but I didn't found way to use it in the gaze_tracking_pipeline repo

[amandacolibridev]

I agree about the canonical_face_model, but this worked best for me. First i tried with MPIIFaceGaze dataset and then I created my own. When you say calibration parameters you mean the camera calibration?

[Ghenazidan]

did you get better results after you created your own dataset? I mean the model isn't working accuratly. I wonder if the dataset causing the inaccuracy @amandacolibridev @cycid

[amandacolibridev]

@Ghenazidan yes, I got better results with my own dataset, but still, the model wasn't very good either.

[Ghenazidan]

thank you for your feedback @amandacolibridev

[Ghenazidan]

did you find another model for gaze tracking?

[amandacolibridev]

@Ghenazidan Not one that suits my needs. There are a lot more here on Github, but this one was the closest to what I wanted. Right now I'm trying to create my own 🙏

[cycid]

I agree about the canonical_face_model, but this worked best for me. First i tried with MPIIFaceGaze dataset and then I created my own. When you say calibration parameters you mean the camera calibration?

I meant person-specific calibration, author write about it here

"Nine calibration samples has become the norm for the comparison of different model architectures using MPIIFaceGaze. When the calibration points are organized in a k×k grid instead of randomly on the screen, or all in one position, the resulting person-specific calibration is more accurate. The three different ways to distribute the calibration point are compared in the figure below, also see "own model 3" in the first figure. Nine calibration samples aligned in a grid result in a lower angular error than 9 randomly positioned calibration samples.
To collect your own calibration data or dataset, please refer to gaze data collection."

Also Im not getting any results from calibration_train variable while running train or eval on both my own and pretrained models in train.py file. This collected person-specific calibration suppose to be biases adding to model results depends on each person features. In case I understand how the project works correctly.

[cycid]

did you get better results after you created your own dataset? I mean the model isn't working accuratly. I wonder if the dataset causing the inaccuracy @amandacolibridev @cycid

Using eval.py I got better results my own model than a pertained model, but when I'm running my model, the results look worse. Maybe I used too small dataset. I got 13 mean angular error on my model, thats a lot.