Skip to content

This pull request fixes issues with submission format unmatching in 0.3.0 and 0.4.2#26

Open
thehir0 wants to merge 3 commits intoai-forever:update/new_harness_codebasefrom
thehir0:update/new_harness_codebase
Open

This pull request fixes issues with submission format unmatching in 0.3.0 and 0.4.2#26
thehir0 wants to merge 3 commits intoai-forever:update/new_harness_codebasefrom
thehir0:update/new_harness_codebase

Conversation

@thehir0
Copy link
Contributor

@thehir0 thehir0 commented Jul 10, 2024

RuTie in 0.3.0:

{
    "data": {
        "test": [
            [
                {
                    "outputs": "1",
                    "meta": {
                        "dialog_id": 0,
                        "question_id": 0
                    }
                },

RuTie in 0.4.2

{
    "data": {
        "test": [
            [
                {
                    "outputs": {
                        "outputs": "1",
                        "meta": {
                            "id": 0
                        }
                    },
                    "meta": {
                        "dialog_id": 0,
                        "question_id": 0
                    }
                },

RuHumanEval in 0.3.0

{
    "data": {
        "test": [
            {
                "outputs": [
                    [
                        null,
                        null,
                        null,
                        null,
                        null,
                        null,
                        null,
                        null,
                        null,
                        null
                    ],
                    [
                        null,
                        null,
                        null,
                        null,
                        null,
                        null,
                        null,
                        null,
                        null,
                        null
                    ],

RuHumanEval in 0.4.2(Has double squared brackets in outputs)

{
    "data": {
        "test": [
            {
                "outputs": [
                    [
                        [
                            "None",
                            "None",
                            "None",
                            "None",
                            "None",
                            "None",
                            "None",
                            "None",
                            "None",
                            "None"
                        ],
                        [
                            "None",
                            "None",
                            "None",
                            "None",
                            "None",
                            "None",
                            "None",
                            "None",
                            "None",
                            "None"
                        ],
                        [

@thehir0
Copy link
Contributor Author

thehir0 commented Jul 10, 2024

Without this fix: i get 0.0 ruTie and 0/1/1 ruHumanEval on submission page.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant