Skip to content

Conversation

@Magus4450
Copy link
Contributor

Summary

  • Add new letter count experiment configuration for easy tasks
  • Add instructions in README for running the letter count experiment

This task trains the model to count letter occurrences in words. Given an input word, the model outputs each unique letter with its count.

Sample Predictions (after training)

Input: <text>Morel<count> 
Generated: M1 O1 R1 E1 L1
Reference: M1 O1 R1 E1 L1

Input: <text>neurosis<count> 
Generated: N1 E1 U1 R1 O1 T1 I1 C1
Reference: N1 E1 U1 R1 O1 S2 I1

Input: <text>mydaleine<count> 
Generated: M1 Y1 D1 E1 R1 I1 S1 H1 T1
Reference: M1 Y1 D1 A1 L1 E2 I1 N1

Input: <text>forgetter<count> 
Generated: F1 O1 R2 E2 T1 A1 N1
Reference: F1 O1 R2 G1 E2 T2

Input: <text>fiddley<count> 
Generated: F1 I1 L2 E1 S1 T1
Reference: F1 I1 D2 L1 E1 Y1

Eval Metrics

Metric Value
eval_byte_accuracy 0.9392
eval_word_accuracy 0.791
eval_chrf 40.5035
eval_sacrebleu 21.3431
eval_loss 0.6054
perplexity 1.8321

Test plan

  • Verified config format matches existing easy tasks
  • Ran training and confirmed model learns the task

🤖 Generated with Claude Code

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds a new "letter count" experiment to the easy-tasks suite, which trains the model to count letter occurrences in words. Given an input word, the model outputs each unique letter with its count (e.g., "Morel" → "M1 O1 R1 E1 L1").

Changes:

  • Added letter-count.yaml configuration file with model, dataset, training, and evaluation parameters
  • Updated README.md with instructions for running the letter count experiment

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
welt_training/experiments/easy-tasks/letter-count.yaml New configuration file for letter count task with model setup, dataset configuration, training parameters, and evaluation metrics
welt_training/experiments/easy-tasks/README.md Added documentation section with command to run the letter count experiment

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@AmitMY AmitMY merged commit 95d3c58 into sign:main Jan 19, 2026
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants