1210 chunks search#208

Open

FarmersWrap wants to merge 14 commits intotexttron:mainfrom

FarmersWrap:1210-chunks-search

Commits on Dec 10, 2025

rewrote passage chunking
Ryan Yu
committed
added logic for left padding
Ryan Yu
committed

Commits on Dec 11, 2025

added search
Ryan Yu
committed

Commits on Dec 14, 2025

changed the tokenizer logic
Ryan Yu
committed
added train collator debug
Ryan Yu
committed
traincollator is done
Ryan Yu
committed
fixed some comments
Ryan Yu
committed
modified chunkedencoder
Ryan Yu
committed
Modified forward and model
Ryan Yu
committed
Modified inference on chunked passage, in progress
Ryan Yu
committed
fixed a chunk size not passed to model
Ryan Yu
committed

Commits on Dec 15, 2025

changed eos to sep
Ryan Yu
committed

Commits on Dec 16, 2025

added logs
Ryan Yu
committed

Commits on Dec 19, 2025

added some scripts
Ryan Yu
committed