Encoder can crash

[kwalcock]

Over 10,000 sentences, the encoder crashes reliably on this one sentence.

1382 Main Korean Marine Organisms of Commercial Value Korean Names Chong-o-ri Myong-tao Chaki Poku-tchi Kodeung-o Myol-tchi Kal-tchi Tcheng-o Sao-u Y Y 9 , Tae-ha 3 , Tae-ku Kaj yami Bom-kajyami ( 92 ) Scientific Names Sardinia melanosticta Thoragra chalcogramma Nibea manchurica Nibea schlegeli Scomber japonieus Engraulis japonicus Clupea pallasi Trichiurus japo & icus Acetes japonicus Pasiphaca sivodo ' Sergestes phosphorous Penaeus japonicus Pandalus hypsitietus Gadus macrocephalus Limanda angustriestris Verusper variegatus English Names Sole Korean dulse Croaker Umukasari Chupmackerel Glue seaweed Y Y Conger Eel including sea eel Flat fish > Y Octopus Korean Names Norang kajyami Myok Min-o Umukasari Sam - tchi Semo Bul-tungi Bungchang-o Kact-chang-o Nop-tchi Se-tai Nak-chi Moon-o Ecienfific Names Verusper moseri Undaria pinnatifida Nibea imbricata Euchouma amalisaousis Sawara japonica Gloiopeltis tenax Gloiopeltis furcata Conger ruyriaster Muraenesox cinerous Paralichthys olivaceus Areliseus holandi Polypus variabilis Polypus vulgaris Appendix 2 List of Types of Fishing Gear Used in Korean Waters FIXED GEAR MOVABLE GEAR Name Stow Net ( 2 kinds ) Square Net ( 3 kinds ) Raising Net Small ' ~ e a d Pole Net Swing Net Fall Net ( 2 kinds ) Bag Net Baulk Net Chubock Net Bang yom Weir Bu Net Chung son net Gill net ( 7 kinds ) Long line ( 10 k ids ) Set net Korean Name Dae Momang Kack Mang Dul Mang Sodae Mang Doi Mang Nack Mang Bing Mang Dadeboshi Mang Chubock Bang yom Sal Bu Mang Chung son Mang Tchock3-kumul Chunak Kun Mang Beach seine Kwon hyon Mang Sail drag net Ang Kan net ( 3 kinds ) Sailing bottom draft net Shell Net Cast Net Purse seine ( 4 kinds ) Drift Net ( 5 kinds ) Draft and gill net ( 2 kinds ) Danish seine Single line ( 6 kinds ) Nude diving Whaling Whaling Scoop net ( tuck stone net ) Double net Korean Name Furi kumul Kwon hyon Mang Bae furi kumul Ang Rang Mang Bae choyo Mang Chokae kulki Mang Too Mang ( che aengi kumul ) Kison Konchaek Mang Fuilim Kumul Fullim Tchocko Kison joyo Mang Nacsi Najam Phokyong Cho Mang Chung son Man

2022-11-21 12:13:55.211743562 [E:onnxruntime:, sequential_executor.cc:369 Execute] Non-zero status code returned while running Expand node. Name:'Expand_192' Status Message: invalid expand shape
Exception in thread "main" ai.onnxruntime.OrtException: Error code - ORT_INVALID_ARGUMENT - message: Non-zero status code returned while running Expand node. Name:'Expand_192' Status Message: invalid expand shape
	at ai.onnxruntime.OrtSession.run(Native Method)
	at ai.onnxruntime.OrtSession.run(OrtSession.java:295)
	at ai.onnxruntime.OrtSession.run(OrtSession.java:238)
	at ai.onnxruntime.OrtSession.run(OrtSession.java:207)
	at org.clulab.scala_transformers.encoder.Encoder.forward(Encoder.scala:19)
	at org.clulab.scala_transformers.encoder.Encoder.forward(Encoder.scala:31)
	at org.clulab.scala_transformers.encoder.timer.TokenClassifierTimerApp$TimedTokenClassifier.$anonfun$predict$2(TokenClassifierTimerApp.scala:30)
	at org.clulab.scala_transformers.encoder.timer.Timer.time(Timer.scala:11)
	at org.clulab.scala_transformers.encoder.timer.TokenClassifierTimerApp$TimedTokenClassifier.predict(TokenClassifierTimerApp.scala:29)
	at org.clulab.scala_transformers.encoder.timer.TokenClassifierTimerApp$.$anonfun$new$2(TokenClassifierTimerApp.scala:66)
	at org.clulab.scala_transformers.encoder.timer.TokenClassifierTimerApp$.$anonfun$new$2$adapted(TokenClassifierTimerApp.scala:61)
	at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
	at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
	at org.clulab.scala_transformers.encoder.timer.TokenClassifierTimerApp$.$anonfun$new$1(TokenClassifierTimerApp.scala:61)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.clulab.scala_transformers.encoder.timer.Timer.time(Timer.scala:11)
	at org.clulab.scala_transformers.encoder.timer.TokenClassifierTimerApp$.delayedEndpoint$org$clulab$scala_transformers$encoder$timer$TokenClassifierTimerApp$1(TokenClassifierTimerApp.scala:61)
	at org.clulab.scala_transformers.encoder.timer.TokenClassifierTimerApp$delayedInit$body.apply(TokenClassifierTimerApp.scala:8)
	at scala.Function0.apply$mcV$sp(Function0.scala:39)
	at scala.Function0.apply$mcV$sp$(Function0.scala:39)
	at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
	at scala.App.$anonfun$main$1$adapted(App.scala:80)
	at scala.collection.immutable.List.foreach(List.scala:431)
	at scala.App.main(App.scala:80)
	at scala.App.main$(App.scala:78)
	at org.clulab.scala_transformers.encoder.timer.TokenClassifierTimerApp$.main(TokenClassifierTimerApp.scala:8)
	at org.clulab.scala_transformers.encoder.timer.TokenClassifierTimerApp.main(TokenClassifierTimerApp.scala)

[MihaiSurdeanu]

I was wondering when this will come up... All transformers have a hard limit on the number of input tokens. This is probably longer than that. Ok to exclude for now. I will handle this in processors.