Adds TextStreamingDecoder

[keunwoochoi]

This is a specialized file opener + decoder that

• works for various types of sources (s3, gcs, local path)
• open any text file and stream the content line by line.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/data/1500

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 4 New Failures

As of commit 8832f44 with merge base 9295079 ():

NEW FAILURES - The following jobs have failed:

• Run Nodes Tests / test (windows-latest, 3.10) (gh)
  Process completed with exit code 1.
• Run Nodes Tests / test (windows-latest, 3.11) (gh)
  Process completed with exit code 1.
• Run Nodes Tests / test (windows-latest, 3.12) (gh)
  Process completed with exit code 1.
• Run Nodes Tests / test (windows-latest, 3.9) (gh)
  Process completed with exit code 1.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[ramanishsingh]

shall we also save METADATA_KEY?

[keunwoochoi]

i) like self.CURRENT_FILE_KEY, i added self.CURRENT_LINE_KEY.

ii) are you sure you meant saving METADATA_KEY? DATA_KEY and METADATA_KEY play similar roles which i don't think needs to be saved.

[keunwoochoi]

thanks for the review. made some changes & pushed three commits.

[keunwoochoi]

(note to myself)

(1018 durations < 0.005s hidden.  Use -vv to show these durations.)
=========================== short test summary info ===========================
FAILED test/nodes/io/test_text_streaming_decoder.py::test_text_stream_metadata - PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpcyw34h51\\test1.txt'
FAILED test/nodes/io/test_text_streaming_decoder.py::test_text_stream_state_management - PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpe4dio47s\\test1.txt'
FAILED test/nodes/io/test_text_streaming_decoder.py::test_text_stream_empty_file - PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpivu3sp7w\\normal.txt'
FAILED test/nodes/io/test_text_streaming_decoder.py::test_text_stream_encoding - PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmp73b21cs9\\utf8.txt'
FAILED test/nodes/io/test_text_streaming_decoder.py::test_error_handling - PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmp4oq398ov\\valid.txt'
===== 5 failed, 383 passed, 15 skipped, 27 warnings in 1320.71s (0:22:00) =====
Error: Process completed with exit code 1.

[keunwoochoi]

@ramanishsingh giving another shot with explicitly .shutdown()ing nodes.

[keunwoochoi]

hi @ramanishsingh, can we try the CI with the latest change?

also, i will have access to a windows laptop in a few days, in case the latest commit doesn't fix the issue.

[ramanishsingh]

hi @ramanishsingh, can we try the CI with the latest change?

also, i will have access to a windows laptop in a few days, in case the latest commit doesn't fix the issue.

Hi @keunwoochoi , somehow I can't run the CI. Trying to figure out.

[keunwoochoi]

i just tested the latest change with my window machine, and it passes the test! let's see how it goes with the CI.

[keunwoochoi]

a reminder ^ and also a question - is it going to be tested with the CI pipeline for Windows? (that was the only problem it used to have.)

[keunwoochoi]

👀

[ramanishsingh]

Hi @keunwoochoi ,
this repo has recently moved from pytorch to meta-pytorch and somehow the CIs are not running. We haven't been able to devote cycles for solving this yet.

[keunwoochoi]

oh i see. yes i noticed that, thanks for the explanation.

[keunwoochoi]

hi all! is there any update? 👀

[keunwoochoi]

is the CI fixed? cc' @aelavender
that'd be so nice.. i'm still waiting!!

[keunwoochoi]

Thanks for initiating the CI.

Actually, the previous issue of my PR is fixed. The new failure is not relevant to my change.

test/nodes/test_snapshot_store.py::TestQueueSnapshotStore::test_thread_dead_error FAILED [ 99%]

Can anyone have a look on this? Or perhaps simply re-run the test? cc' @aelavender