#14 read_file: offset and length read a large file in slices

closed into ⎇ read-file-oversize-hint
  1. @okami.mom opened

    this builds on the first pull request (read_file: say how to read a file over 1 MiB, 3my2jsfk7xb22) and should go in after it. it changes the message that one adds, so the patch here is only this commit.

    colab_read_file could not read a file over 1 MiB at all. it now takes offset and length, in bytes, and returns a slice:

    { path, format, size, offset, length, next_offset, content }
    

    next_offset is where the next slice starts and is null at the end of the file, so a client reads a file with a loop and does not need to run python for it. a base64 slice holds at most 786432 bytes, since 1 MiB of base64 is that many file bytes, and a text slice at most 1 MiB. a text slice is never cut inside a multi-byte character: it stops before one and next_offset says where to go on. an offset inside a character, past the end of the file, or a length over the limit is an error that says so. with neither argument the tool behaves as before.

    the oversize error from the first pull request now points at offset and length. a file over about 6 MB still cannot be fetched this way, because jupyter sends it in one response and colab.js cuts responses off at 8 MiB. that error used to say only "colab response exceeded the 8 MiB limit"; it now says why and gives the colab_execute call that reads a slice from the kernel. the tool description and the readme say the same.

    tests: slices of a 1,000,000-byte binary file that add up to the file, text slices on character boundaries, the bad ranges, the 8 MiB message, and the new argument checks. five of them fail without this commit. npm test (85 tests), npm run check and prettier --check . pass.

  2. this pull request has no versions submitted yet.

  3. closed this
  4. to comment, sign in