#13 read_file: say how to read a file over 1 MiB, and read it in slices

open ⎇ deleted fork:read-file-oversize-hint into ⎇ main +204 −5 3 files
  1. @okami.mom opened

    colab_read_file refuses a file whose content is over 1 MiB with "file exceeds 1 MiB; use Python to read a slice". a model that gets this has no size, no call to make, and no hint that base64 lowers the limit, so it has to work the slicing out on its own. one agent hit it reading a 3 MB file in 786432-byte chunks: each chunk's base64 came to exactly 1 MiB, every read failed, and its loop took the error text for file data. this pull request has two commits.

    say how to read a file over 1 MiB. the error now gives the file's size and the limit for the format, and the tool description and readme say how to read a bigger file. the phrase "exceeds 1 MiB" is kept.

    offset and length. colab_read_file takes offset and length, in bytes, and returns a slice:

    { path, format, size, offset, length, next_offset, content }
    

    next_offset is where the next slice starts and is null at the end of the file, so a client reads a file with a loop and does not need to run python for it. a base64 slice holds at most 786432 bytes, since 1 MiB of base64 is that many file bytes, and a text slice at most 1 MiB. a text slice is never cut inside a multi-byte character: it stops before one and next_offset says where to go on. an offset inside a character, past the end of the file, or a length over the limit is an error that says so. with neither argument the tool behaves as before, and the oversize error now points at offset and length.

    a file over about 6 MB still cannot be fetched this way, because jupyter sends it in one response and colab.js cuts responses off at 8 MiB. that error used to say only "colab response exceeded the 8 MiB limit"; it now says why and gives a colab_execute call that reads a slice from the kernel:

    import base64; f = open(PATH, "rb"); f.seek(OFFSET); print(base64.b64encode(f.read(600000)).decode())
    

    tests: the over-1-MiB test checks the size and the hint, a file that fits as text but not as base64 is covered, and there are new cases for slices of a 1,000,000-byte binary file that add up to the file, text slices on character boundaries, the bad ranges, the 8 MiB message and the new argument checks. npm test (85 tests), npm run check and prettier --check . pass.

  2. @okami.mom pushed 2 commits to ⎇ read-file-oversize-hint v0 +204 −5 3 files
  3. to comment, sign in