HF Hub Commit API isn't accepting LFS files

🤗Hub
1

Hello! I’m re-implementing the HF commit API (don’t ask) and I’m having troubles getting LFS files to be accepted as part of a commit.
I’ve managed to get the files preuploaded to the HF S3 buckets, and sent the completion request to the HF API.

I send

{
  oid: "7d34cce2c40a7089ea8b1d8ea9c25c573c46fcce7aa60579748118183a03f272",
  parts: [
    { part_number: 1, etag: "\"bbb437b7fe0d1d765cfe7c1af9156484\"" },
    { part_number: 2, etag: "\"dc335cf70c0ca26b4d1bf7059ad9226c\"" }
  ]
}

and the API sends me back {"success":true} (thank you API!)
however, when I try to call /api/models/username/repo/commit/main, with this payload (note the identical OID that the LFS multipart finalize endpoint has accepted as uploaded)

[
  {
    "key": "header",
    "value": {
      "summary": "Upload 20m_file.bin with hf_hub",
      "description": "lalalala"
    }
  },
  {
    "key": "lfsFile",
    "value": {
      "path": "20m_file.bin",
      "algo": "sha256",
      "oid": "7d34cce2c40a7089ea8b1d8ea9c25c573c46fcce7aa60579748118183a03f272",
      "size": 20970496
    }
  }
]

I get a 400 back from the API with {"error":"Your push was rejected because an LFS pointer pointed to a file that does not exist. For instance, this can happen if you used git push --no-verify to push your changes. Offending file: - 20m_file.bin"}.

Any ideas as to what I’m doing wrong?

2

Is it the same case as this one?

3

while this is the same error, the circumstances thru which it arose and can be troubleshooted are vastly different & doesn’t help me fix my issue at all :<

4

@pierric It’s a rare error. Any idea what it is?

5

celina fixed my issue !
Quote from her:

I took a quick look at your code, the issue might come from a mismatch in the size between the actual file uploaded to S3 and what it is sent in the verify request. in process_stream, you’re not taking the sample size in total_bytes, which makes the size off by 1024 bytes. you might also want to include the first 1024 bytes of the file in the SHA256 hash.

async fn process_stream<R>(mut reader: R, size: u64) -> io::Result<UploadInfo>
where
    R: AsyncRead + Unpin,
{
    let mut sample = vec![0u8; SAMPLE_SIZE.min(size as usize)];
    reader.read_exact(&mut sample).await?;

    let mut hasher = Sha256::new();
    hasher.update(&sample); // Hash the sample bytes too
    let mut total_bytes = sample.len() as u64; // Start with `sample` size
    let mut buffer = vec![0u8; CHUNK_SIZE];

    loop {
        let bytes_read = reader.read(&mut buffer).await?;
        if bytes_read == 0 {
            break;
        }
        hasher.update(&buffer[..bytes_read]);
        total_bytes += bytes_read as u64;
    }

    Ok(UploadInfo {
        size: total_bytes,
        sample,
        sha256: hasher.finalize().to_vec(),
    })
}

with these changes, I managed to upload lfs files with your script.
I hope this helped! :slightly_smiling_face: