Add MI-GAN fp16 exports (XNNPACK, Core ML, Vulkan)

export-scripts !31 @ 9e0ed87. XNNPACK/Vulkan: ET fork f7bd426 (= v0.10.7-libs). Core ML: ET 1.4.1 (PyPI).

.gitattributes

@@ -33,3 +33,12 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+coreml/256/migan_256_coreml_fp16.pte filter=lfs diff=lfs merge=lfs -text
+coreml/512/migan_512_coreml_fp16.pte filter=lfs diff=lfs merge=lfs -text
+coreml/faces_256/migan_faces_256_coreml_fp16.pte filter=lfs diff=lfs merge=lfs -text
+vulkan/256/migan_256_vulkan_fp16.pte filter=lfs diff=lfs merge=lfs -text
+vulkan/512/migan_512_vulkan_fp16.pte filter=lfs diff=lfs merge=lfs -text
+vulkan/faces_256/migan_faces_256_vulkan_fp16.pte filter=lfs diff=lfs merge=lfs -text
+xnnpack/256/migan_256_xnnpack_fp16.pte filter=lfs diff=lfs merge=lfs -text
+xnnpack/512/migan_512_xnnpack_fp16.pte filter=lfs diff=lfs merge=lfs -text
+xnnpack/faces_256/migan_faces_256_xnnpack_fp16.pte filter=lfs diff=lfs merge=lfs -text

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 Picsart AI Research (PAIR)
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,81 @@
+---
+license: mit
+pipeline_tag: image-to-image
+library_name: executorch
+tags:
+  - inpainting
+---
+
+# migan
+
+This repository hosts **MI-GAN** (image inpainting) exported for the
+[React Native ExecuTorch](https://www.npmjs.com/package/react-native-executorch)
+library as ExecuTorch `.pte` programs, ready to run on device.
+
+Upstream model: [Picsart-AI-Research/MI-GAN](https://github.com/Picsart-AI-Research/MI-GAN),
+revision `2b793c5ece43f4253e32d4afc257120a5deed6f5`, checkpoints
+`migan_256_places2.pt`, `migan_512_places2.pt` (Places2, scenes) and
+`migan_256_ffhq.pt` (FFHQ, faces).
+
+## Inputs and outputs
+
+`forward(image, mask) -> image`, all `float32`:
+
+- `image` `[1, 3, H, W]`, RGB in `[0, 1]`
+- `mask` `[1, 1, H, W]`, `1` = known pixel, `0` = hole
+- output `[1, 3, H, W]`, RGB in `[0, 1]`, the whole frame with the hole filled
+
+`H = W = 256` or `512`. Cropping around the hole and blending the result back
+into the full-resolution photo happen outside the model, as in upstream's
+`scripts/create_onnx_pipeline.py`.
+
+## Variants
+
+| Path | Backend | Precision | Checkpoint |
+| --- | --- | --- | --- |
+| `coreml/256/migan_256_coreml_fp16.pte` | coreml | fp16 | Places2 256 |
+| `coreml/512/migan_512_coreml_fp16.pte` | coreml | fp16 | Places2 512 |
+| `coreml/faces_256/migan_faces_256_coreml_fp16.pte` | coreml | fp16 | FFHQ 256 |
+| `vulkan/256/migan_256_vulkan_fp16.pte` | vulkan | fp16 | Places2 256 |
+| `vulkan/512/migan_512_vulkan_fp16.pte` | vulkan | fp16 | Places2 512 |
+| `vulkan/faces_256/migan_faces_256_vulkan_fp16.pte` | vulkan | fp16 | FFHQ 256 |
+| `xnnpack/256/migan_256_xnnpack_fp16.pte` | xnnpack | fp16 | Places2 256 |
+| `xnnpack/512/migan_512_xnnpack_fp16.pte` | xnnpack | fp16 | Places2 512 |
+| `xnnpack/faces_256/migan_faces_256_xnnpack_fp16.pte` | xnnpack | fp16 | FFHQ 256 |
+
+## Repository structure
+
+```
+config.json                                          27 B
+coreml/256/config.json                               1171 B
+coreml/256/migan_256_coreml_fp16.pte                 12.8 MB
+coreml/512/config.json                               1171 B
+coreml/512/migan_512_coreml_fp16.pte                 14.0 MB
+coreml/faces_256/config.json                         1177 B
+coreml/faces_256/migan_faces_256_coreml_fp16.pte     12.8 MB
+vulkan/256/config.json                               1171 B
+vulkan/256/migan_256_vulkan_fp16.pte                 12.4 MB
+vulkan/512/config.json                               1171 B
+vulkan/512/migan_512_vulkan_fp16.pte                 14.1 MB
+vulkan/faces_256/config.json                         1177 B
+vulkan/faces_256/migan_faces_256_vulkan_fp16.pte     12.4 MB
+xnnpack/256/config.json                              1173 B
+xnnpack/256/migan_256_xnnpack_fp16.pte               24.0 MB
+xnnpack/512/config.json                              1173 B
+xnnpack/512/migan_512_xnnpack_fp16.pte               25.2 MB
+xnnpack/faces_256/config.json                        1179 B
+xnnpack/faces_256/migan_faces_256_xnnpack_fp16.pte   24.0 MB
+```
+
+## Compatibility
+
+These files are published for the **ExecuTorch v1.4.1** runtime. The XNNPACK
+and Vulkan files need the React Native ExecuTorch native libraries
+`v0.10.7-libs` or newer. ExecuTorch gives no forward compatibility guarantee,
+so an older runtime may fail to load them.
+
+For React Native ExecuTorch model loading, see the
+[documentation](https://docs.swmansion.com/react-native-executorch/docs/fundamentals/downloading-models).
+To load these files in your own runtime, read the
+[compatibility note](https://github.com/pytorch/executorch/blob/main/runtime/COMPATIBILITY.md)
+first.

config.json

@@ -0,0 +1,3 @@
+{
+  "modelName": "migan"
+}

coreml/256/config.json

@@ -0,0 +1,56 @@
+{
+  "$schema": "https://huggingface.co/software-mansion/react-native-executorch-spec/resolve/main/config.schema.json",
+  "model": "migan",
+  "family": "migan",
+  "capabilities": [
+    "image-inpainting"
+  ],
+  "backend": "coreml",
+  "license": "mit",
+  "variants": [
+    {
+      "file": "migan_256_coreml_fp16.pte",
+      "precision": "fp16",
+      "quantized": false,
+      "default": true,
+      "methods": {
+        "forward": {
+          "inputs": [
+            {
+              "name": "image",
+              "shape": [
+                1,
+                3,
+                256,
+                256
+              ],
+              "dtype": "float32"
+            },
+            {
+              "name": "mask",
+              "shape": [
+                1,
+                1,
+                256,
+                256
+              ],
+              "dtype": "float32"
+            }
+          ],
+          "outputs": [
+            {
+              "name": "image",
+              "shape": [
+                1,
+                3,
+                256,
+                256
+              ],
+              "dtype": "float32"
+            }
+          ]
+        }
+      }
+    }
+  ]
+}

coreml/256/migan_256_coreml_fp16.pte

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d01df747c4b2a7b7a509ce288996f77f9cd30e9dfd8244234d6843c3f3905420
+size 12769586

coreml/512/config.json

@@ -0,0 +1,56 @@
+{
+  "$schema": "https://huggingface.co/software-mansion/react-native-executorch-spec/resolve/main/config.schema.json",
+  "model": "migan",
+  "family": "migan",
+  "capabilities": [
+    "image-inpainting"
+  ],
+  "backend": "coreml",
+  "license": "mit",
+  "variants": [
+    {
+      "file": "migan_512_coreml_fp16.pte",
+      "precision": "fp16",
+      "quantized": false,
+      "default": true,
+      "methods": {
+        "forward": {
+          "inputs": [
+            {
+              "name": "image",
+              "shape": [
+                1,
+                3,
+                512,
+                512
+              ],
+              "dtype": "float32"
+            },
+            {
+              "name": "mask",
+              "shape": [
+                1,
+                1,
+                512,
+                512
+              ],
+              "dtype": "float32"
+            }
+          ],
+          "outputs": [
+            {
+              "name": "image",
+              "shape": [
+                1,
+                3,
+                512,
+                512
+              ],
+              "dtype": "float32"
+            }
+          ]
+        }
+      }
+    }
+  ]
+}

coreml/512/migan_512_coreml_fp16.pte

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e5e064588bb9ac7cd99b38bcfc0b56a16c40468526101aa9fdfcaaf53ff9a898
+size 13968400

coreml/faces_256/config.json

@@ -0,0 +1,56 @@
+{
+  "$schema": "https://huggingface.co/software-mansion/react-native-executorch-spec/resolve/main/config.schema.json",
+  "model": "migan",
+  "family": "migan",
+  "capabilities": [
+    "image-inpainting"
+  ],
+  "backend": "coreml",
+  "license": "mit",
+  "variants": [
+    {
+      "file": "migan_faces_256_coreml_fp16.pte",
+      "precision": "fp16",
+      "quantized": false,
+      "default": true,
+      "methods": {
+        "forward": {
+          "inputs": [
+            {
+              "name": "image",
+              "shape": [
+                1,
+                3,
+                256,
+                256
+              ],
+              "dtype": "float32"
+            },
+            {
+              "name": "mask",
+              "shape": [
+                1,
+                1,
+                256,
+                256
+              ],
+              "dtype": "float32"
+            }
+          ],
+          "outputs": [
+            {
+              "name": "image",
+              "shape": [
+                1,
+                3,
+                256,
+                256
+              ],
+              "dtype": "float32"
+            }
+          ]
+        }
+      }
+    }
+  ]
+}

coreml/faces_256/migan_faces_256_coreml_fp16.pte

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:93a0ac1cccaff61031780df57ca24ca21f6d2424cfd0cd6952c29bccafbdebe8
+size 12769586

vulkan/256/config.json

@@ -0,0 +1,56 @@
+{
+  "$schema": "https://huggingface.co/software-mansion/react-native-executorch-spec/resolve/main/config.schema.json",
+  "model": "migan",
+  "family": "migan",
+  "capabilities": [
+    "image-inpainting"
+  ],
+  "backend": "vulkan",
+  "license": "mit",
+  "variants": [
+    {
+      "file": "migan_256_vulkan_fp16.pte",
+      "precision": "fp16",
+      "quantized": false,
+      "default": true,
+      "methods": {
+        "forward": {
+          "inputs": [
+            {
+              "name": "image",
+              "shape": [
+                1,
+                3,
+                256,
+                256
+              ],
+              "dtype": "float32"
+            },
+            {
+              "name": "mask",
+              "shape": [
+                1,
+                1,
+                256,
+                256
+              ],
+              "dtype": "float32"
+            }
+          ],
+          "outputs": [
+            {
+              "name": "image",
+              "shape": [
+                1,
+                3,
+                256,
+                256
+              ],
+              "dtype": "float32"
+            }
+          ]
+        }
+      }
+    }
+  ]
+}

vulkan/256/migan_256_vulkan_fp16.pte

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:606ae78feec306ced13544d53f18e3ceb3d63efafc277a45c2129fd7473475c4
+size 12411650

vulkan/512/config.json

@@ -0,0 +1,56 @@
+{
+  "$schema": "https://huggingface.co/software-mansion/react-native-executorch-spec/resolve/main/config.schema.json",
+  "model": "migan",
+  "family": "migan",
+  "capabilities": [
+    "image-inpainting"
+  ],
+  "backend": "vulkan",
+  "license": "mit",
+  "variants": [
+    {
+      "file": "migan_512_vulkan_fp16.pte",
+      "precision": "fp16",
+      "quantized": false,
+      "default": true,
+      "methods": {
+        "forward": {
+          "inputs": [
+            {
+              "name": "image",
+              "shape": [
+                1,
+                3,
+                512,
+                512
+              ],
+              "dtype": "float32"
+            },
+            {
+              "name": "mask",
+              "shape": [
+                1,
+                1,
+                512,
+                512
+              ],
+              "dtype": "float32"
+            }
+          ],
+          "outputs": [
+            {
+              "name": "image",
+              "shape": [
+                1,
+                3,
+                512,
+                512
+              ],
+              "dtype": "float32"
+            }
+          ]
+        }
+      }
+    }
+  ]
+}

vulkan/512/migan_512_vulkan_fp16.pte

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d263e2eae6d2f3d650abad61a9e0d30ba39df85ce5f7824e7ed03477abe13a89
+size 14058114

vulkan/faces_256/config.json

@@ -0,0 +1,56 @@
+{
+  "$schema": "https://huggingface.co/software-mansion/react-native-executorch-spec/resolve/main/config.schema.json",
+  "model": "migan",
+  "family": "migan",
+  "capabilities": [
+    "image-inpainting"
+  ],
+  "backend": "vulkan",
+  "license": "mit",
+  "variants": [
+    {
+      "file": "migan_faces_256_vulkan_fp16.pte",
+      "precision": "fp16",
+      "quantized": false,
+      "default": true,
+      "methods": {
+        "forward": {
+          "inputs": [
+            {
+              "name": "image",
+              "shape": [
+                1,
+                3,
+                256,
+                256
+              ],
+              "dtype": "float32"
+            },
+            {
+              "name": "mask",
+              "shape": [
+                1,
+                1,
+                256,
+                256
+              ],
+              "dtype": "float32"
+            }
+          ],
+          "outputs": [
+            {
+              "name": "image",
+              "shape": [
+                1,
+                3,
+                256,
+                256
+              ],
+              "dtype": "float32"
+            }
+          ]
+        }
+      }
+    }
+  ]
+}

vulkan/faces_256/migan_faces_256_vulkan_fp16.pte

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:33290cb5418d1a6b1a54849e91e9df9f5908e5e8336d6d2291d4c8a988619234
+size 12411650

xnnpack/256/config.json

@@ -0,0 +1,56 @@
+{
+  "$schema": "https://huggingface.co/software-mansion/react-native-executorch-spec/resolve/main/config.schema.json",
+  "model": "migan",
+  "family": "migan",
+  "capabilities": [
+    "image-inpainting"
+  ],
+  "backend": "xnnpack",
+  "license": "mit",
+  "variants": [
+    {
+      "file": "migan_256_xnnpack_fp16.pte",
+      "precision": "fp16",
+      "quantized": false,
+      "default": true,
+      "methods": {
+        "forward": {
+          "inputs": [
+            {
+              "name": "image",
+              "shape": [
+                1,
+                3,
+                256,
+                256
+              ],
+              "dtype": "float32"
+            },
+            {
+              "name": "mask",
+              "shape": [
+                1,
+                1,
+                256,
+                256
+              ],
+              "dtype": "float32"
+            }
+          ],
+          "outputs": [
+            {
+              "name": "image",
+              "shape": [
+                1,
+                3,
+                256,
+                256
+              ],
+              "dtype": "float32"
+            }
+          ]
+        }
+      }
+    }
+  ]
+}

xnnpack/256/migan_256_xnnpack_fp16.pte

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2617a222f44d735a20a76ede65d322cab6bf843d398e29be7cf0f18284a30b15
+size 24002700

xnnpack/512/config.json

@@ -0,0 +1,56 @@
+{
+  "$schema": "https://huggingface.co/software-mansion/react-native-executorch-spec/resolve/main/config.schema.json",
+  "model": "migan",
+  "family": "migan",
+  "capabilities": [
+    "image-inpainting"
+  ],
+  "backend": "xnnpack",
+  "license": "mit",
+  "variants": [
+    {
+      "file": "migan_512_xnnpack_fp16.pte",
+      "precision": "fp16",
+      "quantized": false,
+      "default": true,
+      "methods": {
+        "forward": {
+          "inputs": [
+            {
+              "name": "image",
+              "shape": [
+                1,
+                3,
+                512,
+                512
+              ],
+              "dtype": "float32"
+            },
+            {
+              "name": "mask",
+              "shape": [
+                1,
+                1,
+                512,
+                512
+              ],
+              "dtype": "float32"
+            }
+          ],
+          "outputs": [
+            {
+              "name": "image",
+              "shape": [
+                1,
+                3,
+                512,
+                512
+              ],
+              "dtype": "float32"
+            }
+          ]
+        }
+      }
+    }
+  ]
+}

xnnpack/512/migan_512_xnnpack_fp16.pte

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:34b1f24e703c54787e657a987604907343e40c1edf6d8a1b402f24b0a651c25e
+size 25181068

xnnpack/faces_256/config.json

@@ -0,0 +1,56 @@
+{
+  "$schema": "https://huggingface.co/software-mansion/react-native-executorch-spec/resolve/main/config.schema.json",
+  "model": "migan",
+  "family": "migan",
+  "capabilities": [
+    "image-inpainting"
+  ],
+  "backend": "xnnpack",
+  "license": "mit",
+  "variants": [
+    {
+      "file": "migan_faces_256_xnnpack_fp16.pte",
+      "precision": "fp16",
+      "quantized": false,
+      "default": true,
+      "methods": {
+        "forward": {
+          "inputs": [
+            {
+              "name": "image",
+              "shape": [
+                1,
+                3,
+                256,
+                256
+              ],
+              "dtype": "float32"
+            },
+            {
+              "name": "mask",
+              "shape": [
+                1,
+                1,
+                256,
+                256
+              ],
+              "dtype": "float32"
+            }
+          ],
+          "outputs": [
+            {
+              "name": "image",
+              "shape": [
+                1,
+                3,
+                256,
+                256
+              ],
+              "dtype": "float32"
+            }
+          ]
+        }
+      }
+    }
+  ]
+}

xnnpack/faces_256/migan_faces_256_xnnpack_fp16.pte

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eaa102e4c52b3a6f6c3aacc07800774452b403f46ac3dcfbdf359258db7621aa
+size 24002700