Standardized Build and Distributuon Conditions for Tauri + PyO3

[b-smiley]

Overview

This plugin has a viable future for the Tauri community, I have been long awaiting a Python plugin like this. However, I feel as though there is still work to be done to make this plugin production-ready. I would like to start the conversation about standardizing the Python + PyO3 + Tauri development and production environments such that it can be easily distributed to end-users. PyO3 supports more Python libraries (CPython compilation) than RustPython, which is better for guaranteeing Tauri builds. Therefore, PyO3 is only discussed in this issue.

I have started looking into how to modify the test build github action provided by tauri actions to support pyo3 and this plugin. However, I figured this plugin community should create a standardized development process that aligns with Tauri's CI/CD pipeline for app distribution before I continue.

Feature Requests

Here are some things I think we need to develop and document better. I would be willing to contribute to these.

• Local app development should use a python virtual environments by default.
  Using a venv is good practice in the Python community. The venv should be in a standardized location in the developer's app repository. Currently, I have mine external to the Tauri application, which I believe is the best for developing and distributing because it avoids copying issues inside GA.

  

  Inside main.py, I had to add the following:

    import sys
    import os
    
    if os.path.exists("../../venv/Lib/site-packages"):
        sys.path.append("../../venv/Lib/site-packages")
    else:
        print("Path does not exist, please check your virtual environment setup.", os.getcwd())

• GitHub Actions Workflow - Copying libpython, venv 3rd party packages, and other python requirements into the build distribution.

  I have been reading through the pyo3 documentation, tauri documentation, and this plugin's documentation, but it has been a challenge to get the test build github action to work for all the OSs. The problem is that each OS uses venv and libpython differently (libpython3.10.dll vs. libpython3.10.dylib). Also, how would 3rd party python packages get properly distributed?

• Source Code Protection

  I recognize that source code hiding can still be cracked, but storing the Python files in plain text in distribution is something I would like to avoid. I think this should at least be an option for developers to add to their applications. Give the developer the decision to open-source or close-source their application. Thus, I had 3 different ideas in mind for source code hiding:

  1. Obfuscate the .py files in GitHub Action using pyarmor OR another library.
  2. Encrypt .py files to prevent plain text snooping, then decrypt with a symmetric key stored with keyring on application boot.

     // py_lib_pyo3.rs
     // TODO Instead of passing in code:String as plain text, decrypt it first with keyring
     pub fn run_python_internal(code: String, _filename: String) -> crate::Result<()> {
         marker::Python::with_gil(|py| -> crate::Result<()> {
             let globals = GLOBALS.lock().unwrap().clone_ref(py).into_bound(py);
             let c_code = CString::new(code).expect("CString::new failed");
             Ok(py.run(&c_code, Some(&globals), None)?)
         })
     }

  3. Any other methods!

[marcomq]

Hi!

This plugin has a viable future for the Tauri community, I have been long awaiting a Python plugin like this.

Thx!

However, I feel as though there is still work to be done to make this plugin production-ready.

You are completely right. Currently, I have other priorities and I'm not sure when I will find time to fine tune this plugin for prod.

However, I figured this plugin community should create a standardized development process that aligns with Tauri's CI/CD pipeline for app distribution before I continue.

Sounds good. I also wanted to add testing soon. Having a CI/CD pipeline that also compiles the final binary might also be a part of this to detect issues early.

Local app development should use a python virtual environments by default.

Totally agree. A lot of python libs require venv. I already added some partial venv support for the python-src folder.
I didn't thought about using the standard app folder... Maybe this makes more sense... Not sure yet.

Edit:
To make the venv work with python-src, the venv folder needs to be called .venv
https://github.com/marcomq/tauri-plugin-python/blob/main/src/lib.rs#L103

GitHub Actions Workflow - Copying libpython, venv 3rd party packages, and other python requirements into the build distribution.

Yap. Agree - this is important. First idea for venv dependencies was too use the src-python folder and also deploy it with the final application. I stil have no good idea yet for the python lib folder. I guess for .rpm and .deb, you may just use python as dependency, so that it gets installed. Not sure yet how to handle Win and MacOs. Those are even more important...
Maybe for those, we need to bundle the dll/dylib/so together with the executable.

There are other tools that can bundle a whole python applications into a single executable. Something like this would be great - but those usually just bundle the interpreter, not the python library. So, maybe bundling the dll/dylib would be a good solution.

Source Code Protection

I guess for some apps, delivering the source code might be fine. Especially for those that use open source. The GPL for example requires that the source code is readable and changeable. So if your are using (L)GPL python, you may have to deliver the code anyways and also need to make sure that it can be changed easily.

But for normal apps - yes - it doesn't look professional to have a plain readable python file there that also can be modified by anyone.
Maybe it might be possible to have some .pyd file(s)... An encrypted bundle would also be good, but I have no idea how these can be imported again into python code via import .... It would also be great to have the python code compiled into the binary, similar to the UI code. But not sure how to make the python code then runnable again. I'm open for any good solution.

It would just be great if the development version re-updates quickly if there is a change of the python file. And there should be some good solution to debug the python code in development mode. Maybe also in release mode, maybe by having some additional parameter.

[b-smiley]

Hey Marco!

Sounds good. I also wanted to add testing soon. Having a CI/CD pipeline that also compiles the final binary might also be a part of this to detect issues early.

Yes, plugin testing is also a good idea. I will continue to tinker around with a CI/CD pipeline, and I will let you know if I make any progress. For my own application, I was thinking of having the application testing and build pipelines separated as follows:

• The testing pipeline runs (pytest, vitest, etc.) whenever a PR is made to the main branch.
• The building pipeline runs on merge to main. This is the pipeline I will continue to prioritize. Ideally, people using this plugin could just copy and paste the build.yaml into their application repository and use it.

Edit: To make the venv work with python-src, the venv folder needs to be called .venv https://github.com/marcomq/tauri-plugin-python/blob/main/src/lib.rs#L103

Sure, I can rename it to .venv and enable the venv feature flag. I will report back if I have any issues configuring this.

Yap. Agree - this is important. First idea for venv dependencies was too use the src-python folder and also deploy it with the final application. I stil have no good idea yet for the python lib folder. I guess for .rpm and .deb, you may just use python as dependency, so that it gets installed. Not sure yet how to handle Win and MacOs. Those are even more important... Maybe for those, we need to bundle the dll/dylib/so together with the executable.

Agreed. The dll/dylib/so are version-dependent, which is important for certain Python libraries inside a venv. I have had to downgrade Python versions in the past because the library I needed only works with older Python versions.

As a potential solution, I was thinking the GitHub Actions pipeline could have a PythonVersion parameter which downloads and creates a versioned venv for building. PyO3 can auto-detect the activated venv, which we could use to configure proper linkage to Python dependencies during the build, and then these would be used during app runtime. I think the .pyd discussion below would also tie into this solution ⬇

There are other tools that can bundle a whole python applications into a single executable. Something like this would be great - but those usually just bundle the interpreter, not the python library. So, maybe bundling the dll/dylib would be a good solution.
I guess for some apps, delivering the source code might be fine. Especially for those that use open source. The GPL for example requires that the source code is readable and changeable. So if your are using (L)GPL python, you may have to deliver the code anyways and also need to make sure that it can be changed easily.
But for normal apps - yes - it doesn't look professional to have a plain readable python file there that also can be modified by anyone. Maybe it might be possible to have some .pyd file(s)... An encrypted bundle would also be good, but I have no idea how these can be imported again into python code via import .... It would also be great to have the python code compiled into the binary, similar to the UI code. But not sure how to make the python code then runnable again. I'm open for any good solution.
It would just be great if the development version re-updates quickly if there is a change of the python file. And there should be some good solution to debug the python code in development mode. Maybe also in release mode, maybe by having some additional parameter.

I will do some more research on each solution. I will see if I can get one of them to work.

• Shared Libraries: It looks like pyo3 can handle .pyd and .so files, which may be a solution that allows for both source code hiding during release and regular plain text .py files during development.
• Compiled Binary: This solution seems easier to handle on the distribution side, also a good solution!

Thanks for the conversation! I'll add more to this thread/PR if I make progress.

[Obhenimen]

Everything works in development. But when i build the release version. all python functions that calls a pip module like numpy etc does not work. But if the code does not depends on any external library, it works. Is there any quick fix to make it work in production. I am on a mac os machine

[marcomq]

@Obhenimen

Good question. I wasn't aware of this. There is also some major issue on windows prod, so that currently no python code at all can be called on windows prod.
The current workaround for windows is to enable the console. Hiding the console seems to cause issues on. Windows. Maybe this is similar on macos. But maybe you just missed something to add as tauri "resources". This also happens fast.

It is kind of annoying to debug production issues with tauri...

[Obhenimen]

I can share my repo. I am using a mac os and it seems to work fine in development. But does not work in production
https://github.com/pjohn71/ui-with-tauri.git

[marcomq]

@Obhenimen
I kind of liked the idea of using numpy in tauri.
I checked out your branch and created a PR. By this, I don't have to try to use numpy by myself. :D And it is kind of a good test of using a .venv

Have fun using tauri

[Obhenimen]

I was able to fix the issue. Moving the .venv to the src-python directory fixes the issue. Thanks for this repo

[marcomq]

GitHub Actions Workflow - Copying libpython, venv 3rd party packages, and other python requirements into the build distribution.

I'm just thinking about switching from plain pyo3 to pyoxidizer+pyo3 with pyembed https://docs.rs/pyembed/latest/pyembed/
This would link libpython statically, so the target system doesn't even need to have python installed and no dynamic libs need to be shipped.
I created some small prototype locally and this looked promising. It is a bit complicated to use, so I'm not sure how long this may take to have it in an automation from a tauri plugin. It wouldn't solve the issue with the python source code or the .venv, but this could hopefully be solved differently.

[Obhenimen]

Can you share the small prototype you created so I can have a look. It might be able to guide me properly on the next steps to take.

[marcomq]

@Obhenimen
I just used this simple tutorial here:
https://pyoxidizer.readthedocs.io/en/latest/pyoxidizer_rust_generic_embedding.html#embed-python-with-pyembed
The function signature seems to by PyO3 compatible. It would just require the pyoxidizer pip package and a venv.

[marcomq]

Status Update:
Not sure yet if I should really use Pyembed.

• Pyembed didn't publish a new release in the last 2 years, the original developer is looking for a new maintainer.
• The old embedded pyo3 library has a different function signature and I'm not sure if it can perform similar type conversions
• There are issues with rust analyzer and pyembed, partially due to the different pyo3 versions and partially due to the build.rs script.

It might be better to just try to ship a shared library...

[b-smiley]

Status Update: Not sure yet if I should really use Pyembed.

• Pyembed didn't publish a new release in the last 2 years, the original developer is looking for a new maintainer.
• The old embedded pyo3 library has a different function signature and I'm not sure if it can perform similar type conversions
• There are issues with rust analyzer and pyembed, partially due to the different pyo3 versions and partially due to the build.rs script.

It might be better to just try to ship a shared library...

Okay, thanks for the update. I'm currently working on a shared library solution, I'll see if I can get something functional.

[b-smiley]

Status Update: Not sure yet if I should really use Pyembed.

• Pyembed didn't publish a new release in the last 2 years, the original developer is looking for a new maintainer.
• The old embedded pyo3 library has a different function signature and I'm not sure if it can perform similar type conversions
• There are issues with rust analyzer and pyembed, partially due to the different pyo3 versions and partially due to the build.rs script.

It might be better to just try to ship a shared library...

It looks like the shared library route would also have to also include the python.exe and the PythonXY/Lib folder for modules like json, logging, etc. Thus is looks like at this point this solution would just copy an entire python installation onto the end users machine.

I did look into PyOxidizer as well and personally I think this is the best solution. It abstracts some of the distribution details assiocated with the shared libraries.

I created some small prototype locally and this looked promising. It is a bit complicated to use, so I'm not sure how long this may take to have it in an automation from a tauri plugin. It wouldn't solve the issue with the python source code or the .venv, but this could hopefully be solved differently.

Would you be able to publish a branch with this prototype on it?

Thanks!

[marcomq]

I already sent the link to all I have. I wasn't able to combine pyoxidizer with the tauri plugin and get some compilable state. It would probably be required to modify pyembed so that it doesn't include pyo3 or to update the pyo3 version in pyembed. Also, if those packages still require python.exe and PythonXY/Lib, then they will still require this with pyembed, as pyembed mostly just performs static linking. Unfortunately, I will also not be able to work on this for at least the next 2 weeks...

[marcomq]

I found a recent fork of pyembed and just created some sample branch of it:
mm/pyembed

Not 100% sure yet about it. It needs pyoxidizer to be installed and also requires to set an environment variable. So definitely harder to build. Also, it still seems to make trouble with rust analyzer.