Q:

Logical way to create a custom h5py build with conda

I am having an design issue with some packages in conda. I have done the following steps to get where I am:

Build custom version of hdf5 (enabling certain compiler flags)
Build custom version of h5py (with modifications calling different APIs from the custom version of hdf5. This can only be built properly using my custom version of hdf5.

So my question is:
I want to package this new package that I have built for use in development and production machines throughout our system and I want to know the best way of doing it. I have never made python|conda packages before so I am unaware of best practices.
Most of the documentation about the subject online seems to deal with letting conda build the package for you. Could I, for example, build h5py (from h5py modified source) in a build directory, and then add the meta and build files there for that to be my package. Or would I use the install directory in anaconda/lib/python3.5/site-packages/h5py-2*.egg as the source of my new package.
Once the package is made (with conda build command I assume), is it usually kept on a private server, or can I upload to the conda cloud.
I know this is a very open ended question, so any and all help is appreciated.

A:

I'll walk you through the steps to create your own hdf5 package
Make sure you are using the latest version of conda:
(root) [root@west-world hdf5]# conda update conda

Install the package conda-build
(root) [root@west-world hdf5]# conda install -y conda-build

Download the official repository of the recipes for packages present in the default channel
(root) [root@west-world tmp]# wget -qO- https://github.com/ContinuumIO/anaconda-recipes/archive/4.3.0.tar.gz | tar -xvz
(root) [root@west-world tmp]# cd anaconda-recipes-4.3.0/hdf5/
(root) [root@west-world hdf5]# ls
bld.bat  build.sh  meta.yaml

Edit the file build.sh to add your custom compiler flags
(root) [root@west-world hdf5]# vi build.sh

If the compiler flags need some more dependencies, then add them in the build subsection of the requirements section in the meta.yml file. Also, update the url to https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.8.17/src/hdf5-1.8.17.tar.gz. Since this is an old release, the URL has changed.
(root) [root@west-world hdf5]# vi meta.yaml

Time to start building the package. (You may need to install gcc using the system package manager)
(root) [root@west-world hdf5]# conda build .

If all goes smooth, the package will be built and you will see an output similar to:
# If you want to upload package(s) to anaconda.org later, type:

anaconda upload /conda/conda-bld/linux-64/hdf5-1.8.17-1.tar.bz2

# To have conda build upload to anaconda.org automatically, use
# $ conda config --set anaconda_upload yes

anaconda_upload is not set.  Not uploading wheels: []

####################################################################################
Source and build intermediates have been left in /conda/conda-bld.
There are currently 2 accumulated.
To remove them, you can run the ```conda build purge``` command

Install the anaconda-client package
(root) [root@west-world hdf5]# conda install -y anaconda-client

Visit https://anaconda.org/ and create an account there. Then upload the package you just created to your channel:
(root) [root@west-world hdf5]# anaconda upload /conda/conda-bld/linux-64/hdf5-1.8.17-1.tar.bz2
Using Anaconda API: https://api.anaconda.org
The action you are performing requires authentication, please sign in:
Using Anaconda API: https://api.anaconda.org
Username: nehaljwani
nehaljwani's Password: 
login successful
Using Anaconda API: https://api.anaconda.org
detecting package type ...
conda
extracting package attributes for upload ...
done

Uploading file nehaljwani/hdf5/1.8.17/linux-64/hdf5-1.8.17-1.tar.bz2 ... 
 uploaded 2003 of 2003Kb: 100.00% ETA: 0.0 minutes

Upload(s) Complete

Package located at:
https://anaconda.org/nehaljwani/hdf5

Now, next time you can use your channel to download and install the package from your channel
(root) [root@kun-lun ~]# conda install -c nehaljwani hdf5
Fetching package metadata ...........
Solving package specifications: .

Package plan for installation in environment /conda:

The following NEW packages will be INSTALLED:

    hdf5:  1.8.17-1      nehaljwani

Now, if you want to build h5py, the procedure is almost same, but you have to make sure that while you are building this package, the hdf5 package is taken from your channel. To do that, make sure your channel has the highest priority.
(root) [root@kun-lun ~]# conda config --prepend channels nehaljwani

Now, since you also need to make subtle modifications, you can create a patch and apply it during the build process by creating the patch in a file and adding it in the patches subsection in the meta.yml file, just like the one here: https://github.com/ContinuumIO/anaconda-recipes/blob/master/h5py/meta.yaml
For more info on how to build package, visit: https://conda.io/docs/build_tutorials/pkgs2.html#

