Consolidate installer setup steps

[MCGallaspy]

All the steps we take before releasing an installer should be as consolidated as possible, so that we don't duplicate effort across OSes. This issue attempts to document the current state and make a roadmap to consolidation.

@benjaoming made this document, which at the time of this posting is a bit out of date.

The steps I take to create the Windows installer are documented here: https://github.com/learningequality/installers/blob/master/windows/README.md#instructions-to-build-kalitesetupexe
This issue is related: #48

In particular tasks like including documentation, running node build.js, including assessment items, and running kalite manage collectstatic are platform-agnostic. If I understand it correctly, then we can make these and other pre-release tasks the responsibility of setup.py, so that they can be run with a command like python setup.py [your cool command name]. @aronasorman this is the main point of our discussion from 9/4 dev meeting.

Roadmap

• Update description of platform-independent prerelease tasks in this document
• Implement above tasks in setup.py
• Use setuptools for Windows installer (see Windows installer should use setup.py (setuptools) #48)
• Use setuptools for Ubuntu/Debian and OSX (I think we already do, right?)

Milestone for this is up for discussion -- if these changes for each individual task are safe, then we should aim to include them in 0.15. If it will require more extensive work/testing then we should consider it for 0.16.

[MCGallaspy]

@cpauya your input appreciated here too!

[jamalex]

Another example of something we want to do during this "pre-build" stage is: #80

[jamalex]

If I understand it correctly, then we can make these and other pre-release tasks the responsibility of setup.py, so that they can be run with a command like python setup.py [your cool command name]

We should look (in a thorough way) at what the best tool/stage in the pipeline is for doing this "pre-build" step. Another option to examine, for example, would be to use node.js (since we need it for the build process anyway) to coordinate the early stages of the build process.

The overall goal is to build a standard output format (e.g. a zip file, or target "built" directory), with all of the common cross-platform build steps completed, that can then serve as input to the various installers.

[jamalex]

Update description of platform-independent prerelease tasks in this document

This document currently seems focused on the steps to run for prepping things and building/uploading/releasing particular installers, but there's no list of the "pre-build" tasks that we want to automate. @MCGallaspy listed some in the issue description above -- I'll add a section in the doc where we can list this out more fully.

[benjaoming]

The overall goal is to build a standard output format (e.g. a zip file, or target "built" directory), with all of the common cross-platform build steps completed, that can then serve as input to the various installers.

This is the source distribution or in short, sdist :)

It's here, .zip available, too: https://pypi.python.org/pypi/ka-lite-static

[jamalex]

Yes, the sdist zip was what I had in mind as a possible output format here (or at least, one component of it), but I don't know enough about it to know how well it will handle everything we need -- e.g. "bundling in an empty database that gets unpacked to the user directory, but only if there's no DB in the user directory yet".

I added a preliminary list of things we want to do when building from source -> distributable bundle (to use as input to installers). @rtibbles @aronasorman @MCGallaspy @benjaoming, please add things to there as you see fit, so we can assess and discuss:
https://docs.google.com/document/d/1Z06ING-sgMoq5EqZPVK2Zu1cUIpF9SKJunG17Q8Fcr8/edit#heading=h.q1tnequspos4

[benjaoming]

"bundling in an empty database that gets unpacked to the user directory, but only if there's no DB in the user directory yet"

It would have to be a data file that gets installed unconditionally, and then it would have to be the application (kalite manage setup) that makes the assessment of whether or not to bootstrap the user .kalite directory with an empty database. But this could happen several times within the same installation.

One possible gotcha: Downgrading KA Lite. But I think this is a problem already, anyways. So no need to deal with it.

[cpauya]

The OSX installer has a setup.sh script which does everything to build the .app and generate the .dmg (installer).

Just a note that the OSX .app uses Pyrun and not virtualenv yet. But we already have an issue to drop pyrun and use virtualenv which needs to be tested if feasible for OSX.

The script downloads an archive of a ka-lite branch and will run python setup.py install on it.

These are the pre-release steps that the setup script must do so particular packages/data are included in the bundled ka-lite or in the Pyrun's site-packages folder:

1. Download the assessment.zip. -- This is extracted by the .app on the installed machine upon first run.
2. Run pip install requirements.txt.
3. Build the docs
   1. Run pip install -r requirements_sphinx.txt.
   2. Run sphinx-build inside the docs directory.
4. Compile yml to json
   1. Run pip install pyyaml==3.11
   2. Run kalite manage compileymltojson
   3. Run pip uninstall pyyaml

When successful, these are copied into the .appsResources` folder and the .app and it's .dmg are then built:

1. assessment.zip file
2. the Pyrun directory (which includes ka-lite and other modules in it'ssite-packages`)

I'm sure the above can be streamlined, so I'd also definitely like these steps to be consolidated so that the install/build script for the .app will only do the OS-specific tasks.

I have to ask though how is the assessment.zip included in the other OS installers? Is it also as an .zip archive and later extracted on the installed machine? If so, then we can consolidate that into the pre-release steps but I don't know how setup.py can do that or for any other non-package inclusions? Perhaps the same data location if we also include a pre-migrated empty database?

[benjaoming]

@cpauya please don't build the docs, we can include them in the sdist

All these steps can be made redundant if you use the sdist:

# 4. Download KA-Lite zip based on develop branch.
# 5. Extract KA-Lite and move into `ka-lite` folder.
# 6. Make ka-lite repo production-ready (delete .KALITE_SOURCE_DIR, etc).
# 7. Run pyrun-2.7/bin/pip install -r ka-lite/requirements.txt
# 8. Run `bin/kalite manage compileymltojson`, needs `pyrun/pip install pyyaml==3.11

[jamalex]

@cpauya please don't build the docs, we can include them in the sdist

This is the point of the list we're building -- once we know what the necessary steps are, and make sure they're handled by a particular part of the cross-OS build process, we can trim back the work being done by each installer.