webapp for unglue.it
 
 
 
 
 
 
Go to file
eric 350a2a0ba2 Merge branch 'master' into production 2017-06-29 13:56:32 -04:00
api add publishers, ebooks to api 2017-06-29 13:22:58 -04:00
bisac patterns in urlpatterns are deprecated 2016-07-27 13:02:47 -04:00
bookdata update doab.json 2017-01-12 14:31:47 -08:00
booxtream passes all tests without secret keys 2016-12-30 22:26:16 -05:00
core delete regluit.questionnaire 2017-06-20 11:08:14 -04:00
deploy delete the various specific apache conf files (which are no longer needed because we have teh apache.conf.j2 template) 2017-03-22 14:23:01 -07:00
distro Re-enable distro admin 2017-06-12 15:40:57 -04:00
docs typo 2017-01-16 17:48:50 -08:00
frontend delete regluit.questionnaire 2017-06-20 11:08:14 -04:00
libraryauth fix block edit 2016-11-09 15:24:11 -05:00
logs need this log directory 2011-09-04 05:40:12 +00:00
marc implement EditionNote 2016-08-16 17:16:44 -04:00
mobi add checker for mobi 2014-02-05 18:17:26 -05:00
payment add donation box to homepage 2017-02-13 13:33:26 -05:00
pyepub switch from deprecated get_model a app registry 2016-07-24 18:39:36 -04:00
selenium Adding explicit waits to selenium payment tests in order to wait for very slow js when running headless on ec2 2012-01-26 19:50:14 +00:00
settings remove old static folder 2017-06-20 14:47:29 -04:00
static add donation box to homepage 2017-02-13 13:33:26 -05:00
sysadmin Added rrsets_for_domain to aws.py and some mods to my notebooks 2015-05-04 10:51:11 -07:00
test updated for recent Se 2017-03-21 11:10:29 -07:00
utils add sanitization code to edition form 2016-10-20 15:24:47 -04:00
vagrant getting ready to commit by setting branch to master for please, just, just2 2017-03-21 16:27:22 -07:00
.gitattributes fix problems for please 2017-01-06 18:35:49 -08:00
.gitignore revive settings files 2016-12-20 14:55:53 -05:00
LICENSE.txt add license 2017-01-19 10:37:59 -05:00
README.md Add comment about current status 2017-01-19 10:50:53 -05:00
__init__.py
admin.py refactor admin 2016-07-26 10:34:45 -04:00
context_processors.py update noftifications 2016-07-22 18:44:54 -04:00
manage.py update manage.py for Django 1.6 2016-04-11 13:17:16 -07:00
requirements_versioned.pip delete regluit.questionnaire 2017-06-20 11:08:14 -04:00
urls.py delete regluit.questionnaire 2017-06-20 11:08:14 -04:00

README.md

regluit - "The Unglue.it web application and website"

Another repo - https://github.com/EbookFoundation/regluit will be the place for collaborative development for Unglue.it. Add issues and submit pull requests there. As of January 19, 2017, https://github.com/Gluejar/regluit is still being used for production builds.

The first version of the unglue.it codebase was a services-oriented project named "unglu". We decided that "unglu" was too complicated, so we started over and named the new project "regluit". regluit is a Django project that contains four main applications: core, frontend, api and payment that can be deployed and configured on as many ec2 instances that are needed to support traffic. The partitioning between these modules is not as clean as would be ideal. payment is particularly messy because we had to retool it twice because we had to switch from Paypal to Amazon Payments to Stripe.

regluit was originally developed on Django 1.3 (python 2.7) and currently runs on Django 1.8.

Develop

Here are some instructions for setting up regluit for development on an Ubuntu system. If you are on OS X see notes below to install python-setuptools in step 1:

  1. aptitude install python-setuptools git python-lxml
  2. sudo easy_install virtualenv virtualenvwrapper
  3. git clone git@github.com:Gluejar/regluit.git
  4. cd regluit
  5. mkvirtualenv regluit
  6. pip install -r requirements_versioned.pip
  7. add2virtualenv ..
  8. cp settings/dev.py settings/me.py
  9. mkdir settings/keys/
  10. cp settings/dummy/* settings/keys/
  11. edit settings/me.py with path info, etc. to make things work
  12. edit the files in settings/keys filling in account and key information OR if you have the ansible vault password, install ansible, then from the vagrant directory run ansible-playbook create_keys.yml
  13. echo 'export DJANGO_SETTINGS_MODULE=regluit.settings.me' >> ~/.virtualenvs/regluit/bin/postactivate
  14. deactivate ; workon regluit
  15. django-admin.py syncdb --migrate --noinput
  16. django-admin.py celeryd --loglevel=INFO start the celery daemon to perform asynchronous tasks like adding related editions, and display logging information in the foreground.`
  17. django-admin.py celerybeat -l INFO to start the celerybeat daemon to handle scheduled tasks.
  18. django-admin.py runserver 0.0.0.0:8000 (you can change the port number from the default value of 8000)
  19. point your browser at http://localhost:8000/

CSS development

  1. We are using Less version 2.8 for CSS. http://incident57.com/less/. We use minified CSS.

Production Deployment

OBSOLETE Below are the steps for getting regluit running on EC2 with Apache and mod_wsgi, and talking to an Amazon Relational Data Store instance. Instructions for setting please are slightly different.

  1. create an ubuntu ec2 instance (e.g, go http://alestic.com/ to find various ubuntu images)
  2. sudo aptitude update
  3. sudo aptitude upgrade
  4. sudo aptitude install git-core apache libapache2-mod-wsgi mysql-client python-virtualenv python-mysqldb redis-server python-lxml postfix python-dev libmysqlclient-dev
  5. sudo mkdir /opt/regluit
  6. sudo chown ubuntu:ubuntu /opt/regluit
  7. cd /opt
  8. git config --global user.name "Raymond Yee"
  9. git config --global user.email "rdhyee@gluejar.com"
  10. ssh-keygen
  11. add ~/.ssh/id\_rsa.pub as a deploy key on github https://github.com/Gluejar/regluit/admin/keys
  12. git clone git@github.com:Gluejar/regluit.git
  13. cd /opt/regluit
  14. create an Amazon RDS instance
  15. connect to it, e.g. mysql -u root -h gluejardb.cboagmr25pjs.us-east-1.rds.amazonaws.com -p
  16. CREATE DATABASE unglueit CHARSET utf8;
  17. GRANT ALL ON unglueit.\* TO unglueit@ip-10-244-250-168.ec2.internal IDENTIFIED BY 'unglueit' REQUIRE SSL;
  18. update settings/prod.py with database credentials
  19. virtualenv ENV
  20. source ENV/bin/activate
  21. pip install -r requirements_versioned.pip
  22. echo "/opt/" > ENV/lib/python2.7/site-packages/regluit.pth
  23. django-admin.py syncdb --migrate --settings regluit.settings.prod
  24. sudo mkdir /var/www/static
  25. sudo chown ubuntu:ubuntu /var/www/static
  26. django-admin.py collectstatic --settings regluit.settings.prod
  27. sudo ln -s /opt/regluit/deploy/regluit.conf /etc/apache2/sites-available/regluit
  28. sudo a2ensite regluit
  29. sudo a2enmod ssl rewrite
  30. cd /home/ubuntu
  31. copy SSL server key to /etc/ssl/private/server.key
  32. copy SSL certificate to /etc/ssl/certs/server.crt
  33. sudo /etc/init.d/apache2 restart
  34. sudo adduser --no-create-home celery --disabled-password --disabled-login (just enter return for all?)
  35. sudo cp deploy/celeryd /etc/init.d/celeryd
  36. sudo chmod 755 /etc/init.d/celeryd
  37. sudo cp deploy/celeryd.conf /etc/default/celeryd
  38. sudo mkdir /var/log/celery
  39. sudo mkdir /var/run/celery
  40. sudo chown celery:celery /var/log/celery /var/run/celery
  41. sudo /etc/init.d/celeryd start
  42. sudo cp deploy/celerybeat /etc/init.d/celerybeat
  43. sudo chmod 755 /etc/init.d/celerybeat
  44. sudo cp deploy/celerybeat.conf /etc/default/celerybeat
  45. sudo mkdir /var/log/celerybeat
  46. sudo chown celery:celery /var/log/celerybeat
  47. sudo /etc/init.d/celerybeat start

setup to enable ckeditor to work properly

  1. mkdir /var/www/static/media/
  2. sudo chown ubuntu:www-data /var/www/static/media/

Updating Production

  1. Study the latest changes in the master branch, especially keep in mind how it has changed from what's in production.
  2. Update the production branch accordingly. If everything in master is ready to be moved into production, you can just merge master into production. Otherwise, you can grab specific parts. (How to do so is something that should probably be described in greater detail.)
  3. Login to unglue.it and run /opt/regluit/deploy/update-prod

OS X Developer Notes

To run regluit on OS X you should have XCode installed

Install virtualenvwrapper according to the process at http://blog.praveengollakota.com/47430655:

  1. sudo easy\_install pip
  2. sudo pip install virtualenv
  3. pip install virtualenvwrapper

Edit or create .bashrc in ~ to enable virtualenvwrapper commands:

  1. mkdir ~/.virtualenvs

  2. Edit .bashrc to include the following lines:

    export WORKON_HOME=$HOME/.virtualenvs source your_path_to_virtualenvwrapper.sh_here

In the above web site, the path to virtualenvwrapper.sh was /Library/Frameworks/Python.framework/Versions/2.7/bin/virtualenvwrapper.sh In Snow Leopard, this may be /usr/local/bin/virtualenvwrapper.sh

Configure Terminal to automatically notice this at startup: Terminal > Preferences > Settings > Shell Click "run command"; add source ~/.bashrc

If you get 'EnvironmentError: mysql_config not found' edit the line ~/.virtualenvs/regluit/build/MySQL-python/setup_posix.py

  1. mysql_config.path = "mysql_config" to be (using a path that exists on your system)
  2. mysql_config.path = "/usr/local/mysql-5.5.20-osx10.6-x86_64/bin/mysql_config"

You may need to set utf8 in /etc/my.cnf collation-server = utf8_unicode_ci

init-connect='SET NAMES utf8'
character-set-server = utf8

Selenium Install

Download the selenium server: http://selenium.googlecode.com/files/selenium-server-standalone-2.5.0.jar

Start the selenium server: 'java -jar selenium-server-standalone-2.5.0.jar'

MARC Records

For unglued books with existing print edition MARC records

  1. Get the MARCXML record for the print edition from the Library of Congress.
    1. Find the book in catalog.loc.gov
    2. Click on the permalink in its record (will look something like lccn.loc.gov/2009009516)
    3. Download MARCXML
  2. At /marc/ungluify/ , enter the unglued edition in the Edition field, upload file, choose license
  3. The XML record will be automatically...
    • converted to suitable MARCXML and .mrc records, with both direct and via-unglue.it download links
    • written to S3
    • added to a new instance of MARCRecord
    • provided to ungluers at /marc/
  1. Use /admin to create a new MARC record instance
  2. Upload the MARC records to s3 (or wherever)
  3. Add the URLs of the .xml and/or .mrc record(s) to the appropriate field(s)
  4. Select the relevant edition
  5. Select an appropriate marc_format:
    • use DIRECT if it links directly to the ebook file
    • use UNGLUE if it links to the unglue.it download page
    • if you have records with both DIRECT and UNGLUE links, you'll need two MARCRecord instances
    • if you have both kinds of link, put them in separate records, as marc_format can only take one value
      ungluify_record.py should only be used to modify records of print editions of unglued ebooks. It will not produce appropriate results for CC/PD ebooks.

For unglued ebooks without print edition MARC records, or CC/PD books without ebook MARC records

  1. Get a contract cataloger to produce quality records (.xml and .mrc formats)
    • we are using ungas the format for our accession numbers, where is the id of the MARCRecord instance, plus leading zeroes
  2. Upload those records to s3 (or wherever)
  3. Create a MARCRecord instance in /admin
  4. Add the URLs of the .xml and .mrc records to the appropriate fields
  5. Select the relevant edition
  6. Select an appropriate marc_format:
    • use DIRECT if it links directly to the ebook file
    • use UNGLUE if it links to the unglue.it download page
    • if you have records with both DIRECT and UNGLUE links, you'll need two MARCRecord instances
    • if you have both kinds of link, put them in separate records, as marc_format can only take one value

vagrant / ansible

How to build machines using Vagrant/ansible