e6eaa24440 | ||
---|---|---|
api | ||
bookdata | ||
booxtream | ||
core | ||
deploy | ||
docs | ||
experimental | ||
frontend | ||
libraryauth | ||
logs | ||
marc | ||
mobi | ||
not_maintained | ||
notebooks | ||
payment | ||
pyepub | ||
selenium | ||
settings | ||
static | ||
sysadmin | ||
test | ||
test-data | ||
utils | ||
vagrant | ||
.gitignore | ||
README.md | ||
STAR_unglue_it.ca-bundle | ||
__init__.py | ||
admin.py | ||
aws_cleanup.ipynb | ||
bitnami_launch.ipynb | ||
build_ec2_instances_for_django.ipynb | ||
build_ec2_instances_for_django.py | ||
build_just.ipynb | ||
context_processors.py | ||
deploy_gluejar_dot_com.ipynb | ||
deploy_gluejar_dot_com.py | ||
fabfile.py | ||
manage.py | ||
requirements_versioned.pip | ||
ssh_fingerprint.ipynb | ||
urls.py |
README.md
regluit
A 'monolithic' alternative to unglu
for the unglue.it website. regluit is essentially a Django project that
contains three applications: frontend
, api
and core
that can be deployed
and configured on as many ec2 instances that are needed to support traffic.
The key difference with unglu is that the
frontend
app is able to access database models from core
in the same
way that the api
is able to...which hopefully should simplify some things.
Develop
Here are some instructions for setting up regluit for development on an Ubuntu system. If you are on OS X see notes below to install python-setuptools in step 1:
aptitude install python-setuptools git python-lxml
sudo easy_install virtualenv virtualenvwrapper
git clone git@github.com:Gluejar/regluit.git
cd regluit
mkvirtualenv regluit
pip install -r requirements_versioned.pip
add2virtualenv ..
cp settings/dev.py settings/me.py
- edit
settings/me.py
and setEMAIL_HOST_USER
andEMAIL_HOST_PASSWORD
to your gmail username and password, if you want to see that registration emails will work properly. - edit
settings/me.py
and look at the facebook, twitter and google auth settings to enable federated logins from those sites echo 'export DJANGO_SETTINGS_MODULE=regluit.settings.me' >> ~/.virtualenvs/regluit/bin/postactivate
deactivate ; workon regluit
django-admin.py syncdb --migrate --noinput
django-admin.py celeryd --loglevel=INFO
start the celery daemon to perform asynchronous tasks like adding related editions, and display logging information in the foreground.`django-admin.py celerybeat -l INFO
to start the celerybeat daemon to handle scheduled tasks.django-admin.py runserver 0.0.0.0:8000
(you can change the port number from the default value of 8000)- point your browser at http://localhost:8000/
CSS development
- We are using Less version 2.8 for CSS. http://incident57.com/less/. We use minified CSS.
Production Deployment
Below are the steps for getting regluit running on EC2 with Apache and mod_wsgi, and talking to an Amazon Relational Data Store instance. Instructions for setting please are slightly different.
- create an ubuntu ec2 instance (e.g, go http://alestic.com/ to find various ubuntu images)
sudo aptitude update
sudo aptitude upgrade
sudo aptitude install git-core apache libapache2-mod-wsgi mysql-client python-virtualenv python-mysqldb redis-server python-lxml postfix python-dev libmysqlclient-dev
sudo mkdir /opt/regluit
sudo chown ubuntu:ubuntu /opt/regluit
cd /opt
git config --global user.name "Raymond Yee"
git config --global user.email "rdhyee@gluejar.com"
ssh-keygen
- add
~/.ssh/id\_rsa.pub
as a deploy key on github https://github.com/Gluejar/regluit/admin/keys git clone git@github.com:Gluejar/regluit.git
cd /opt/regluit
- create an Amazon RDS instance
- connect to it, e.g.
mysql -u root -h gluejardb.cboagmr25pjs.us-east-1.rds.amazonaws.com -p
CREATE DATABASE unglueit CHARSET utf8;
GRANT ALL ON unglueit.\* TO ‘unglueit’@’ip-10-244-250-168.ec2.internal’ IDENTIFIED BY 'unglueit' REQUIRE SSL;
- update settings/prod.py with database credentials
virtualenv ENV
source ENV/bin/activate
pip install -r requirements_versioned.pip
echo "/opt/" > ENV/lib/python2.7/site-packages/regluit.pth
django-admin.py syncdb --migrate --settings regluit.settings.prod
sudo mkdir /var/www/static
sudo chown ubuntu:ubuntu /var/www/static
django-admin.py collectstatic --settings regluit.settings.prod
sudo ln -s /opt/regluit/deploy/regluit.conf /etc/apache2/sites-available/regluit
sudo a2ensite regluit
sudo a2enmod ssl rewrite
cd /home/ubuntu
- copy SSL server key to
/etc/ssl/private/server.key
- copy SSL certificate to
/etc/ssl/certs/server.crt
sudo /etc/init.d/apache2 restart
sudo adduser --no-create-home celery --disabled-password --disabled-login
(just enter return for all?)sudo cp deploy/celeryd /etc/init.d/celeryd
sudo chmod 755 /etc/init.d/celeryd
sudo cp deploy/celeryd.conf /etc/default/celeryd
sudo mkdir /var/log/celery
sudo mkdir /var/run/celery
sudo chown celery:celery /var/log/celery /var/run/celery
sudo /etc/init.d/celeryd start
sudo cp deploy/celerybeat /etc/init.d/celerybeat
sudo chmod 755 /etc/init.d/celerybeat
sudo cp deploy/celerybeat.conf /etc/default/celerybeat
sudo mkdir /var/log/celerybeat
sudo chown celery:celery /var/log/celerybeat
sudo /etc/init.d/celerybeat start
setup to enable ckeditor to work properly
mkdir /var/www/static/media/
sudo chown ubuntu:www-data /var/www/static/media/
Updating Production
- Study the latest changes in the master branch, especially keep in mind how it has changed from what's in production.
- Update the production branch accordingly. If everything in
master
is ready to be moved intoproduction
, you can just mergemaster
intoproduction
. Otherwise, you can grab specific parts. (How to do so is something that should probably be described in greater detail.) - Login to unglue.it and run
/opt/regluit/deploy/update-prod
OS X Developer Notes
To run regluit on OS X you should have XCode installed
Install virtualenvwrapper according to the process at http://blog.praveengollakota.com/47430655:
sudo easy\_install pip
sudo pip install virtualenv
pip install virtualenvwrapper
Edit or create .bashrc in ~ to enable virtualenvwrapper commands:
-
mkdir ~/.virtualenvs
-
Edit .bashrc to include the following lines:
export WORKON_HOME=$HOME/.virtualenvs source your_path_to_virtualenvwrapper.sh_here
In the above web site, the path to virtualenvwrapper.sh was /Library/Frameworks/Python.framework/Versions/2.7/bin/virtualenvwrapper.sh In Snow Leopard, this may be /usr/local/bin/virtualenvwrapper.sh
Configure Terminal to automatically notice this at startup:
Terminal –> Preferences –> Settings –> Shell
Click "run command"; add source ~/.bashrc
If you get 'EnvironmentError: mysql_config not found' edit the line ~/.virtualenvs/regluit/build/MySQL-python/setup_posix.py
- mysql_config.path = "mysql_config" to be (using a path that exists on your system)
- mysql_config.path = "/usr/local/mysql-5.5.20-osx10.6-x86_64/bin/mysql_config"
You may need to set utf8 in /etc/my.cnf collation-server = utf8_unicode_ci
init-connect='SET NAMES utf8'
character-set-server = utf8
Selenium Install
Download the selenium server: http://selenium.googlecode.com/files/selenium-server-standalone-2.5.0.jar
Start the selenium server: 'java -jar selenium-server-standalone-2.5.0.jar'
MARC Records
For unglued books with existing print edition MARC records
- Get the MARCXML record for the print edition from the Library of Congress.
- Find the book in catalog.loc.gov
- Click on the permalink in its record (will look something like lccn.loc.gov/2009009516)
- Download MARCXML
- At /marc/ungluify/ , enter the unglued edition in the Edition field, upload file, choose license
- The XML record will be automatically...
- converted to suitable MARCXML and .mrc records, with both direct and via-unglue.it download links
- written to S3
- added to a new instance of MARCRecord
- provided to ungluers at /marc/
For CC/PD books with existing records that link to the ebook edition
- Use /admin to create a new MARC record instance
- Upload the MARC records to s3 (or wherever)
- Add the URLs of the .xml and/or .mrc record(s) to the appropriate field(s)
- Select the relevant edition
- Select an appropriate marc_format:
- use DIRECT if it links directly to the ebook file
- use UNGLUE if it links to the unglue.it download page
- if you have records with both DIRECT and UNGLUE links, you'll need two MARCRecord instances
- if you have both kinds of link, put them in separate records, as marc_format can only take one value
ungluify_record.py
should only be used to modify records of print editions of unglued ebooks. It will not produce appropriate results for CC/PD ebooks.
For unglued ebooks without print edition MARC records, or CC/PD books without ebook MARC records
- Get a contract cataloger to produce quality records (.xml and .mrc formats)
- we are using ungas the format for our accession numbers, where is the id of the MARCRecord instance, plus leading zeroes
- Upload those records to s3 (or wherever)
- Create a MARCRecord instance in /admin
- Add the URLs of the .xml and .mrc records to the appropriate fields
- Select the relevant edition
- Select an appropriate marc_format:
- use DIRECT if it links directly to the ebook file
- use UNGLUE if it links to the unglue.it download page
- if you have records with both DIRECT and UNGLUE links, you'll need two MARCRecord instances
- if you have both kinds of link, put them in separate records, as marc_format can only take one value