d37cd6b097
Merge pull request #873 from Gluejar/maintenance2020
...
update ubiquity sites
2020-06-25 14:42:06 -04:00
3ebccdbe88
update ubiquity sites
2020-06-25 14:21:37 -04:00
6d9fa0548f
Merge pull request #869 from Gluejar/maintenance2020
...
Maintenance June 2020
2020-06-03 16:08:55 -04:00
9c231bc401
no work if no title
...
Amazon does this to robots
2020-06-01 18:54:01 -04:00
22b06107bb
catche certificate errors for thumbnails
2020-04-29 21:19:48 -04:00
9a411f1906
fic empty new edition
...
fix error when user click new edition without entering anything
2020-04-27 13:10:32 -04:00
f4a9697971
fix some problems
2020-04-01 17:18:37 -04:00
b11493332f
fix gb integration test
2020-03-31 13:48:10 -04:00
a7e1c59ba5
Update ku.py
2020-03-23 13:07:46 -04:00
a5830669a6
pylint
2020-03-20 14:18:49 -04:00
af0e92f90d
fix goodreads- py3 issues
2020-03-20 13:01:42 -04:00
e14c972f41
make task args jsonable
2020-03-05 14:30:20 -05:00
6705583c89
added automation tasks
2020-03-04 14:37:40 -05:00
7d175185f3
Revert "tweak"
...
This reverts commit 7b440d1de7
2020-02-28 15:54:09 -05:00
7b440d1de7
tweak
...
wondering why celery/redis is balky on m
2020-02-28 15:35:31 -05:00
0b689f7fdb
didn't pay attention to method signature
2020-02-28 11:20:21 -05:00
c8262a765f
skip celery tests on travis
2020-02-28 10:54:33 -05:00
81e0687f19
address test failures
...
add the redis backend
celery worker has different db than test fixture
celery removed teasksets
2020-02-27 14:54:03 -05:00
c9d93efa1d
upgrade celery to 4.4
2020-02-26 23:15:55 -05:00
88a39daae5
new celery config
2020-02-26 17:15:22 -05:00
373df95927
p3 lint
2020-02-18 11:15:19 -05:00
e3f5459f68
lint
2020-02-18 09:58:02 -05:00
01770ecaee
passes all tests
2020-02-17 16:34:02 -05:00
5eaff720f9
remove unuse pycrypto
2020-02-17 15:28:19 -05:00
a9b00fabb6
remove Key model
...
stripe keys now encrypted by ansible
2020-02-17 15:18:56 -05:00
a8125853cf
fix import
2020-02-17 14:06:17 -05:00
176dedd439
open binary files as binary
2020-02-17 14:05:59 -05:00
c8a76cc925
xrange
2020-02-17 14:02:10 -05:00
e5aaff843b
get_hex
2020-02-17 14:01:23 -05:00
6bd3a8ea0f
itervalues
2020-02-17 13:59:52 -05:00
a06f32d3e6
fixed io errors in mobi
2020-02-14 18:46:52 -05:00
824ba51e8b
migrations
2020-02-14 13:55:12 -05:00
70d026e841
HTMLParser
2020-02-13 22:26:34 -05:00
0edb9004ab
<>
2020-02-13 22:25:27 -05:00
abde580a8c
httplib import
2020-02-13 17:42:36 -05:00
fee7cfcc0a
one more urllib
2020-02-12 23:23:52 -05:00
abff651f52
StringIO
2020-02-12 23:22:11 -05:00
a783be2e48
urllib
2020-02-12 17:57:35 -05:00
2382d036d5
StringIO
2020-02-12 17:56:56 -05:00
7aa6148b42
exception syntax
2020-02-12 17:56:04 -05:00
7f08799ca7
use py pyepub
2020-02-12 11:39:47 -05:00
61dc601d7c
unicode -> str
2020-02-12 11:36:49 -05:00
60017ab959
remove has_key
2020-02-12 11:12:56 -05:00
82db131740
change print to print()
2020-02-12 10:43:41 -05:00
e2349213e5
clean up core tests imports
2020-02-07 12:45:09 -05:00
8911868a35
resolve import issue
2020-02-07 12:06:07 -05:00
c8b6dc0d75
new edition at google books
2020-02-07 11:31:32 -05:00
962334e325
punctum books moved
2020-02-07 11:25:26 -05:00
70581b26e7
2020 -> 2030
...
Fix test broken by passage of time
2020-01-01 10:13:46 -05:00
07e4d9b937
weird pubdate instability
2019-11-30 22:50:11 -05:00
e650596a64
pass tests
2019-11-30 18:00:54 -05:00
c69de41628
refactor add_subject
2019-11-30 18:00:32 -05:00
4ec9b73a1a
fix subject auth loading
2019-11-30 15:29:59 -05:00
21810d5641
handle redirected ebook links
2019-11-06 12:40:35 -05:00
49929763ce
seems import order mattered
2019-11-05 15:45:32 -05:00
0bc92ea98d
add routledge md scraper
2019-11-05 15:37:55 -05:00
e2571abc42
fix bisac loading, add headings
2019-11-05 15:34:12 -05:00
ac599f5d69
fix strip
2019-07-01 17:21:22 -04:00
917d90aee1
delint
2019-07-01 16:32:41 -04:00
1c5c48ac42
Update cc.py
2019-07-01 16:25:18 -04:00
0d748b2498
don't get fooled by version strings on CC
2019-07-01 16:21:22 -04:00
4969994a87
urllib2 didn't handle chunked method
2019-06-13 16:20:05 -04:00
e3a5a50f34
catch S3 exception
2019-06-13 16:18:54 -04:00
703db9ed98
add SciELO to good providers
2019-06-12 17:12:54 -04:00
6814380aa4
tweak scielo handling
...
and add a management command to fix the old ones
2019-06-12 17:02:57 -04:00
d5f5656d3c
fix missing logger
2019-06-12 17:02:11 -04:00
e60e8bfbf8
get dl url from dl link
2019-06-07 15:20:05 -04:00
e42d77589b
tighten exception handling
...
got a bunch of integrity errors failure; probably some other exception being throughn here.
2019-06-06 17:23:45 -04:00
e5ba5caab4
revert search method
...
fulltext search returned too many results
2019-06-05 14:21:02 -04:00
de3e6c499c
try to fix missing scheme
2019-05-05 12:50:52 -04:00
14346ed868
delint
2019-03-27 21:46:25 -04:00
c142533898
db cleaning
2019-03-27 21:22:56 -04:00
e563da9655
refactor lang validation
2019-03-27 21:22:37 -04:00
6fd33d989c
don't create bad works
2019-03-27 21:21:25 -04:00
5fc6a2ee82
harvest more ebooks
2019-03-25 12:47:20 -04:00
fe05ff9f88
don't stall on super big pdf files
2019-03-25 12:47:04 -04:00
2396e23ae4
fix missing lang string
2019-03-25 12:46:20 -04:00
174b46abd1
add mobied to ebf admin
2019-03-25 12:45:53 -04:00
c190fc0bb1
fix undefined "stapled"
2019-03-08 23:45:54 -05:00
9b12418ada
catch more pdf errors
2019-03-05 12:02:42 -05:00
cefbc7c56f
bugfix
2019-03-05 10:12:51 -05:00
d87578c5a0
harden stapler
2019-03-04 17:27:55 -05:00
52b1621633
bugfix
2019-03-02 20:55:42 -05:00
7c33cae82e
refinements
...
- handle dropbox urls with no params
- catch exceptions in stapler
- fix dedupe summary
2019-03-02 19:16:47 -05:00
9bf2d85108
fix degruyter signifier
...
also propagate user_agent
2019-03-02 16:00:11 -05:00
943031ca22
whoops
2019-03-01 22:38:46 -05:00
02170c9bc2
management commands
...
1. run an update of providers
2. dedupe the online ebooks
3. should have half the onlines to harvest
2019-03-01 21:26:39 -05:00
ac5c241e09
resolve doi in doab provider
...
- resolve the doi before setting the provider
- strip "www." from netloc
- strip url before setting provider
2019-03-01 21:23:54 -05:00
1fdac9c548
remove dead code
2019-02-28 16:34:14 -05:00
0282ed8136
delint
2019-02-28 16:22:23 -05:00
72a40976bc
add degruyter handling
...
- move harvest to separate module
- add ratelimiter class
- add pdf stapler
- add a googlebot UA
- add base url storage in get_soup
2019-02-28 15:32:41 -05:00
e162308191
change to a fulltext query and indices
...
(this is only a ~20% improvement)
2019-02-27 16:40:21 -05:00
390f403e6c
missing import
2019-02-18 15:29:16 -05:00
1a8f22411a
change to ku sso
2019-02-18 15:06:40 -05:00
8652ce0b77
add rounds to ku
2019-01-18 12:03:04 -05:00
c6771f2eed
fix limit on harvest_online
2018-12-10 14:30:54 -05:00
260650ba92
handle application/binary
2018-12-10 14:28:39 -05:00
24ab902e00
added ebook activation
2018-11-05 18:48:35 -05:00
ed64dc2b3f
bugfix
2018-11-05 18:17:46 -05:00
6535505e4d
Revert "Merge branch 'master' into master"
...
This reverts commit bd52df020de455d9a766
2018-11-03 17:23:07 -04:00
bd52df020d
Merge branch 'master' into master
2018-11-03 17:06:09 -04:00
f4d7e6f888
working ku code
2018-11-03 14:47:41 -04:00
f98de7114e
add oapn id
2018-11-03 14:33:23 -04:00
add0375ac3
working scraper
2018-11-02 14:03:30 -04:00
b727aaf9a9
Merge pull request #813 from Gluejar/kuscrape
...
Kuscrape
2018-11-02 13:58:24 -04:00
57769f65a1
Update core/loaders/multiscrape.py
...
update to facilitate merg
2018-11-02 13:24:23 -04:00
53995ffb4a
allow scrapers to set parser
...
needed to support xml harvests
2018-10-29 22:42:49 -04:00
3697789274
wip
2018-10-09 09:05:31 -04:00
272616895d
fix github3 issue
2018-09-10 12:04:12 -04:00
a87cdfc8ef
make sure cc url is not garbage
2018-09-09 22:12:42 -04:00
04aed3bf16
add opentextbc to pressbooks list
2018-09-09 21:55:38 -04:00
265d63cf75
corrected test for social_auth
2018-08-14 15:17:32 -04:00
d6d6841515
new markup for title
2018-08-08 23:09:39 -04:00
7efb048a5c
adapt to newer pressbooks version
2018-08-02 17:27:04 -04:00
f01ccd168a
update management commands
...
updated argument declaration
use stdout.write() in stead of print
2018-07-31 13:12:22 -04:00
c2ed76b57f
don't send mailchimp invite until activation
2018-07-30 18:37:38 -04:00
7d2b052ef0
really fix pyepub
2018-07-29 20:56:05 -04:00
26bab0af68
last 2.0 deprecations
2018-07-24 11:33:39 -04:00
cb1ec66a08
addressed deprecations
...
mostly adding on_delete to models
2018-07-23 23:48:35 -04:00
18c0afd38e
update tastypie
2018-07-23 22:51:01 -04:00
50f0294b59
is_anonymous and is_authenticated are properties
2018-07-23 22:17:05 -04:00
b35aa2ce93
fix test failures for django 1.10.8
2018-07-22 13:14:27 -04:00
456a341885
fix test fixture
2018-07-20 15:23:14 -04:00
5455c21d26
cleanup after the facebook changes
2018-07-20 15:10:38 -04:00
26d65e8793
facebook id not useful; get pic instead
2018-07-20 13:03:51 -04:00
311d6fa0be
fix rare merge issue
2018-07-12 12:56:57 -04:00
ee03d2d434
add hosts
2018-07-12 12:56:09 -04:00
da601a77f6
final fixes
2018-07-11 13:41:52 -04:00
1b4beb0b0b
fix thumbnail fails
2018-07-10 13:59:05 -04:00
40794ee3f9
use rights info to set rights
2018-07-10 13:58:38 -04:00
ec3d26118e
fr/en
2018-07-10 13:58:06 -04:00
2f532b97f9
scrape multiple books from one url
2018-07-09 15:46:36 -04:00
6f2391b0b9
Merge branch 'update' into master
2018-06-26 14:29:02 -05:00
e82c389236
make sure to queue the notification emmisions
...
so that we don't need the emit_notices cron job
2018-06-20 15:57:34 -04:00
4dbf3ab725
Merge pull request #788 from Gluejar/fix-mailchimp
...
Fix mailchimp
2018-06-20 10:56:13 -04:00
1d83e7af78
update to mailchimp 3.0 api
2018-06-20 10:03:48 -04:00
fadba3ce1b
syntax
2018-06-19 14:15:53 -04:00
042613b0ae
wip
2018-06-19 14:14:10 -04:00
59b49e2c2e
fix exception when there's no google result
2018-06-19 13:34:29 -04:00
e49fbd1826
correct handling of google books returning multiple items
2018-06-19 10:44:15 -04:00
b709ad93f8
fix handling of cascade argument
2018-06-19 10:43:42 -04:00
d8171088a6
support lists of doabids
2018-06-19 10:42:02 -04:00
604260363f
deleting works should delete attached objects
...
I think the only place this should get invoked is via admin
2018-06-18 22:07:29 -04:00
2f9dda8432
less agressive merging in doab
2018-06-18 17:04:40 -04:00
53a0791da0
handle series properly
2018-06-18 17:04:19 -04:00
3bc7d5c003
fix loader tests
2018-06-18 17:03:41 -04:00
7593944dc0
reset default to 15 days
2018-06-15 15:30:04 -04:00
bade8e7f4d
handle records without downloads
2018-06-15 10:34:23 -04:00
ea278c3f41
add reset option
2018-06-11 14:43:56 -04:00
88a50073de
Merge remote-tracking branch 'Gluejar/master'
2018-06-09 15:54:14 -04:00
6408c878c4
pull test fixes from Gluejar/master
2018-06-09 15:00:45 -04:00
35569dd88b
Merge branch 'master' into add-travis
2018-06-08 20:40:30 -04:00
ab3c1934d5
fix pdf tester
...
boto3 storage returns unicode not str someplace
2018-06-08 13:56:07 -04:00
c422965bdb
add column to EbookFile to indicate a failed make_mobi
...
so we don't keep trying with a bad epub file
2018-06-06 22:26:27 -04:00
e1553f8121
switch to dockerized mobi
2018-06-06 17:29:12 -04:00
eba9af93c3
debug travis mobigen
2018-06-06 11:26:02 -04:00
f6310fde58
fix test
2018-06-05 14:48:22 -04:00
3661faec0a
Merge remote-tracking branch 'Gluejar/master' into catchup
...
# Conflicts:
# core/models/__init__.py
# core/models/bibmodels.py
# vagrant/host_vars/prod/secrets.yml
2018-05-23 13:56:27 -04:00
38e151224d
Add ansible role to provision production server
...
Add new ansible role which provisions production instance based on newfoundation branch.
2018-05-13 18:55:29 -04:00
05fae60ddb
delint
2018-05-11 11:46:04 -04:00
a303c08333
delete doab should work
2018-05-10 22:00:47 -04:00
db9b6e5221
harvest_online_ebooks should count books actually harvested
2018-05-10 16:17:16 -04:00
6585bdd52a
provide fallback for hathi scraper
...
It turns out http://hdl.handle.net/2027/ is used for all of umich, not
just hathitrust
2018-04-27 10:54:41 -04:00
a0dc106f6d
fix issue with merged works coming back from related editions
2018-04-26 14:57:55 -04:00
8f600f1556
make it work in django 1.8
2018-04-25 17:20:44 -04:00
3337de9a2e
Merge remote-tracking branch 'Gluejar/master' into dj111
...
# Conflicts:
# core/tests.py
# payment/tests.py
2018-04-25 16:23:00 -04:00
8d5da39e5f
make populate edition synchronous for doab
2018-04-25 11:21:02 -04:00
52957407ad
Merge in master
2018-04-24 09:24:58 -04:00
fa82411921
don't load chapters
2018-04-23 15:41:42 -04:00
680caad71d
deprecations
2018-04-19 21:27:20 -04:00
146f26c5f8
deprecate django unittest
2018-04-19 21:26:29 -04:00
42ccc01255
no models loaded with app
2018-04-19 21:23:24 -04:00
39fdd9683a
rip out dead code
2018-04-19 12:24:34 -04:00
d0ffa6a002
forgot to remove debugging break
2018-04-18 22:21:41 -04:00
29ca00277d
fix test
2018-04-18 22:00:36 -04:00
6bca7f0983
bugs
2018-04-18 21:39:40 -04:00
bbd421d1f2
fix various bugs
2018-04-18 17:53:21 -04:00
c9e7d5d5ac
avoid errors with using string methods on content_type
2018-04-18 14:56:26 -04:00
8c1799ebc4
freshen clean_subjects
2018-04-18 14:55:26 -04:00
b0abceb678
deal with more punctuation in isbn string
2018-04-18 14:54:09 -04:00
3590c1a59f
default load_doab to last 45 days
2018-04-18 14:53:42 -04:00
1d6af73cf2
handle isbns separated by '/'
2018-04-18 11:29:57 -04:00
fd02b5e66c
allow unicode dashes in isbns
2018-04-18 11:15:46 -04:00
78d66a247e
don't fail if null edition
2018-04-17 14:21:21 -04:00
447ed4b2d5
fix cover loading
2018-04-17 14:20:44 -04:00
5671856daa
... and the management command
2018-04-16 13:46:33 -04:00
8dd1fb1822
remove doab author loader
...
now uses oai functionality
2018-04-16 13:44:10 -04:00
b849f3a6ef
finish mapping languages
2018-04-16 12:32:21 -04:00
a6039e4015
better handling of language codes
2018-04-13 14:39:03 -04:00
e433c13108
fix online_to_download bugs
2018-04-13 14:38:39 -04:00
9a6b1efd0d
fix bugs for records with missing fields
2018-04-13 14:37:50 -04:00
add1ed54e7
improve logging for bookloader
2018-04-13 14:35:38 -04:00
ba7b02b939
add alternate url pattern for doab_id
2018-04-12 15:09:07 -04:00
bf7a9d8106
patch for missing language
2018-04-12 15:08:29 -04:00
748b0eaa63
add test
2018-04-09 17:26:04 -04:00
cdcd322210
added harvest_online_ebooks management command
2018-04-09 16:59:35 -04:00
c26e365a64
fixed imports
2018-04-09 16:58:58 -04:00
ca94c128de
online to download handling
...
+ fix bug that made everythong 'online'
+ handle online ebooks with multiple format downloads
+ download ebooks with volatile links
+ move contenttyper to core.loaders.utils
+ add handling for really html ebooks
2018-04-09 16:32:52 -04:00
9e101fcd99
inconsequential bug
2018-04-09 16:29:04 -04:00
07fd095b9a
fix bugs
2018-04-09 11:54:16 -04:00
a982588808
bugfix
...
handle librarything outage
2018-04-09 11:52:39 -04:00
0ba2906c62
delint
2018-04-07 18:38:33 -04:00
e03fa239b4
revamp doab loading
...
- doab loading now done primarily by oai, no processing of csv.
- added pyoai and updated lxml
- doab ids or urls in ebook submission now handled by oai scrape
- doab_load_books removed
- doab_utils moved from Gluejar/DOAB
- licenses now recognizes OpenEdition
- new ebook type "online" will implement in UI after mobile launch;
ebooks now creaded for html contenttype
2018-04-07 17:11:36 -04:00
533eb94152
load springer improvements
...
We've loaded about half the Springer Open books catalog, adding 20
books at a time. I wanted to load page 23 of results without having to
load pages 1-22. Also added some exception handling.
2018-03-22 16:13:55 -04:00
957cf615b4
delint
2018-02-22 11:29:52 -05:00
1425b29d49
[unrelated] fix librarything error handling
2018-02-22 11:11:04 -05:00
fae854cd57
fix max == none
2018-02-22 11:06:40 -05:00
53d904dfc6
Merge pull request #761 from Gluejar/bugfix
...
Misc Bugfix
2018-02-21 12:08:13 -05:00
b88d678058
add loading from sitemap list
2018-02-20 13:35:00 -05:00
ad9523314d
fix bug in ubiquity scraper
2018-02-20 13:07:44 -05:00
97708c41d6
Merge in master
2018-02-13 17:42:09 -05:00
e015539fee
change language about "gift credit"
2018-02-07 16:44:29 -05:00
7e69ac95eb
add negative id facets
2018-02-01 21:47:29 -05:00
b148a1925e
harden search
2018-02-01 15:18:14 -05:00
33f4b75417
stricter RE
2018-01-04 16:53:29 -05:00
ba381add02
add smashwords
2018-01-03 15:53:02 -05:00
59388933a9
one scraper per file
2018-01-03 13:58:45 -05:00
6dfa1bccb4
lint
2018-01-03 13:43:02 -05:00
e837dd6ff2
added date validation
2018-01-03 13:30:36 -05:00
c8837c3c74
make check_metas case insensitive for name
2018-01-03 11:54:48 -05:00
3f3428a68b
add some opengraph support
2018-01-02 18:20:34 -05:00
f1213d590c
fix can_scrape
2018-01-01 19:25:00 -05:00
72ae3c73a5
trouble with mobis when title is unicode?
2017-12-27 12:20:56 -05:00
cf093c945d
add some custom code for ubiquity press sites
2017-12-23 18:29:16 -05:00
e6dbae05db
update springer
2017-12-23 18:15:59 -05:00
f701f1ba36
refactor can_scrape
2017-12-23 18:12:07 -05:00
2e70da0a4a
delint
2017-12-22 13:19:26 -05:00
2f649ef9a2
admin can now fix ebook/ebookfile problems
2017-12-22 12:42:29 -05:00
d1cf6e6fb3
fix some scraping bugs
2017-12-15 19:26:50 -05:00
7d847019a1
Merge branch 'master' into campaign-donations
2017-12-14 21:38:34 -05:00
4aeae6e67c
implement donation options
2017-12-14 16:24:26 -05:00
15d281c988
make mobi wasn't setting provider
2017-12-14 14:19:20 -05:00
4499b556c6
protect long descriptions
...
scraper was over-writing edited descriptions
2017-12-11 13:45:47 -05:00
886068a6ee
clean up after change to work id
...
doi and http_id were changed to work only
2017-12-10 17:05:58 -05:00
ebf68befeb
add Springer publisher
2017-12-10 16:38:30 -05:00
678474b985
do conversion before making ebf
2017-12-10 16:33:59 -05:00
cc1c5b6ee8
add management command
2017-12-10 16:33:07 -05:00
3c7c9ade00
add Springer to get_scraper
2017-12-07 17:36:35 -05:00
d53b3bcc8d
delint
2017-12-07 17:36:08 -05:00
5ccd7a0a47
add get_role to scraper
2017-12-07 17:35:52 -05:00
a3f1509cc2
fix multiple editor setting
2017-12-07 17:33:29 -05:00
c6885ff84b
fix springer descriptions
2017-12-07 16:35:11 -05:00
81c3268f70
fix license url
2017-12-07 16:34:25 -05:00
6bba688f03
fix kw loading
2017-12-07 16:33:53 -05:00
5c3137a85d
delint
2017-12-07 12:50:08 -05:00
82784778c4
add springer scraper
2017-12-06 18:13:46 -05:00
5f39729d74
fix doi validation
2017-12-06 18:12:46 -05:00
d134ef0606
core.WorkTests.test_valid_subject were technically not working correctly because the test strings should be unicode not str
2017-11-27 12:40:03 -08:00
af4cac5cf8
http should be a work id
2017-11-21 15:47:02 -05:00
28fa60ffba
fix cover finding
2017-11-21 11:10:46 -05:00
Eric Hellman
John Kelly
jonahmania
Raymond Yee