e0c0d98c5d
add a command to remove dead doabids
2021-05-03 22:17:46 -04:00
60ff850f4d
don't add multiple doab ids to a work
2021-04-28 15:53:31 -04:00
55a5875036
explode bic
...
doab has a lot of bic headings need to be chopped upp to fit in our schema
2021-03-10 15:32:03 -05:00
1a28aca817
check for null doab_id
2021-03-05 12:16:12 -05:00
b02c9f459c
sometimes stream data returns None
2021-03-04 21:48:26 -05:00
835ee6a5a6
delint
2021-03-03 20:27:25 -05:00
6297edf95c
get everything working
2021-03-03 18:00:40 -05:00
56c5c7ae56
clean up line endings
2020-09-26 16:34:16 -04:00
bbf2247a8f
deal with bad doab covers
2020-09-26 12:30:50 -04:00
06360e6e76
refactor provider setting
2020-09-07 10:14:51 -04:00
a6b02d387e
refactor ebf(url)
2020-08-15 20:21:56 -04:00
15424eaf4d
add error handling for doab 404s
2020-07-26 16:06:33 -04:00
79aa49a1f1
one more thing for doi
2020-07-22 19:28:02 -04:00
42899559e2
enrich management command
...
can now harvest doab from a date or starting at an doab_id
2020-07-22 19:27:45 -04:00
5882c07854
add dois from doab
2020-07-22 19:10:05 -04:00
7aa6148b42
exception syntax
2020-02-12 17:56:04 -05:00
82db131740
change print to print()
2020-02-12 10:43:41 -05:00
14346ed868
delint
2019-03-27 21:46:25 -04:00
6fd33d989c
don't create bad works
2019-03-27 21:21:25 -04:00
ac5c241e09
resolve doi in doab provider
...
- resolve the doi before setting the provider
- strip "www." from netloc
- strip url before setting provider
2019-03-01 21:23:54 -05:00
40794ee3f9
use rights info to set rights
2018-07-10 13:58:38 -04:00
2f9dda8432
less agressive merging in doab
2018-06-18 17:04:40 -04:00
7593944dc0
reset default to 15 days
2018-06-15 15:30:04 -04:00
bade8e7f4d
handle records without downloads
2018-06-15 10:34:23 -04:00
a0dc106f6d
fix issue with merged works coming back from related editions
2018-04-26 14:57:55 -04:00
8d5da39e5f
make populate edition synchronous for doab
2018-04-25 11:21:02 -04:00
fa82411921
don't load chapters
2018-04-23 15:41:42 -04:00
bbd421d1f2
fix various bugs
2018-04-18 17:53:21 -04:00
3590c1a59f
default load_doab to last 45 days
2018-04-18 14:53:42 -04:00
1d6af73cf2
handle isbns separated by '/'
2018-04-18 11:29:57 -04:00
78d66a247e
don't fail if null edition
2018-04-17 14:21:21 -04:00
447ed4b2d5
fix cover loading
2018-04-17 14:20:44 -04:00
8dd1fb1822
remove doab author loader
...
now uses oai functionality
2018-04-16 13:44:10 -04:00
9a6b1efd0d
fix bugs for records with missing fields
2018-04-13 14:37:50 -04:00
bf7a9d8106
patch for missing language
2018-04-12 15:08:29 -04:00
ca94c128de
online to download handling
...
+ fix bug that made everythong 'online'
+ handle online ebooks with multiple format downloads
+ download ebooks with volatile links
+ move contenttyper to core.loaders.utils
+ add handling for really html ebooks
2018-04-09 16:32:52 -04:00
07fd095b9a
fix bugs
2018-04-09 11:54:16 -04:00
0ba2906c62
delint
2018-04-07 18:38:33 -04:00
e03fa239b4
revamp doab loading
...
- doab loading now done primarily by oai, no processing of csv.
- added pyoai and updated lxml
- doab ids or urls in ebook submission now handled by oai scrape
- doab_load_books removed
- doab_utils moved from Gluejar/DOAB
- licenses now recognizes OpenEdition
- new ebook type "online" will implement in UI after mobile launch;
ebooks now creaded for html contenttype
2018-04-07 17:11:36 -04:00
b5e52effd9
optimize id access
...
See
https://docs.djangoproject.com/en/1.11/topics/db/optimization/#use-forei
gn-key-values-directly
2017-10-28 18:33:58 -04:00
1ce4323bc4
precheck every new subject
...
fix bug with '/' in subject
interpret ';' as list delimiter
add cleaner script
2017-09-15 15:55:37 -04:00
5bbeb45053
improve merge_works
...
work_relations were not being updated
2017-09-04 16:10:24 -04:00
652d9a3456
modify doab load to handle authlists
...
also fix a few encoding issues and null data problems resulting in
non-loading and ftp redirects
2016-12-02 15:50:07 -05:00
1c52c42e60
doab author parsing and loader command
2016-11-29 15:37:02 -05:00
671017fced
pass edition to update_cover_doab
2016-11-18 13:28:59 -05:00
60e4994756
remove debugging prints
2016-11-01 13:42:40 -04:00
b82b51f358
forgot to re-enable error catching
2016-10-31 22:19:00 -04:00
1c7df5e00e
get rid of some loader issues
2016-10-28 14:40:16 -04:00
39cf8c9c0b
fix load errors
2016-10-27 20:05:43 -04:00
182887fdc2
remove async option
2016-10-12 16:19:43 -04:00
eric