Commit Graph

157 Commits (28b6a2d53588eaaa905b45d8cbf49eb6eedd338b)

Author SHA1 Message Date
Manuel Kaufmann 91a205e5ab Last pre-commit pass 2019-01-21 19:00:50 +01:00
Manuel Kaufmann 7d08eed428 Run add-trailing-comma over all the files 2019-01-21 18:49:11 +01:00
Manuel Kaufmann b5bc9c7ab2 pre-commit all the files 2019-01-21 18:42:43 +01:00
Manuel Kaufmann 5798557a0d isort over migration files
Removes `from __future__ import ...`
2019-01-21 18:16:25 +01:00
Manuel Kaufmann d8dacdee8f Merge branch 'master' of github.com:rtfd/readthedocs.org into humitos/deprecate-python2 2019-01-21 18:04:42 +01:00
Manuel Kaufmann 2ae1c26744 Lint fix 2019-01-10 19:25:53 +01:00
Manuel Kaufmann e17d0cdb45 Handle 401, 403 and 404 status codes when hitting GitHub for webhook 2019-01-10 18:43:23 +01:00
Manuel Kaufmann 632e18c121 pre-commit all the files again 2019-01-09 13:19:00 +01:00
Manuel Kaufmann f1820596f0 pre-commit run on some missing files (manually applyed) 2019-01-09 12:53:25 +01:00
Manuel Kaufmann 0c535feb20 More linting issues fixed 2019-01-09 12:51:48 +01:00
Manuel Kaufmann 3219f7acbd Run pre-commit on all the files
$ pre-commit run --files `find readthedocs -name '*'`
2019-01-08 21:51:36 +01:00
Manuel Kaufmann c78187b62f isort all the files
$ pre-commit run python-import-sorter --files `find readthedocs -name '*.py'`

It was ran with this configuration: https://github.com/rtfd/common/pull/28
2019-01-08 20:56:20 +01:00
Manuel Kaufmann f7dea72b86 pyupgrade 1.11.0 on all code
pyupgrade --py3-plus --py3-only (*.py)
2019-01-03 16:56:33 +01:00
dojutsu-user 81ee2ae2ae Remove deprecated imports from urlresolvers 2018-12-09 15:29:52 +05:30
Manuel Kaufmann bd33e1d402 Use is_authenticated and is_anonymous as attribute 2018-12-04 12:20:54 +01:00
Santos Gallegos 05baf4fe39 Listen to create/delete events on GitHub 2018-11-06 20:07:33 -05:00
Eric Holscher 953a4ee2ef Merge remote-tracking branch 'origin/master' into davidfischer/enable-timezone-support 2018-11-01 14:21:37 -05:00
Eric Holscher f94aaaa020 Update migrations for latest master 2018-10-31 11:26:15 -05:00
Eric Holscher e9441eb005 Add all migrations that are missing from model changes 2018-10-31 10:10:21 -05:00
Santos Gallegos a8bd00a517 Refactor PublicTask into a decorator task (#4656)
* Refactor PublicTask into a decorator task

* Refactor and docs
2018-10-02 11:33:12 -06:00
David Fischer 73ad35c36c
Merge branch 'master' into davidfischer/enable-timezone-support 2018-09-18 15:11:48 -07:00
Manuel Kaufmann 8b7c2797f8 Upgrade all packages using pur (#4318)
* Upgrade all packages using pur

for reqs in `ls requirements`;
do
  pur
  --skip django-tastypie,django,docker,elasticsearch,pyelasticsearch,commonmark,stripe,djangorestframework,mkdocs,django-allauth,django-filter,mercurial
  --requirement requirements/$reqs;
done

* Run lint with Python 3.6

* Downgrade gitpython to 2.1.10

* Linting error fixed

* Upgrade common

Branch: https://github.com/rtfd/common/pull/17

* Remove old note for testing

* Add note for django-filter

* no-else-return fixed in all the files

Some block needed to be re-idented to fulfill this check.

Basically, if you have a "if something: return another" the "else"
clause is not needed since it will be the common case for the flow to
continue if the condition is not met.

* Only # noqa comment

* Note about gitpython
2018-09-05 09:34:49 -06:00
David Fischer 46db5d9e81
Enable timezone support and set timezone to UTC 2018-08-20 12:09:31 -07:00
Manuel Kaufmann cedda9bf53 Allow to hook the initial build from outside (#4033)
* Allow to hook the initial build from outside

Add a new attribute to `trigger_build` to do not execute the task but
just return the immutable signature of it, so it can be chained into a
bigger task from outside when it's needed to do something else
before/after it's execution.

* Remove unused basic attribute on trigger build

* Split trigger_build to be able to prepare_build first

Added a ``ImportWizard.trigger_initial_build`` method to be override
from corporate site.

Also, instead of using ``trigger_build`` to create the Celery task and
call it immediately, split it to use ``prepare_build`` first to create
the task and use ``trigger_build`` to effectively triggers it.

* Remove now unused signal

* Fix Celery signature creation

* Fix testcases with basic on trigger_build

* Fix mock calls in test cases

* Fix test mock check

* Use async task to attach webhook

* Use proper context object

* Make it compatible with newer Django versions

* Py2 and Py3 compatible line

* Dismiss the sticky message and stay in the same page

* Use Context to render the template

* Lint errors fixed

* Call the task in a sync way properly

* Update common submodule to latest

* Define NonPersistentStorage for one-time notifications

NonPersistentNotification backed together with SiteNotification class
helps us to create one-time site notifications and attach it to a user
without the needs of a request/session object.

The message/notification is saved into the database as a
PersistentMessage and it's marked as read while it's retrived the
first time.

This system is useful for notifications that need to be attached from
a Celery task where we know the user but we don't have the request.

* Make string translatable

* Fix merge conflicts

* Recover accidentally deleted line

* Fixed linting errors

* Replace message_key with reason to be clearer

* Rename non persistent storage class

* Remove old templates

* Make SiteNotification more flexible

- render strings messages as Django Templates
- accept extra_context for the template
- do not crash if the reason is incorrect

* Adapt AttachWebhookNotification to the new features

* Refactor the task to attach a webhook

Instantiate only once the notification and adapt it depending the
context.

Also, if there are no connected services into our application do not
show a message to the user, but log it as a warning.

* Test cases for SiteNotification and NonPersistentStorage

* Remove unnecessary lines

* Show a persistent message for invalid project webhook

If we can setup a valid webhook, we show a persistent message with an
actionable link using our notifications abstraction.

At this point, the message is duplicated because we have a "fixed
template message" also which is planned to be removed soon.

* Improve copy

* Allow to hook the initial build from outside

Add a new attribute to `trigger_build` to do not execute the task but
just return the immutable signature of it, so it can be chained into a
bigger task from outside when it's needed to do something else
before/after it's execution.

* Remove unused basic attribute on trigger build

* Split trigger_build to be able to prepare_build first

Added a ``ImportWizard.trigger_initial_build`` method to be override
from corporate site.

Also, instead of using ``trigger_build`` to create the Celery task and
call it immediately, split it to use ``prepare_build`` first to create
the task and use ``trigger_build`` to effectively triggers it.

* Remove now unused signal

* Fix Celery signature creation

* Fix testcases with basic on trigger_build

* Fix mock calls in test cases

* Fix test mock check

* Use proper context object

* Make it compatible with newer Django versions

* Py2 and Py3 compatible line

* Dismiss the sticky message and stay in the same page

* Use Context to render the template

* Lint errors fixed

* Call the task in a sync way properly

* Update common submodule to latest

* Define NonPersistentStorage for one-time notifications

NonPersistentNotification backed together with SiteNotification class
helps us to create one-time site notifications and attach it to a user
without the needs of a request/session object.

The message/notification is saved into the database as a
PersistentMessage and it's marked as read while it's retrived the
first time.

This system is useful for notifications that need to be attached from
a Celery task where we know the user but we don't have the request.

* Make string translatable

* Use async task to attach webhook

* Fix merge conflicts

* Recover accidentally deleted line

* Fixed linting errors

* Replace message_key with reason to be clearer

* Rename non persistent storage class

* Remove old templates

* Make SiteNotification more flexible

- render strings messages as Django Templates
- accept extra_context for the template
- do not crash if the reason is incorrect

* Adapt AttachWebhookNotification to the new features

* Refactor the task to attach a webhook

Instantiate only once the notification and adapt it depending the
context.

Also, if there are no connected services into our application do not
show a message to the user, but log it as a warning.

* Test cases for SiteNotification and NonPersistentStorage

* Remove unnecessary lines

* Show a persistent message for invalid project webhook

If we can setup a valid webhook, we show a persistent message with an
actionable link using our notifications abstraction.

At this point, the message is duplicated because we have a "fixed
template message" also which is planned to be removed soon.

* Improve copy

* Remove fixed template notification about Project.has_valid_webhook
2018-06-14 12:29:08 -06:00
Manuel Kaufmann 1fa6cf6e57 Handle revoked oauth permissions by the user (#4074)
* Handle general oauth services errors

This is a way to protect ourselves from random errors when syncing
OAuth services (create repository/organizations).

If a global exception is raised, we just skip this service and
continue with the rest of the connected ones.

* Handle revoked oauth permissions by the user

If the user revoked our app permission from the external service web
page, we are not notified. So, next time we try to use it, we will
receive a 401 status code in the first request.

Here we handle this case and we show a Persistent Notification to the
user so he/she knows that a reconnection of the account is needed.

* Remove unnecessary exception handling

* Style

* Add a user persistent notification when social access is revoked

* Raise an Exception when permissions revoked

Raise a simple Exception when the permissions from a particular social
service are revoked instead of using a persistent notification.

We want an Exception because it's immediately communicated using the
PublicTask mechanism. Otherwise, the user needs to refresh the page to
see the notification.

* Remove invalid imports

* Improve comment

* Add final dot to the phrase

* Remove unused return
2018-06-14 11:22:40 -06:00
Eric Holscher 2f5ae97d54
Merge pull request #4134 from italia/paginatealwaysreturn
oauth/services: correct error handling in paginate
2018-05-24 12:27:53 -04:00
Riccardo Magliocchetti 9086279b5d oauth/services: correct error handling in paginate
the else code was dead code as the code in the try block already
returned. Instead return an empty list when it is needed.
2018-05-23 15:34:23 +02:00
Riccardo Magliocchetti 086b0610a2 oauth/services: don't abuse log.exception
log.exception should not be used outside exception handlers
2018-05-23 11:57:00 +02:00
Manuel Kaufmann 5300c90584 Allow import Gitlab repo manually and set a webhook automatically (#3934)
* Allow import Gitlab repo manually and set a webhook automatically

When we import a Project manually we don't have a RemoteRepository
associated with that Project, so we use the old technique of getting
the username and repo names from the URL using
`get_gitlab_username_repo` function.

This case (Manual Import) won't support custom Gitlab installations in
the future with this code.

* Handle username,repo not found

* Test case
2018-05-03 16:31:31 -06:00
Manuel Kaufmann 0c9fafbde5 Resync valid webhook for project manually imported (#3935)
When a project is imported manually, if the user has a valid social
account we setup a webhook for this project.

If the user goes to Integrations and try to resync, we handle that
properly (use the Project.remote_repository only when available,
otherwise use the accounts registered for that service even if they
are not associated with that project)
2018-04-13 15:49:18 -06:00
Manuel Kaufmann 64def2d3b7
Merge pull request #3930 from rtfd/humitos/integrations/fixes
Fix attribute order call
2018-04-10 17:27:46 -05:00
Manuel Kaufmann 65ee5dac4c Resync webhooks from Admin
If the user delete the RTD webhook integration for whatever reason
from the project under the external service (GitHub, etc) when the
Resync button is pressed we re-create / setup the webhook as it was
never existed.
2018-04-10 13:43:25 -05:00
Manuel Kaufmann 6cc40e603d Fix attribute order call
Keep compatibility (order) with the other services
2018-04-10 12:45:15 -05:00
Manuel Kaufmann 5cb3ba6b7b Organize logging levels (#3893)
* Organize logging levels

Log only what we need as ERROR when it's something that we need/want
to take a look that something could be a bug/issue.

There are some ERROR that were replaced by WARNING since they are some
failures but that we don't need to take a look and the message shown
to the user should be enough for them.

* Fix lint issues

* Use the proper exc_info instead of custom formatting
2018-04-05 16:57:30 -06:00
Manuel Kaufmann df8079a18e Allow to import public repositories on corporate site (#3537)
When the DEFAULT_PRIVACY_LEVEL is ``private`` we allow to import
public _and_ private repositories from VCS providers.
2018-01-30 12:21:41 +07:00
Manuel Kaufmann 745dc7483c Fix some linting 2018-01-25 20:43:54 -05:00
Eric Holscher 3cfba49a74
Merge pull request #3419 from rtfd/humitos/oauth/default-avatar
Use default avatar for User/Orgs in OAuth services
2017-12-18 14:05:04 -08:00
Manuel Kaufmann 74ee015e9a Use default avatar for User/Orgs in OAuth services 2017-12-18 14:54:26 -05:00
Manuel Kaufmann 4d10f2e287 Fix D213 using docformatter
prospector \
  --profile-path=/home/humitos/rtfd/code/readthedocs.org \
  --profile=prospector --die-on-tool-error \
  | \
  grep -e "^[a-z] \
  > \
  list-files.txt

for x in `cat list-files.txt`; do \
  docformatter \
    --wrap-summaries=80 \
    --wrap-descriptions=80 \
    --pre-summary-newline \
    --no-blank \
    --in-place readthedocs/$x\
; done
2017-12-15 20:18:06 -05:00
Manuel Kaufmann ba71aa1860 GitLab Integration (#3327)
This adds GitLab integration for project import, including remote repositories and webhooks.
2017-12-07 10:57:49 -07:00
Anthony Johnson 28ae63010e
Fix python usage around oauth pagination
Because of a python quirk, we are currently returning an empty list during
pagination against oauth proviers. The quirk breaks down to this:

```python
def foo():
    try:
        return 'duck'
    except Exception:
        return 'duck'
    finally:
        return 'goose'

assert foo() == 'goose'
```

I didn't realize `finally` behaved in this manner with respect to `return`, so
hey, The More You Know™
2017-11-30 23:30:53 -07:00
Manuel Kaufmann e737d78abd Do not convert to bytes the `refresh_token` (#3273)
* Do not convert to bytes the `refresh_token`

When the body is encoded, a mix of bytes and unicode happens and it
fails at this line

c472e6bf04/requests_oauthlib/oauth2_session.py (L285)

(Pdb) body
u'grant_type=refresh_token&allow_redirects=True&refresh_token=b%27z9BSRKqf2QD4pcDhqj%27&client_id=CLIENT_ID&client_secret=CLIENT_SECRET'

(note those weird `%27` in the string)

* Handle OAuth specific exception properly at `paginate`

* Do not convert the `acess_token` to bytes

* Fix lint :(

* Refactor `paginate` to handle exception in Bitbucket and Github

`paginate` method is shared between both services and each service
implement `get_next_url_to_paginate` and `get_paginated_results`.

* Names and attributes fixes
2017-11-20 10:00:53 -07:00
Anthony 8f9a41947b
Fix missing sync repo task (#3265)
* Fix missing sync repo task

This was not caught on celery 4 upgrade

* Missing apps file, oops
2017-11-15 15:05:20 -07:00
Anthony 96b23c8915
Use new Celery, use new application pattern (#3237)
* Use new Celery, use new application pattern

* Use modern celery
* Drop djcelery
* New pattern for starting celery
* Bump redis to 2.10.6 to avoid startup bug, change autodiscover call

* Update docs mentioning Celery

* Goof on package name

* Missed djcelery import

* Fix rebased task that was missed

* Handle change to Celery group calls in 4.x

* Fix tests

* Fix up some linting issues and problems starting the application

* Fix celery task registration

* We don't need shared_task anymore, swap for readthedocs.worker.app.task

* Fix call to chord
2017-11-10 17:17:36 -07:00
Eric Holscher 547cfb0045 Clean up logging around exceptions 2017-11-08 12:48:14 -07:00
Anthony Johnson 8727cda695
Fix call to allauth adapter
This apparently changed at some point. This was triggered when no accounts were
connected and a project import was attempted. Webhook setup would try to emit a
warning message for the provider.
2017-07-10 12:16:52 -07:00
Anthony 14f0fb72b1 Remove privacy app (#2954)
The privacy app was a strange mixture of various application models managers,
querysets, and syncers. Instead, logic is moved to where we should be using it,
inside the other applications
2017-06-21 14:41:43 -07:00
Anthony accc01535c Finish linting on py3ish code (#2949)
* Resolve first round of linting errors in restapi.

This is a first step and only adds docstrings, etc. It doesn't refactor any
code.

Conflicts:
	prospector-more.yml
	readthedocs/restapi/views/model_views.py

* Reduce unused params.

In several cases, we can use the provided `request` rather than access `self`
(which could make these functions easier to test in isolation). In others, the
unused args could be folded into an unnamed kwarg param.

* Don't replace section variable.

This worked as intended, but only because the overridden section variable
would later evaluate to True. Using a different variable name makes this a bit
less error-prone.

* Minor docstring update

* Fix indent on docstring

* Punt on refactoring

* Fix call args

* Try unpinning testing reqs

* Bump up prospector higher level strictness

* Linting issues: arguments-differ, len-as-condition, redefined variables

* Remove unnecessary elses

This addresses no-else-return, which removes a `return` in an `else` after an
`if` that already has a `return`. This makes the default return more obvious in
most cases.

* Fix new linting issues with import order

Also, update travis and tox config for multiple versions of python

* Another fix on travis envs
2017-06-14 09:38:03 -07:00
Hank Gay 136c59add3 Use @python_2_unicode_compatible to handle __str__/__unicode__ on Django models. 2017-06-01 13:51:53 -04:00
Hank Gay 64116bc541 Make a pass with futurize on stage 2.
Depend on futurize.
2017-05-30 18:10:44 -04:00