Skip to content

query validation filter #481

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 30 commits into from
Sep 19, 2018
Merged
Show file tree
Hide file tree
Changes from 25 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
7b29f36
initial integration of JSONAPIDjangoFilter
n2ygk Aug 23, 2018
dc5ca38
documentation, isort, flake8
n2ygk Aug 24, 2018
6b0dc8c
Forgot to add django_filters to installed_apps
n2ygk Aug 24, 2018
d4fbf24
backwards compatibility for py27 + django-filter
n2ygk Aug 24, 2018
d86d217
handle optional django-filter package
n2ygk Aug 24, 2018
83c4cc0
fix travis to match new TOXENVs due to django-filter
n2ygk Aug 24, 2018
f5792c1
fixed a typo
n2ygk Aug 24, 2018
cbc9d55
add a warning if django-filter is missing and JSONAPIDjangoFilter is …
n2ygk Aug 25, 2018
2f6ba1d
JSONAPIQueryValidationFilter implementation
n2ygk Aug 27, 2018
4f2b75b
improve filter_regex
n2ygk Aug 28, 2018
48b4c51
Merge branch 'JSONAPIDjangoFilter' into JSONAPIQueryValidationFilter
n2ygk Aug 28, 2018
2742d60
rename tests from filter to param
n2ygk Aug 29, 2018
6a8d7ae
easy changes recommended by @sliverc review
n2ygk Aug 29, 2018
db9e1f9
resolve @sliverc review method of using optional django-filter.
n2ygk Aug 29, 2018
68f5e02
Merge branch 'JSONAPIDjangoFilter' into JSONAPIQueryValidationFilter
n2ygk Aug 30, 2018
f0bdbd4
Merge branch 'master' into JSONAPIQueryValidationFilter
n2ygk Sep 17, 2018
64d4af0
remove JSONAPI prefix per #471
n2ygk Sep 17, 2018
23616a2
inadvertently removed when merging master
n2ygk Sep 17, 2018
2c476d9
add QueryValidation filter to NonPaginatedEntryViewset to avoid break…
n2ygk Sep 17, 2018
9b5ab9d
flake8
n2ygk Sep 17, 2018
dbd3d32
100% test coverage for QueryParamaterValidationFilter
n2ygk Sep 17, 2018
11aaf06
move QueryValidationFilter earlier and document how to extend query_r…
n2ygk Sep 17, 2018
57e95cc
QueryValidationFilter to README
n2ygk Sep 17, 2018
a22ca21
py2.7 fix for a non-ASCII quotation mark
n2ygk Sep 17, 2018
0252096
ugh I added back this junk file by mistake again
n2ygk Sep 17, 2018
9e715fa
Change "invalid filter" to "invalid query parameter" for malformed fi…
n2ygk Sep 18, 2018
af10543
renamed to QueryParameterValidationFilter to be clear that this is qu…
n2ygk Sep 18, 2018
c928d72
clearer language
n2ygk Sep 18, 2018
6e008ad
flake8 line length after renaming the class
n2ygk Sep 18, 2018
eed8133
Merge branch 'master' into JSONAPIQueryValidationFilter
sliverc Sep 19, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -173,6 +173,7 @@ override ``settings.REST_FRAMEWORK``
),
'DEFAULT_METADATA_CLASS': 'rest_framework_json_api.metadata.JSONAPIMetadata',
'DEFAULT_FILTER_BACKENDS': (
'rest_framework_json_api.filters.QueryValidationFilter',
'rest_framework_json_api.filters.OrderingFilter',
'rest_framework_json_api.django_filters.DjangoFilterBackend',
'rest_framework.filters.SearchFilter',
Expand Down
32 changes: 30 additions & 2 deletions docs/usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ REST_FRAMEWORK = {
),
'DEFAULT_METADATA_CLASS': 'rest_framework_json_api.metadata.JSONAPIMetadata',
'DEFAULT_FILTER_BACKENDS': (
'rest_framework_json_api.filters.QueryValidationFilter',
'rest_framework_json_api.filters.OrderingFilter',
'rest_framework_json_api.django_filters.DjangoFilterBackend',
'rest_framework.filters.SearchFilter',
Expand Down Expand Up @@ -104,9 +105,32 @@ class MyLimitPagination(JsonApiLimitOffsetPagination):

### Filter Backends

Following are descriptions for two JSON:API-specific filter backends and documentation on suggested usage
Following are descriptions for three JSON:API-specific filter backends and documentation on suggested usage
for a standard DRF keyword-search filter backend that makes it consistent with JSON:API.

#### `QueryValidationFilter`
`QueryValidationFilter` validates query parameters to be one of the defined JSON:API query parameters
(sort, include, filter, fields, page) and returns a `400 Bad Request` if a non-matching query parameter
is used. This can help the client identify misspelled query parameters, for example.

If you want to change the list of valid query parameters, override the `.query_regex` attribute:
```python
# compiled regex that matches the allowed http://jsonapi.org/format/#query-parameters
# `sort` and `include` stand alone; `filter`, `fields`, and `page` have []'s
query_regex = re.compile(r'^(sort|include)$|^(filter|fields|page)(\[[\w\.\-]+\])?$')
```
For example:
```python
import re
from rest_framework_json_api.filters import QueryValidationFilter

class MyQPValidator(QueryValidationFilter):
query_regex = re.compile(r'^(sort|include|page|page_size)$|^(filter|fields|page)(\[[\w\.\-]+\])?$')
```

If you don't care if non-JSON:API query parameters are allowed (and potentially silently ignored),
simply don't use this filter backend.

#### `OrderingFilter`
`OrderingFilter` implements the [JSON:API `sort`](http://jsonapi.org/format/#fetching-sorting) and uses
DRF's [ordering filter](http://django-rest-framework.readthedocs.io/en/latest/api-guide/filtering/#orderingfilter).
Expand Down Expand Up @@ -176,6 +200,7 @@ for `GET http://127.0.0.1:8000/nopage-entries?filter[bad]=1`:
]
}
```

#### `SearchFilter`

To comply with JSON:API query parameter naming standards, DRF's
Expand All @@ -186,6 +211,7 @@ adding the `.search_param` attribute to a custom class derived from `SearchFilte
use [`DjangoFilterBackend`](#djangofilterbackend), make sure you set the same values for both classes.



#### Configuring Filter Backends

You can configure the filter backends either by setting the `REST_FRAMEWORK['DEFAULT_FILTER_BACKENDS']` as shown
Expand All @@ -200,13 +226,15 @@ from models import MyModel
class MyViewset(ModelViewSet):
queryset = MyModel.objects.all()
serializer_class = MyModelSerializer
filter_backends = (filters.OrderingFilter, django_filters.DjangoFilterBackend,)
filter_backends = (filters.QueryValidationFilter, filters.OrderingFilter,
django_filters.DjangoFilterBackend, SearchFilter)
filterset_fields = {
'id': ('exact', 'lt', 'gt', 'gte', 'lte', 'in'),
'descriptuon': ('icontains', 'iexact', 'contains'),
'tagline': ('icontains', 'iexact', 'contains'),
}
search_fields = ('id', 'description', 'tagline',)

```


Expand Down
63 changes: 47 additions & 16 deletions example/tests/test_filters.py
Original file line number Diff line number Diff line change
Expand Up @@ -251,13 +251,16 @@ def test_filter_invalid_association_name(self):
def test_filter_empty_association_name(self):
"""
test for filter with missing association name
error texts are different depending on whether QueryValidationFilter is in use.
TODO: Just change the "invalid filter" to "invalid query parameter" in JSONAPIDjangoFilter?
"""
response = self.client.get(self.url, data={'filter[]': 'foobar'})
self.assertEqual(response.status_code, 400,
msg=response.content.decode("utf-8"))
dja_response = response.json()
self.assertEqual(dja_response['errors'][0]['detail'],
"invalid filter: filter[]")
self.assertIn(dja_response['errors'][0]['detail'],
["invalid filter: filter[]",
"invalid query parameter: filter[]"])

def test_filter_no_brackets(self):
"""
Expand All @@ -270,6 +273,17 @@ def test_filter_no_brackets(self):
self.assertEqual(dja_response['errors'][0]['detail'],
"invalid filter: filter")

def test_filter_missing_right_bracket(self):
"""
test for filter missing right bracket
"""
response = self.client.get(self.url, data={'filter[headline': 'foobar'})
self.assertEqual(response.status_code, 400, msg=response.content.decode("utf-8"))
dja_response = response.json()
self.assertIn(dja_response['errors'][0]['detail'],
["invalid filter: filter[headline",
"invalid query parameter: filter[headline"])

def test_filter_no_brackets_rvalue(self):
"""
test for `filter=` with missing filter[association] and value
Expand Down Expand Up @@ -300,19 +314,9 @@ def test_filter_malformed_left_bracket(self):
self.assertEqual(response.status_code, 400,
msg=response.content.decode("utf-8"))
dja_response = response.json()
self.assertEqual(dja_response['errors'][0]['detail'],
"invalid filter: filter[")

def test_filter_missing_right_bracket(self):
"""
test for filter missing right bracket
"""
response = self.client.get(self.url, data={'filter[headline': 'foobar'})
self.assertEqual(response.status_code, 400,
msg=response.content.decode("utf-8"))
dja_response = response.json()
self.assertEqual(dja_response['errors'][0]['detail'],
"invalid filter: filter[headline")
self.assertIn(dja_response['errors'][0]['detail'],
["invalid filter: filter[",
"invalid query parameter: filter["])

def test_filter_missing_rvalue(self):
"""
Expand All @@ -331,7 +335,7 @@ def test_filter_missing_rvalue_equal(self):
"""
test for filter with missing value to test against
this should probably be an error rather than ignoring the filter:
"""
"""
response = self.client.get(self.url + '?filter[headline]')
self.assertEqual(response.status_code, 400,
msg=response.content.decode("utf-8"))
Expand Down Expand Up @@ -478,3 +482,30 @@ def test_search_multiple_keywords(self):
self.assertEqual(len(dja_response['data']), expected_len)
returned_ids = set([k['id'] for k in dja_response['data']])
self.assertEqual(returned_ids, expected_ids)

def test_param_invalid(self):
"""
Test a "wrong" query parameter
"""
response = self.client.get(self.url, data={'garbage': 'foo'})
self.assertEqual(response.status_code, 400,
msg=response.content.decode("utf-8"))
dja_response = response.json()
self.assertEqual(dja_response['errors'][0]['detail'],
"invalid query parameter: garbage")

def test_param_duplicate(self):
"""
Test a duplicated query parameter:
`?sort=headline&page[size]=3&sort=bodyText` is not allowed.
This is not so obvious when using a data dict....
"""
response = self.client.get(self.url,
data={'sort': ['headline', 'bodyText'],
'page[size]': 3}
)
self.assertEqual(response.status_code, 400,
msg=response.content.decode("utf-8"))
dja_response = response.json()
self.assertEqual(dja_response['errors'][0]['detail'],
"repeated query parameter not allowed: sort")
6 changes: 6 additions & 0 deletions example/views.py
Original file line number Diff line number Diff line change
@@ -1,11 +1,14 @@
import rest_framework.exceptions as exceptions
import rest_framework.parsers
import rest_framework.renderers
from rest_framework.filters import SearchFilter

import rest_framework_json_api.metadata
import rest_framework_json_api.parsers
import rest_framework_json_api.renderers
from django_filters import rest_framework as filters
from rest_framework_json_api.django_filters import DjangoFilterBackend
from rest_framework_json_api.filters import OrderingFilter, QueryValidationFilter
from rest_framework_json_api.pagination import PageNumberPagination
from rest_framework_json_api.utils import format_drf_errors
from rest_framework_json_api.views import ModelViewSet, RelationshipView
Expand Down Expand Up @@ -91,6 +94,9 @@ class NoPagination(PageNumberPagination):

class NonPaginatedEntryViewSet(EntryViewSet):
pagination_class = NoPagination
# override the default filter backends in order to test QueryValidationFilter without
# breaking older usage of non-standard query params like `page_size`.
filter_backends = (QueryValidationFilter, OrderingFilter, DjangoFilterBackend, SearchFilter)
ordering_fields = ('headline', 'body_text', 'blog__name', 'blog__id')
rels = ('exact', 'iexact',
'contains', 'icontains',
Expand Down
1 change: 1 addition & 0 deletions rest_framework_json_api/filters/__init__.py
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
from .sort import OrderingFilter # noqa: F401
from .queryvalidation import QueryValidationFilter # noqa: F401
39 changes: 39 additions & 0 deletions rest_framework_json_api/filters/queryvalidation.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
import re

from rest_framework.exceptions import ValidationError
from rest_framework.filters import BaseFilterBackend


class QueryValidationFilter(BaseFilterBackend):
"""
A backend filter that performs strict validation of query parameters for
jsonapi spec conformance and raises a 400 error if non-conforming usage is
found.

If you want to add some additional non-standard query parameters,
simply override `.query_regex` adding the new parameters but, "with the additional
requirement that they MUST contain contain at least one non a-z character (U+0061 to U+007A).
It is RECOMMENDED that a U+002D HYPHEN-MINUS, "-", U+005F LOW LINE, "_", or capital letter is
used (e.g. camelCasing)." -- http://jsonapi.org/format/#query-parameters
"""
#: compiled regex that matches the allowed http://jsonapi.org/format/#query-parameters
#: `sort` and `include` stand alone; `filter`, `fields`, and `page` have []'s
query_regex = re.compile(r'^(sort|include)$|^(filter|fields|page)(\[[\w\.\-]+\])?$')

def validate_query_params(self, request):
"""
Validate that query params are in the list of valid query keywords
Raises ValidationError if not.
"""
# TODO: For jsonapi error object conformance, must set jsonapi errors "parameter" for
# the ValidationError. This requires extending DRF/DJA Exceptions.
for qp in request.query_params.keys():
if not self.query_regex.match(qp):
raise ValidationError('invalid query parameter: {}'.format(qp))
if len(request.query_params.getlist(qp)) > 1:
raise ValidationError(
'repeated query parameter not allowed: {}'.format(qp))

def filter_queryset(self, request, queryset, view):
self.validate_query_params(request)
return queryset