Matthias Kestenholz: Posts about Django

Keep content managers' admin access up-to-date with role-based permissions

2023-09-20T12:00:00Z

Keep content managers’ Django admin access up-to-date with role-based permissions

Django’s built-in permissions system is great if you want fine-grained control of the permissions content managers should have. The allowlist-based approach where users have no permissions by default and must be granted each permission individually makes a lot of sense to me and is easy to understand.

When we build a CMS at Feinheit we often use the Django administration panel as a CMS. Unfortunately, Django doesn’t provide a way to specify that content managers should have all permissions in the pages and articles app (just as an example). Adding all current permissions in a particular app is straightforward when using the filter_horizontal interface but keeping the list up-to-date later isn’t. When we add an additional content block plugin we always have to remember to also update the permissions after deploying the change – and often, deployment happens some time after the code has been written, e.g. because clients want to approve the change first. What happens all too often is that the manual step of updating permissions gets forgotten.

This has annoyed me (intermittently) for a long time and my preferred solution has always been to give superuser permissions to everyone and trust them to not make changes which they aren’t supposed to according to the Trusted Users Editing Structured Content principle which was mentioned in a Django book I read early in my Django journey.

The basic ideas of my role-based permissions implementation

A recent project has resurfaced this annoyance and I did finally bite the bullet and implement a solution for this in the form of a django-authlib extension. The basic ideas are:

All users are assigned a single role: Single roles sound inflexible, but is good enough for my default use case. Examples for roles could be default (no additional permissions granted), content managers (grant access to the pages and articles apps) or maybe deny auth (deny access to users, groups and permissions).

The permission check is implemented using a single callable: A custom backend is provided whose only job is to call the correct callable for the user’s current role.

The callable either returns a boolean or raises PermissionDenied to prevent other backends from granting access: No new ideas here, it’s exactly what Django’s authentication backends are supposed to do.

Permission checkers for the most common scenarios are bundled: django-authlib only ships one permission checker right now, allow_deny_globs, which allows specifying a list of permission name globs to allow and to deny. Deny overrides allow as is probably expected.

Using roles in your own project

Specify the available roles in your settings and add the authentication backend:

from functools import partial
from authlib.roles import allow_deny_globs
from django.utils.translation import gettext_lazy as _

AUTHLIB_ROLES = {
    "default": {"title": _("default")},
    "staff": {
        "title": _("editorial staff"),
        "callback": partial(
            allow_deny_globs,
            allow={
                "pages.*",
                "articles.*",
            },
        ),
    },
}

AUTHENTICATION_BACKENDS = (
    # This is the necessary additional backend
    "authlib.backends.PermissionsBackend",
    # Maybe you want to use authlib's email authentication ...
    "authlib.backends.EmailBackend",
    # ... or the standard username & password combination:
    "django.contrib.auth.backends.ModelBackend",
)

You have to extend your user model (you have to use a custom user model if you’re not using django-authlib’s little_user.User):

from authlib.roles import RoleField

class User(AbstractUser):
    # ...
    role = RoleField()

And that’s basically it.

Of course the globbing is flexible, you could also allow users to view all objects:

partial(allow_deny_globs, allow={"*.view_*"})

Or you could block users from deleting anything:

partial(allow_deny_globs, deny={"*.delete_*"})

And as mentioned above, you can also combine allow and deny (deny wins over allow) or even provide your own callables. If you provide your own callable it must accept user, perm and obj (which may be None) as keyword arguments. Implementing such a callable is probably less work than implementing an authentication backend yourself; I had to do more work than initially expected because only implementing .has_perm isn’t sufficient if you want to see any apps and models in the admin index page. The current allow_deny_globs implementation is nice and short:

def allow_deny_globs(user, perm, obj, allow=(), deny=()):
    for rule in deny:
        if fnmatch(perm, rule):
            raise PermissionDenied
    return any(fnmatch(perm, rule) for rule in allow)

My reaction to the block-driven CMS blog post

2023-08-23T12:00:00Z

My reaction to the block-driven CMS blog post

This morning I read an interesting post on the Lincoln Loop blog called Building a Future-Proof Platform with Block-Driven CMS. It shouldn’t come as a surprise to those (few 😄) who know my work in the area of content management systems that the post resonated with me. I found the description of the advantages of block-based CMS editing very clear and I like the emphasis on structuring data well so that it can be reused for multiple distribution channels.

Of course django CMS isn’t the only way to implement a block-driven CMS using Django. Since its inception FeinCMS was always the smaller, faster and nimbler counterpart to it, achieving the same basic goals with a fraction of the code and maintenance headaches. django CMS always seems to trail the official releases of Django. django-content-editor and feincms3 are almost always compatible with the development version of Django by way of running the tests with the main branch as well. This allows me to be an early adopter of upcoming Django releases with a software stack that’s already well tested, or also to report bugs to the Django project itself. All that probably wouldn’t be possible if feincms3 and its dependencies supported all the things django CMS does, but it doesn’t have to to be useful.

django-content-editor and feincms3 are the legacy of FeinCMS in an even smaller, even more maintainable and even more composable package and while I’m definitely always checking out other Django-based CMS I’m persuaded that sticking with feincms3 is a good choice.

Weeknotes (2023 week 33)

2023-08-20T12:00:00Z

Weeknotes

I’m not sure if I should call these posts weeknotes when I see the posting schedule, but oh well. Keep expectations up but also practice forgiveness when not meeting them, it’s fine really.

`py_modules` using hatchling

I converted speckenv and django-sitemaps after finding the following very helpful post on packaging projects consisting of Python modules without any packages: Packaging of single Python module projects with Hatch/Hatchling. It’s very easy in hindsight, but that’s basically always the case.

The relevant part is including the files in the build:

[tool.hatch.build]
include = [
  "speckenv.py",
  "speckenv_django.py",
  "speckenv_django_patch.py",
]

That’s all.

django-debug-toolbar and tracing the cause of DB queries in an async world

I have also started investigating what would have to be changed in django-debug-toolbar to make it fully support async Django. We currently patch Django’s database cursors per thread, which works fine in sync Django land to attribute SQL queries to a particular request/response cycle.

Since async Django executes DB queries in a thread pool executor and the rest of the work happens inside awaitables (async land) I don’t immediately see a way how we could do the same thing. It doesn’t seem possible to find out which task spawned another task (without dropping down to C?) but maybe there’s something I’m overlooking. I hope that someone smarter than me finds a way :-) or that I find the time and motivation to either find a way using Python or using C/Rust/whatever.

Releases

feincms3-sites 0.16: I added basic support for i18n_patterns when using feincms3-sites with its default_language_middleware (which allows setting a default language per site in case there is no other mechanism overriding it, such as i18n_patterns).
feincms3-cookiecontrol 1.4.1: The privacy policy is now linked inside the banner text instead of adding a link after the text. Looks much nicer.
speckenv 5.0: Finally released changes made a long time ago which make one edge case when parsing settings less surprising.
django-debug-toolbar 4.2: I didn’t do much work here again, mostly code reviews, some changes to the ruff configuration and general polishing. I also didn’t do the release itself, that was handled by Tim. Thanks!
FeinCMS 23.8: Fixes for Pillow 10, and some feincms3 / django-content-editor interoperability improvements which make it easier to reuse plugins/content types.
feincms3 4.1: Some basic support for using the apps middleware with async Django. Not documented yet and not deployed anywhere but it basically works. Some documentation edits and changes to the inline CKEditor styling because of the recent changes to Django admin’s CSS.

Composition over inheritance: The case for function-based views

2023-08-11T12:00:00Z

Composition over inheritance: The case for function-based views

A recent conversation with Carlton on Mastodon prompted me to write down some of my thoughts re. function- vs class-based views in Django.

The early days

When I started using Django some time after 0.96 and 1.0 all views were function based. Except when you added a class with a def __call__() method yourself – that was always possible but not really comparable to today’s class-based views.

The introduction of class-based views

Class based views (both generic versions and the base View) were introduced to Django in 2010. Judging from the ticket tracker the main motivation was to avoid adding yet another argument to the generic function-based views (GFBV) which were available in Django back then.

The GFBV’s argument count was impressive. Two examples follow:

def object_detail(request, queryset, object_id=None, slug=None,
    slug_field='slug', template_name=None, template_name_field=None,
    template_loader=loader, extra_context=None,
    context_processors=None, template_object_name='object',
    mimetype=None):
    ...

def archive_month(request, year, month, queryset, date_field,
    month_format='%b', template_name=None, template_loader=loader,
    extra_context=None, allow_empty=False, context_processors=None,
    template_object_name='object', mimetype=None, allow_future=False):
    ...

The GFBVs were immediately deprecated when GCBVs were introduced and later removed in 2012.

Class-based views have to be adapted by calling the View.as_view() method; as_view() returns arguably the thing which is viewed (sorry) as the view by Django, it’s the thing which gets called with a request and is expected to return a response. This thing in turn instantiates the view object once per request; this means that self can be used to save request-specific data such as self.request, self.args but also custom attributes.

The GCBV code is extremely factored and decomposed. The Classy Class-Based Views site mentions that the UpdateView has 10 separate ancestors and its code is spread across three files. But, the view code for instantiating a model object and handling a form really isn’t that complex. Most of the complexity is handled by Django itself, in the request handler and in the django.forms package. So, what’s the reason for all this?

Generic views could be simple

I wish that the existing generic views had better building blocks instead of a big hierarchy of mixins and multiple inheritance which is probably not understood by anyone without checking and re-checking the documentation, the code, or the excellent Classy Class-Based Views. Certainly not by me.

In my ideal world, generic views would be composed of small reusable and composable functions which wuld cover 80% of use cases with 20% of the code. And if not, you could copy the whole code of the view, change or introduce a line or two and leave it at that. And since the functions do one thing (but do that well) you can immediately see what they are doing and why. You’d avoid the Hollywood Principle (Don’t call us, we’ll call you) in your code. Sure, your view is called by Django but you don’t have to introduce more and more layers of indirection.

The internet is full of advice that you should prefer composition over inheritance. Let’s try to outline what generic views could look like if they followed the composition paradigm. Note that the goal isn’t to gain points by showing that the resulting code is shorter. One important goal is maintainability by being easier to understand. Another important goal is showing a better path from a beginner’s use of views to an experts understanding of everything underneath it by bridging the gap using more powerful building blocks which don’t leave all the minutiae to you if the defaults don’t work.

Some repetition caused by copy pasting is fine. Not all identical three lines of code are the same. The Three Strikes And You Refactor rule¹ leads to better and more maintainable code than following an extreme interpretation of the DRY (Don’t Repeat Yourself) principle.

ListView and DetailView

I’m going to profit from Django’s shortcuts module and also from feincms3’s shortcuts module which offers functions for rendering pages for single objects or lists of objects. The render_list and render_detail functions implement the same way of determining the template paths as the generic views use (for example <app_name>/<model_name>_detail.html) and the same way of naming context variables (object and <model_name> for the object, object_list and <model_name>_list for the list) as well as pagination but nothing more.

Here’s a possible minimal implementation of a list and detail object generic view:

# _get_queryset runs ._default_manager.all() on models and returns
# everything else as-is. It's the secret sauce which allows using models,
# managers or querysets with get_object_or_404 and friends.
from django.shortcuts import get_object_or_404, _get_queryset
from feincms3.shortcuts import render_list, render_detail

def object_list(request, *, model, paginate_by=None):
    return render_list(request, _get_queryset(model), paginate_by=paginate_by)

def object_detail(request, *, model, slug, slug_field="slug"):
    object = get_object_or_404(model, **{slug_field: slug})
    return render_detail(request, object)

You want to change the way a single object is retrieved? You could do that easily but not by adding configuration-adjacent values in your URLconf but rather by adding a view yourself:

def article_detail(request, year, slug):
    object = get_object_or_404(Article.objects.published(), year=year, slug=slug)
    return render_detail(request, object)

urlpatterns = [
    ...
    path("articles/<year:int>/<slug:slug>/", article_detail, name=...),
    ...
]

I don’t think that was much harder than a hypothetical alternative:

urlpatterns = [
    ...
    path(
        "articles/<year:int>/<slug:slug>/",
        object_detail,
        {
            "model": Article.objects.published(),
            "object_kwargs": ["year", "slug"],
        },
    ),
    ...
]

And think about the internal implementation of the object_detail view. Viewed one additional feature at a time it may be fine but when adding up everything it would probably be quite gross.

The additional benefit is that it shows beginners the way to intermediate skills – writing views isn’t hard, and shouldn’t be.

Finally, the official way of overriding DetailView.get_object() (I think!) doesn’t look that good compared to the def article_detail() view above:

class ArticleDetailView(generic.DetailView):
    def get_object(self, queryset=None):
        if queryset is None:
            queryset = self.get_queryset()
        return get_object_or_404(queryset, year=self.kwargs["year"], slug=self.kwargs["slug"])

Did you know that get_object() has an optional queryset argument? I certainly didn’t. It seems to be used by the date-based generic views but they also have their own get_object() implementation so who knows, really.

Detail view with additional behavior

def article_detail(request, year, slug):
    object = get_object_or_404(Article.objects.published(), year=year, slug=slug)
    data = (request.POST, request.FILES) if request.method == "POST" else ()
    form = CommentForm(*data)
    if form.is_valid():
        form.instance.article = object
        form.save()
        return HttpResponseRedirect(".#comments")
    return render_detail(request, object, {"comment_form": form})

A counterexample would be to move the endpoint which accepts a comment POST request somewhere else. But then you’d also have to keep the different CommentForm instantiations in sync.

You could also override get_context_data() to add the comment form to the context and override post() to instantiate check the form’s validity. But then you’d have to make sure that an eventual invalid form is handled correctly by get_context_data(). It’s not hard but it certainly isn’t as straightforward as the example above either.

The custom view is the most obvious way of keeping the form instantiation in one place.

Form views

Generic create and update views could look something like this, again reusing the shortcuts mentioned above:

def save_and_redirect_to_object(request, form):
    object = form.save()
    return redirect(object)

def get_form_instance(request, *, model, form_class, instance=None):
    assert model or form_class, "Provide at least one of model and form_class"
    if form_class is None:
        form_class = modelform_factory(model)
    data = (request.POST, request.FILES) if request.method == "POST" else ()
    return form_class(*data)

def object_create(request, *, model=None, form_class=None, form_valid=save_and_redirect_to_object):
    form = get_form_instance(request, model=model, form_class=form_class)
    if form.is_valid():
        return form_valid(request, form)
    return render_detail(request, form.instance, {"form": form}, template_name_suffix="_form")

def object_update(request, *, model, slug, slug_field="slug", form_class=None, form_valid=save_and_redirect_to_object):
    object = get_object_or_404(model, **{slug_field: slug})
    form = get_form_instance(request, model=object.__class__, form_class=form_class, instance=object)
    if form.is_valid():
        return form_valid(request, form)
    return render_detail(request, form.instance, {"form": form}, template_name_suffix="_form")

You want to redirect to a different URL and maybe emit a success message? Easy:

def article_form_valid(request, form):
    form.save()
    messages.success(request, _("Successfully updated the article."))
    return redirect("articles:list")

urlpatterns = [
    ...
    path(
        "<slug:slug>/update/",
        object_update,
        {"model": Article, "form_valid": article_form_valid},
        name=...
    ),
    ...
]

Yes, these generic views wouldn’t allow overriding the case when a form was invalid. But, I’d assume that displaying the form with error messages is the right thing to do in 90% of the cases. And if not, write your own specific or generic view? After all, with the mentioned tools it won’t take up more than a few lines of straightforward code. (If the code was tricky it would be different. But views shouldn’t be tricky.)

Adding more form_valid handlers should be mostly painless. A few examples inspired by Django’s generic editing documentation:

def save_and_redirect_to(url):
    def fn(request, form):
        form.save()
        return redirect(url)
    return fn

def send_mail(request, form):
    form.send_email()
    return HttpResponseRedirect("/thanks/")

def set_author_and_save(request, form):
    form.instance.created_by = request.user
    object = form.save()
    return redirect(object)

You could also couple the form a bit to the request and do something like:

def process_form(request, form):
    return form.process(request)

Sure, forms probably shouldn’t know much about requests. But then, Django is a framework for perfectionists with deadlines and sometimes practicality beats purity.

Date-based generic views

I think I would want to offer a few analyzers which allow easily returning a data structure suitable for rendering links for yearly, monthly, weekly or even daily (who writes that much?) archives. The .dates() queryset method method should be a big help there.

The archive views themselves are straightforward adaptations of the object_list view above.

It may feel like leaving out the actually hard part but I’d have to be convinced that this is actually a hard problem and not just a problem of making basically arbitrary choices which people then adapt to and then think that this is the way things should be since it’s the way things are.

Wrapping up

Some points this post could have made or tried to make are made much better by Luke Plant in the guide Django Views - The Right Way. I don’t generally think that class-based views never make sense. I also don’t think that people shouldn’t use the available tools. I just think that I, myself, don’t want to use them, and I also think that I’m still happier with lambda request: HttpResponseRedirect(...) than with generic.RedirectView.as_view(url=...). The point isn’t to compare the character count. The point is: Does the RedirectView cause a permanent or a temporary redirect? I had to look it up for a long time, and then it changed. The former is completely obvious.

Closing words

I know that people have strong opinions. I’m not interested in all of them. I’m mostly interested in design critiques and arguments regarding the beginner to intermediate skills argument. It’s fine if CBVs work fine for you, and there’s no need to feel challenged by this post.

Thanks for reading!

Also called the WET rule (Write Everything Twice). (Not coined by me.) ↩

Weeknotes (2023 week 30)

2023-07-28T12:00:00Z

Weeknotes

Async Django

I have used Django Channels successfully in a few projects from 2017 to 2019. A few months back I have worked with Starlette. And now I have finally started digging into using Django itself with an ASGI server, and not just for one or two views but also including the middleware stack etc since I also need authentication, not just an endpoint forwarding requests to a remote server. I have looked at Granian, an RSGI/ASGI server written in Rust. But for now I am using uvicorn.

Django truly has come a long way but there’s much left to do. Django 5.0 is looking great already, but 4.2 misses many pieces still. I am really really glad Django wants to stay backwards compatible but I wish I could wave a magic wand and upgrade everything to async. Adding a prefixes everywhere for the async version is certainly a good compromise and probably the way to go but it’s just not that nice.

I have been playing around with making feincms3’s applications middleware async compatible because I want the full middleware stack to be async. The code is already released but undocumented and not even mentioned in the changelog. So, feel free to play around with it but it’s not supposed to be stable or supported yet.

Releases

feincms3 4.1: Switched to hatchling and ruff. Updated the feincms3-sites docs. Some async updates mentioned above. A Django 4.2 admin CSS update for the inline CKEditor.
feincms3-forms 0.4: Switched to hatchling and ruff. Started defining default icons for the form fields content editor plugins.
django-ckeditor 6.7: I’m still maintaining the CKEditor 4 integration for Django even though CKEditor 4 itself isn’t supported anymore. Minor updates to the editor itself and Pillow compatibility updates.
feincms3-cookiecontrol 1.3.2: The cookie banner doesn’t generate an empty <div class="f3cc"> element anymore if there’s nothing to add inside (e.g. if the user only accepted necessary cookies).

How ruff changed my Python programming habits

2023-07-26T12:00:00Z

How ruff changed my Python programming habits

ruff isn’t just a faster replacement for flake8, isort and friends.

With other Python-based formatters and linters there’s always a trade off between development speed (waiting on git commit is very boring) and strictness.

ruff is so fast that enabling additional rules is practically free in terms of speed; the only question is if those rules lead to better, or maybe just to more correct and consistent code.

I have long been using pre-commit, and even longer flake8, black, isort. I have written a piece about flake8 and the value of standards almost 9 years ago and have continued moving in the mentioned direction ever since.

These days I have enabled a wide variety of rules. I’m not sold on all of them (looking at you, pylint) and I’m definitely not of the opinion that rules which I’m not using currently are worthless. I didn’t even know most of these rules before starting to use ruff, and ruff making them easy and painless to use (without a measureable performance penalty) has certainly lead to me annoying my coworkers with a growing set of enabled rules.

Rules

The current ruleset and some justifications for it follows.

pyflakes, pycodestyle

No justification necessary, really.

mccabe

I like the cyclomatic complexity checker, but I have relaxed it a bit. I find it very useful to avoid complex code, but some code is totally straightforward (e.g. building a queryset from a wide variety of query parameters) but still has many if statements. I’d rather allow more complexity instead of sprinkling the code with # noqa statements.

isort

Sorted imports are great.

pep8-naming

Mostly great except when it flags Django’s migration files. The filenames always start with numbers and that’s obviously not a valid Python module name, but it’s not supposed to be.

pyupgrade

pyupgrade is totally awesome.

flake-2020

Avoiding non future-proof uses of sys.version and sys.version_info is a good idea, no questions about that.

flake8-boolean-trap

Sometimes annoying, mostly useful. I don’t like that the plugin flags e.g. with_tree_fields(True) or with_tree_fields(False) because I don’t think this could be possibly misread. But, apart from these edge cases it really is a good idea, especially since keyword-only arguments exist and those aren’t flagged by this rule.

flake8-bugbear

Mostly useful. I have disabled the zip() without strict= warning.

flake8-comprehensions

Checks for unnecessary conversions between generators and lists, sets, tuples or dicts.

flake8-django

I actually like consistency. I also like flagging fields = "__all__", but this check shouldn’t trigger in admin ModelForm classes, really. I probably have to add another entry to [tool.ruff.per-file-ignores] for this.

flake8-pie

Quite a random assortment of rules. I like the no-unnecessary-pass and no-pointless-statements rules, among others.

flake8-simplify

Nice simplifications. I’m not sure if ternary opeartors are always a plus, especially since they hide the branching from coverage.

flake8-gettext

Enormously useful and important. I don’t know how many times I have encountered broken code like gettext("Hello {name}".format(name=name)) instead of gettext("Hello {name}").format(name=name).

pygrep-hooks

Avoids eval(). Avoids blanket # noqa rules (always be specific!)

pylint

I have been using all pylint rules for some time. The pylint refactoring rules (PLR) did prove to be very annoying so I have reverted to only enabling errors and warnings.

The two main offenders were PLR0913 (too many arguments) and PLR2004 (magic value comparison). The former would be fine if it would count keyword-only arguments differently; it’s certainly a good idea to avoid too many positional parameters, I don’t think keyword-only parameters are that bad. The latter is bad because often the magic value is really obvious. If you’re writing code for the web you shouldn’t have to use constants for the 200 or 404 status codes; one can assume that they are well known.

RUF100

Ruff is able to automatically remove # noqa statements which don’t actually silence any warnings. That’s a great feature to have.

Line length

Yes, let’s go there. I still don’t use longer lines than +/- 80 characters, but I have disabled all line length warnings these days. I don’t want to be warned because I didn’t break a string across lines.

Rules I don’t like

flake8-builtins: Too many boring warnings. I didn’t even want to know that copyright is a Python builtin.
flake8-logging-format: Not generally helpful. Avoiding different exception strings so that e.g. Sentry can group exceptions more easily is a good idea, but the rule generated so many false positives as to be not useful anymore.

Final words (for now)

I really hope that Black is integrated into ruff one day.

Also, I hope that ESLint and prettier will be replaced by a faster tool. I have my eyes on a few alternatives, but they are not there yet for my use cases.

Weeknotes (2023 week 29)

2023-07-21T12:00:00Z

Weeknotes

I have mainly done work in private projects this week. Not much to talk about. Except for the ZIP file content-type bug which was interesting enough to justify its own blog post.

Releases

django-cabinet 0.13: I converted the package to use ruff, hatchling; started running CI tests using Python 3.11. The internals of the Django admin’s filters have changed to allow multi-valued filters, this has required some changes to the implementation of the folder filter. I opted to using a relatively ugly django.VERSION hack; but that’s not too bad since such branches will be automatically removed by the awesome django-upgrade. I would have tried finding other ways in the past but now that old compatibility code can be removed by a single run of django-upgrade (respectively pre-commit) there really is no point to doing it in a different way.

Serving ZIP files using Django

2023-07-18T12:00:00Z

Serving ZIP files using Django

I have generated ZIP files on the fly and served them using Django for a time. Serving ZIP files worked well until it didn’t and browsing StackOverflow etc. didn’t produce clear answers either. The development server worked fine, but gunicorn/nginx didn’t.

In the end, I had to change content_type="application/zip" to content_type="application/x-zip-compressed". I still don’t know what changed and I have only theories why that’s necessary, but maybe it helps someone else. Sometimes it’s better to be dumber about it.

Weeknotes (2023 week 28)

2023-07-12T12:00:00Z

Weeknotes (2023 week 28)

Releases

html-sanitizer 2.2: Made the sanitizer’s configuration initialization more strict. Strings cannot be used anymore in places where the sanitizer expects a set (resp. any iterable). It’s useful that strings are iterable in Python and I wouldn’t want to change that, but the fact that ("class") is a string and not a tuple makes me sad. The fact that tuples are created by , and not by () will always trip up people.
feincms3-language-sites 0.1: The version number is wrong but whatever. I’m certainly happy with the state of things. The big change in 0.1 is that Page.get_absolute_url no longer generates protocol-relative URLs. Depending on the value of SECURE_SSL_REDIRECT it automatically prepends either http: or https:.
django-authlib 0.15: django-authlib’s admin Single Sign On module now supports a hook to automatically create staff users when a matching user doesn’t exist already. I don’t plan to use this functionality myself and I have recommended people to implement the functionality themselves using the tools in django-authlib if they need it, but the change was so small and well-contained that adding it to the core made sense to me.

pipx inject

We learned that pipx seems to remember injected packages even across pipx reinstall invocations. Not too surprising now that we know it, but we certainly spent some time scratching our heads. pipx uninject was the thing we needed to stop pipx from installing an old version of a dependency instead of the one being specified in pyproject.toml.

hatchling and data files

I’m very confused by the way hatchling sometimes includes data files and sometimes it doesn’t. I had to add [tool.hatch.build] include=["authlib/"] to django-authlib’s pyproject.toml file to make it include HTML files from subpackages. Maybe the subpackages are the reason, but I’m not sure.

Payment providers that must not be named

I have spent hours and hours battling with the badly documented, incomplete, inconsistent and confusing API of a (not that well known) payment provider based in Switzerland. I’m surprised that this still happens years and years after Stripe started offering a really well thought out and documented API geared towards programmers. It’s really sad because when the same structure is named with differing naming conventions (e.g. snake_case vs. camelCase) in different parts of the API you just know that somebody spent too much time writing too much code instead of reusing already existing functionality.

Weeknotes (2023 week 26)

2023-06-30T12:00:00Z

Weeknotes (2023 week 26)

Releases

I released updates to a few of my packages; I have continued converting packages to hatchling and ruff while doing that.

New releases in the last two weeks include:

django-tree-queries 0.15: Added a new function, .without_tree_fields() to the queryset which can be used to avoid the .with_tree_fields(False) boolean trap warning.
feincms3-cookiecontrol 1.3.1: This small update allows replacing the feincms3 noembed.com oEmbed code using other libraries such as micawber which support a wider range of URLs while still gating the embed behind users’ explicit consent.
feincms3-downloads 0.5.3: Updated translations.
django-ckeditor 6.6.1: Updated the bundled CKEditor 4 and merged a pull request adding better integration with Django admin’s dark mode.
django-js-asset 2.1: Just basic maintainability and packaging updates. The JS() implementation itself is untouched since February 2022.
html-sanitizer 2.0: Not really a backwards incompatible change (at least not according to the tests); I just wanted to avoid 1.10 and go directly to 2.0 this time.

GitHub projects

We are using GitHub project boards more and more. It definitely isn’t the most versatile way of managing projects but it sort-of hits the sweet spot for us. I’m mostly happy with it, and it seems to me that applying the rule of least power to project management software may not be such a bad idea after all.

The built-in workflows are a bit boring and limited; especially the fact that it seems impossible to automatically add issues to the project when using multiple repositories. Luckily, actions/add-to-project exists so that’s not really a big problem.

To cloud or not

I had a long discussion with a colleague about containerization, Kubernetes, self-hosting, etc. etc. and I still don’t know where I stand. I can honestly say that the old way of hosting (ca. 2010) still works fine. I worry about the deterioriation of service quality we’re seeing and sometimes I really would like to have root to apply quick fixes where now I have to jump to hoops just to get what I already know I need. Annoying. But migrations are annoying as well.

Scheduled publishing

I augmented the script generating this website with scheduled publishing support while again reducing the number of lines in the file. The code is still formatted using black and ruff, while only ignoring line-length errors (I do this everywhere now to avoid breaking up long strings, not to put much code onto single lines) and allowing named lambdas. The weeknotes from two weeks ago where published by GitHub actions’ cron scheduling support.

I like programming more than writing (even though I like writing)

I notice that writing is the first thing I start skipping when I have to prioritize. Programming, biking, gardening come first. That’s fine, really. But I’m still a bit sad that I do not manage to at least put out a short weekly weeknotes entry.

FeinCMS is a dead end (but feincms3 is not)

2023-06-19T12:00:00Z

FeinCMS is a dead end (but feincms3 is not)

I wouldn’t encourage people to start new sites with FeinCMS. Five years ago I wrote that FeinCMS is used in a few flagship projects which we’re still actively developing, which means that FeinCMS won’t be going away for years to come. That’s still true but less and less so. We’re actively moving away from FeinCMS where we can, mostly towards feincms3 and django-content-editor.

FeinCMS lives on in django-content-editor and feincms3; not only in spirit but also in (code) history, since django-content-editor contains the whole history of FeinCMS up to and including the beginning of 2016.

The implementation of FeinCMS is too expensive to clean up without breaking backwards compatibility. I still wish I had pursued an incremental way back then which would have allowed us to evolve old projects to the current best way of doing things (tm), but it didn’t happen and I’m not shedding too many tears about that since I’m quite happy with where we’re at today.

That basically means that I won’t put any effort into bringing FeinCMS and django-content-editor closer together. I haven’t spent much time on that anyway but now my mind is made up that this wouldn’t be time well spent. That being said, some of the items mentioned in the blog post linked above are available in django-content-editor now.

Weeknotes (2023 week 24)

2023-06-16T12:00:00Z

Weeknotes (2023 week 24)

Life happened and I missed a month of weeknotes. Oh well.

django-debug-toolbar 4.1

We have released django-debug-toolbar 4.1. Another cycle where I mostly contributed reviews and not much else. Feels great :-)

Going all in on hatch and hatchling

I got to know hatch because django-debug-toolbar was converted to it. I was confused as probably anyone else with the new state of packaging in Python world. After listening to a few Podcasts (for example Hatch: A Modern Python Workflow) I did bite the bullet and started converting projects to hatch as mentioned some time ago. I have converted a few other projects in the meantime because the development experience is nicer. Not much, but enough to make it worthwile. feincms3-sites is the latest package I converted.

CKEditor 5’s new license and django-ckeditor

The pressure is on to maybe switch away from CKEditor 4 since it probably will not be supported after June 2023. It’s totally understandable that the CKEditor 5 license isn’t the same as before, but I’m not sure what that means for the Django integration django-ckeditor which I’m maintaining since a few years. I don’t actually like the new capabilities of CKEditor all that much and don’t intend to use them; maybe it would be better to use a build of ProseMirror in the CMS since we’re intentionally only using a very small subset of the features most rich text editors offer.

Mountain biking.

My mountain bike is repaired, I’m back on the trail.

CSS variables and immutability

2023-06-14T12:00:00Z

Using CSS variables¹ to ship customizable CSS in Django apps

I have been working with SASS for a long time but have been moving towards writing CSS with a few PostCSS goodies in the last years. At first, I just replaced the $... with var(--...) and didn’t think much about it. The realization that CSS variables can be more than that came later. Edit basic values directly in the browser and immediately see the results! Change CSS depending on media queries or the cascade!

With all that power came back the wish to not just ship backend and HTML code in Django apps I (help) maintain but also reusable CSS, with a few overrideable CSS variables for basic changes to the visual style. Loading .scss files from somewhere inside venv/lib/python3.11/site-packages/<package>/styles/ would of course have been possible, but very obscure. Also, not everyone puts their virtualenv at venv, the README instructions for those packages would quickly have become unwieldy. CSS variables paved the way for shipping CSS as a Django static file while still allowing customizability by leveraging the functionality of the browser itself instead of the frontend build toolchain.

Patterns for overrideable values

A pattern for defining defaults for CSS variables is to always define the fallback (the example is intentionally bad but inspired by real world experiences when developing feincms3-cookiecontrol):

.box {
  background: var(--box-background, #222);
  color: var(--box-foreground, #fff);
}

Less repetition (but trouble awaits)

If --box-background isn’t set the var() function falls back to the second argument, #222. Repeating this value over and over gets annoying quickly, so you define a few defaults on the :root element and use those variables in the code, without specifying the default again:

:root {
  --box-background: #222;
  --box-foreground: #fff;
}

.box {
  background: var(--box-background);
  color: var(--box-foreground);
}

The project can now override the default background color using:

:root {
  --box-background: #444;
  --box-foreground: #ccc;
}

Of course now you’re back at the mercy of CSS loading order. If the app’s CSS is loaded first, everything works. If not, your custom value is immediately overwritten. You could avoid this by overwriting the default lower in the cascade:

.box {
  --box-background: #444;
  --box-foreground: #ccc;
}

Great, everything works again!

Later, the box also contains a button which uses a different background but the same foreground, so of course we add more variables in the package:

:root {
  --box-background: #222;
  --box-foreground: #fff;
  --box-button-background: #333;
  --box-button-foreground: var(--box-foreground);
}

What happens now when overwriting the --box-foreground variable just for the .box element?

You’re not sure? I certainly wasn’t and am not. But what I remember happening was that the overridden foreground color was just applied to the text and not to the button itself. I was confused (it seems clearer in hindsight…)

A better way

If values are supposed to be overridden and only used inside components, a better way is to define local CSS for components by following a convention (underscore prefix for local/private variables):

/* Defined on .box, not :root */
.box {
  --_background: var(--box-background, #222);
  --_foreground: var(--box-foreground, #fff);
  --_button-background: var(--box-button-background, #333);
  --_button-foreground: var(--box-button-foreground, var(--_foreground));
}

And then you only use the prefixed versions inside the component:

.box {
  background: var(--_background);
  color: var(--_foreground);
}

.box__button {
  background: var(--_button-background);
  color: var(--_button-foreground);
}

The --box-* variables are undefined by default; the only time when they are set is when the user of the package wants to override those values. If you only overide the box foreground the button also inherits the new foreground color. And while there would certainly be a way to achieve the same thing with the old way above it’s certainly not as simple to explain.

The reason why it’s simple to explain is immutability. The CSS variables which are overrideable by the user are only ever read by the package, never written to.

Custom properties would probably be the more correct naming, but CSS variables is nicer to say. ↩

Weeknotes (2023 week 17)

2023-04-28T12:00:00Z

Weeknotes (2023 week 17)

Birthday

Another year achieved. Feels the same as last year. I’m glad.

feincms3-cookiecontrol

I have released feincms3-cookiecontrol 1.3. Mostly cleanups since 1.2, but also a new translation (already announced here). The script size has been reduced from 4519 bytes to 4228 bytes (-6.5%) while keeping all features intact. The reduction is totally meaningless but it was fun to do.

oEmbed

I have been digging into the oEmbed spec a bit. I didn’t even know that a central list of providers exists. Noembed still works great to embed many different types of content but I worry more and more about its maintenance state. Reimplementing the interesting parts shouldn’t be that hard, but maybe I don’t have to do this myself. oEmbedPy looks nice, I hope I get a chance to play around with it.

Weeknotes (2023 week 16)

2023-04-21T12:00:00Z

Weeknotes (2023 week 16)

Experiments with Stable Diffusion

A friend and myself threw a few scripts together to automatically finetune a Stable Diffusion model using images downloaded from Google Image search. It’s terrifying how easy and fast generating fake news including photorealistic images can be and will be already. And seeing how fast those models improve it’s just a matter of time until we can trust photos even less than now. Manipulating images has been possible for a long time of course, but it hasn’t been a “commodity” until now.

I definitely also see upsides in the new machine learning technologies but I fear that there’s a real danger to trust, and in extension to democracy.

This technology and what we did will be a part of an upcoming SRF Kulturplatz broadcast, or so I hope. It’s high time that the public knows what’s possible. It’s about ,edia literacy really.

I’m not that pessimistic though. I just hope that this time the thoughtfulness will prevail over pure profit seeking. (Did I really write that.)

django-ckeditor

Many people are noticing that the CKEditor 4 integration for Django doesn’t work that well when using the dark color scheme of the Django admin panel. That’s not surprising. What does surprise me is the number of reports and the total absence of pull requests. Subjectively, most other packages I help maintain have a comfortable ratio of issues and pull requests.

I’m not complaining and people aren’t complaining, almost everyone is nice and tries to be helpful. Since I’m not a heavy user of django-ckeditor anymore I don’t really find the motivation to fix this issue myself since it’s not all that enjoyable work to me. I would just hope that after all this time someone would finally come and do the necessary work – if they cared enough.

django-mptt

I have marked django-mptt as unmaintained two years ago. I’m still looking at pull requests from time to time but without feeling an obligation to do so and without feeling bad when I miss something.

I wonder if there are still many people out there who still depend on this library and if any of them would be willing to pick up the maintenance? On the off chance that someone is out there who has the time, ability and motivation and just didn’t know that django-mptt could use some love: Here’s your invite!

The insides of my static site generator

2023-04-19T12:00:00Z

The insides of my static site generator

Last sunday I wrote that I’m now using a hacky ~200 LOC Python script to generate this blog. The ~200 LOC became a challenge to myself immediately and I started refactoring the code while adding additional features, adding a licensing comment at the top and further reducing the lines of code in there.

I don’t intend to stop working on it, but I’m really happy with the result as it is currently. The script is less than 190 lines long and supports:

Generating individual HTML files for the front page, the category pages and posts
Generating robots.txt and sitemap.xml
Generating Atom feeds for all posts and posts of each category
Minifying HTML and CSS using minify-html and rcssmin; the CSS is outputted as a single file and includes the content hash in the filename for better cacheability
Keeping the link structure of the old Django-based website

I used Django’s feedgenerator.py module at first to generate the Atom feed; I have since switched to directly working with the ElementTree API. Yes, it’s probably less efficient since it has to keep the whole XML tree in memory but who cares when the largest file’s file size is under 100 KiB at the time of writing.

I’m using tox to generate the site locally; the local build includes published and draft posts. The production build uses GitHub actions and automatically deploys to GitHub pages, while only including published posts. There is no incremental build right now but rebuilding the whole site using tox (with an initialized virtualenv) takes less than one second, so that’s not really a pain point for me right now.

A difficulty was that I have used URLs ending in slashes in the past, not just for the browsable pages but also for the Atom feeds themselves. nginx only serves index.html in folders by default so I couldn’t just add a index.xml file in those folders. Luckily enough the internet is made of lots and lots of duct tape and saving the atom feed as .../feed/index.html actually works. It seems that RSS readers, aggregators and some libraries such as feedparser do not really need the correct HTTP headers.

I have licensed the script under the WTFPL, so if you’re interested you can do what you want with it, without any obligations. I would certainly enjoy hearing about it though!

Weeknotes (2023 week 15)

2023-04-16T12:00:00Z

Weeknotes (2023 week 15)

Romansh translations for feincms3-cookiecontrol and django-fineforms

The feincms3 cookie control banner and django-fineforms have received a small update: Support for the Romansh language. I would be surprised if any readers of this blog even knew about this language at all. Switzerland has four official languages: German, French, Italian and the mentioned Romansh.

It’s not a coincidence of course that those two packages have received an update at the same time. Both packages are used for an upcoming campaign site where people may express their support for political action to preserve or protect the biodiversity. It’s at the same time laughable and horrifying that this is even a thing though: Who in their right mind could NOT agree that preserving biodiversity is important? It’s incomprehensible. Maybe I’m just a romantic Gutmensch after all…

django-ckeditor

I have set up the stale GitHub action for the django-ckeditor repository. So many support requests, so little time and almost no actual collaborators. Also, many support requests actually concern CKEditor itself, not its Django integration. I shouldn’t complain though, CKEditor has served me well and still does, especially when it’s being used with a very minimal configuration which basically makes most pain points of HTML editors go away.

django-debug-toolbar

I have reviewed and merged a few changes to django-debug-toolbar in the last week. Still a fun project, especially since it’s so widely used and loved.

Static site generation

2023-04-15T12:00:00Z

Static site generation

I did what I threatened (myself) to do: I replaced the Django code base for this weblog with a static site generator.

My main goal was to preserve as much as possible of the existing structure, including the Atom feeds and the IDs of posts so that the rewrite wouldn’t flood any aggregators.

The end result is a hacky ~200 LOC Python script which uses Markdown, Jinja2 and minify-html. Markdown is great for blogging and I have been using it for a long time, basically since I started this website in 2005. I don’t like it that much for documentation, but that’s a story for another day.

For now I still deploy the blog to a VPS but there’s nothing stopping me from uploading it somewhere else. I’m thinking about using GitHub actions for the deployment, but I can do that another day.

Run less code in production or you’ll end up paying the price later

2023-04-07T12:00:00Z

Run less code in production or you’ll end up paying the price later

At work we do have the problem of dependencies which aren’t maintained anymore. That’s actually a great problem to have because it means that the app or website ended up running for a long time, maybe longer than expected initially. I think that websites have a lifetime of 3-5 years¹ which is already much longer than the lifetime of major versions of some rapid development frameworks, especially frontend frameworks. We have been using Zurb Foundation in the past. It has served us well and I don’t want to dunk on it – after all it is free, it has great documentation and it works well. It has many features and many components, but as changes happen, not just in the framework itself but also in the tooling it uses upgrades stay hard and things start to break somewhere down the road (for example when Dart Sass 2.0 will be released. And when that happens you have to pick up the parts and maintain them yourself – having grown your codebase by a considerable amount practically overnight.

Of course it’s also hard to argue for starting to write all HTML, CSS and JavaScript from scratch. Standards are very helpful. Building without some sort of standardized tooling will not only lead to rank growth but also means that you have to make many many small and (relatively) irrelevant decisions and you have to justify those decisions when working in a team because you could just as well have decided differently.

So after all that, should you use frontend frameworks? The answer is – as always – it depends.

You should definitely use a framework when prototyping.

But already when working on a MVP it gets less and less clear. You’re kidding yourself if you think that sometime in the future you’ll have the time to clean up your code base; of course you’re always refactoring and of course you’ll try to always leave each part behind a little bit nicer than it was before, but even then the big rewrite didn’t happen and you’ll continue using at least some aspects of the code that was hastily written for the initial release.

The exceptions to the rule are frameworks and tools which have a proven track record of maintaining stability over time. So, despite the fact that the Django framework has more than 100‘000 lines of code I feel good about using it. Django was released as Open Source in July 2005 and has been steadily maintained over the last nearly 18 years. Funding is always a problem (contribute if you can!) but at least the Django software foundation manages to employ two Django fellows which are certainly key in driving the framework forward.

Again the picture gets less clear when third party apps are involved. Some third party apps were actively maintained several years ago and now they aren’t. I have personally taken over the maintenance of several such apps or more often than not reimplemented them with less functionality and much less code. Sometimes it’s easier to maintain a little more of your own code instead of much more of someone else’s code.

So, what’s the take away?

Maybe pay attention to the following points:

Recent activity on the repository, recent releases and good docs are always a good sign for the health of a project.
Take the time to read at least a part of the code of third party projects before starting to rely on them. It’s a bad idea to just read the documentation and nothing else. When it comes to Django apps I always at least read the model files to understand how the data’s modeled and skim the views and forms.
Have a look at issues and pull/merge requests. Don’t worry about usage questions etc but mainly about “hard problem” issues.
Try to stay away from packages which are too comprehensive; reusing these packages is hard even for their authors and when they are abandoned the amount of responsibility transferred to you will be so much bigger.

Citation required obviously, don’t take my word on it. Data is certainly more long lived, but I have my doubts regarding the code running many small sites. ↩

Weeknotes (2023 week 13 and 14)

2023-04-05T12:00:00Z

Weeknotes (2023 week 13 and 14)

My son will be a teenager soon

My eldest is now 12 years old and will be a teenager soon. We had a good time and two nice Birthday parties, one with his friends and one with family and our friends. Good times.

django-debug-toolbar 4.0

Django 4.2 was released, Hatch gained support for the Django 4.2 Trove classifier and we released django-debug-toolbar 4.0, with support for Django 4.2, psycopg 3 and all the existing goodies.

feincms3-cookiecontrol

feincms3-cookiecontrol has gained support for consciously embedding stuff via oEmbed. It can now use Noembed (via feincms3’s external plugin) and only actually embed the third party content if users consented explicitly.

I have since learned through the Datenschutz-Plaudereien podcast that laws regarding consent are not that strict in Switzerland compared to the European Union, also not when the DSG is put into effect in September. What’s right and what’s legal are two different things and while I don’t really like the ubiquitous cookie banners (especially not when they aren’t actually doing anything) I like the idea of explicit consent and of not sending data unnecessarily to third party providers. The additional click isn’t that bad.

Diving into hatch for Python packaging

I listened to the TalkPython podcast episode with Ofek Lev on his Hatch packaging tool. After a long period of uncertainty and waiting I bit the bullet and started to migrate a few of my Python packages from setuptools and setup.py to hatch and pyproject.toml, until now feincms3-cookiecontrol and feincms3. It was surprisingly painless.

ruff

I started learning Rust during the last Advent of Code; it’s a nice language. ruff is a linter and (more and more) formatter for Python code written in Rust. After years of working with Python and Python-based tools it’s surprisingly fast, almost worryingly so. It’s true what they say: ruff finishes so fast that I’m always left wondering if it even did anything at all.

I’m configuring ruff through pyproject.toml, so switching from setuptools to hatch (see above) also helped in this regard. The main trouble I had was that I’m running Python 3.11 locally but Python 3.10 in the server environment (no dev-prod parity…), and rebuilding requirements.txt locally of course didn’t add TOML support because it’s built into Python 3.11, but Python 3.10 needs an external package. So of course I broke the build. That’s not all bad though: If stuff broke it definitely helps with remembering the reasons later.

Matthias Kestenholz: Posts about Django

Keep content managers' admin access up-to-date with role-based permissions

Keep content managers’ Django admin access up-to-date with role-based permissions

The basic ideas of my role-based permissions implementation

Using roles in your own project

My reaction to the block-driven CMS blog post

My reaction to the block-driven CMS blog post

Weeknotes (2023 week 33)

Weeknotes

py_modules using hatchling

django-debug-toolbar and tracing the cause of DB queries in an async world

Releases

Composition over inheritance: The case for function-based views

Composition over inheritance: The case for function-based views

The early days

The introduction of class-based views

Generic views could be simple

ListView and DetailView

Detail view with additional behavior

Form views

Date-based generic views

Wrapping up

Closing words

Weeknotes (2023 week 30)

Weeknotes

Async Django

Releases

How ruff changed my Python programming habits

How ruff changed my Python programming habits

Rules

pyflakes, pycodestyle

mccabe

isort

pep8-naming

pyupgrade

flake-2020

flake8-boolean-trap

flake8-bugbear

flake8-comprehensions

flake8-django

flake8-pie

flake8-simplify

flake8-gettext

pygrep-hooks

pylint

RUF100

Line length

Rules I don’t like

Final words (for now)

Weeknotes (2023 week 29)

Weeknotes

Releases

Serving ZIP files using Django

Serving ZIP files using Django

Weeknotes (2023 week 28)

Weeknotes (2023 week 28)

Releases

pipx inject

hatchling and data files

Payment providers that must not be named

Weeknotes (2023 week 26)

Weeknotes (2023 week 26)

Releases

GitHub projects

To cloud or not

Scheduled publishing

I like programming more than writing (even though I like writing)

FeinCMS is a dead end (but feincms3 is not)

FeinCMS is a dead end (but feincms3 is not)

Weeknotes (2023 week 24)

Weeknotes (2023 week 24)

django-debug-toolbar 4.1

Going all in on hatch and hatchling

CKEditor 5’s new license and django-ckeditor

Mountain biking.

CSS variables and immutability

Using CSS variables1 to ship customizable CSS in Django apps

Patterns for overrideable values

Less repetition (but trouble awaits)

A better way

`py_modules` using hatchling

Using CSS variables¹ to ship customizable CSS in Django apps