Matthias Kestenholz: Posts about Django

Building forms with the Django admin

2024-04-12T12:00:00Z

Building forms with the Django admin

The title of this post was shamelessly copied from Jeff Triplett’s post on Mastodon.

Why?

Many websites need a simple way of embedding forms, for example as a contact form or for simple surveys to collect some data or inputs from visitors. Django’s forms library makes building such forms straightforward but changing those forms requires programming skills and programmer time. Both of them may not be readily available. More importantly, sometimes it’s just nice to give more tools to web publishers.

The simple way to build something like this is to use a form builder such as Google Forms, Typeform, Paperform or anything of the sort. Those options work nicely. The downsides are that embedded forms using those services load slowly, look differently, cost a lot or collect a lot of data on users, or all of those options. Because of that there’s still a place for building such functionality locally.

If I wanted to use PHP and WordPress I could just use WPForms and call it a day. Since I do not actually want that this blog post is a bit longer.

The early days: form-designer

One of the first Django-based third party apps I published was the form-designer. The first version was uploaded to PyPI in 2012 but it had already been used in production for more than two years at that point in time. I had used Git submodules for the deployment back then, before switching to Python virtualenvs some time later (and never looking back!)

The form-designer is still maintained actively. Because of Django’s stability and because of the fact that the app doesn’t do all that much it doesn’t require much development at all.

The form designer supports a selection of standard HTML5 input fields out of the box and also has an optional django-recaptcha integration. All fields support some basic configuration such as setting a title, a help text, marking the field as required etc. Submissions can be sent to a configurable email address and can be saved in the database and later exported as an XLSX file. It’s also possible to define your own actions.

More flexibility needed: feincms3-forms

A few years back I mentioned feincms3-forms in a weeknotes entry. The reasons why form-designer wasn’t sufficient for a project back then are outlined in the blog post:

feincms3-forms – A new forms builder for the Django admin interface

For a current project we needed a forms builder with the following constraints:

Simple fields (text, email, checkboxes, dropdowns etc.)

Custom validation and processing logic

It should be possible to add other content, e.g. headings and explanations between form fields

The form_designer fulfilled a few of these requirements but not all. It still works well but I wanted a forms builder based on django-content-editor for a long time already. Also, I really like the feincms3 pattern where the third party app only provides abstract models. Yeah, it is much more work to start with but the flexibility and configurability is worth it – especially since it’s possible to write straightforward code to handle special cases[^2] instead of configuring even more settings.

The humble beginnings are here in the feincms3-forms repository. The test suite already shows how things work together but as of now no documentation exists and no release has been made yet. I hope it will be ready for a first beta release in the next few weeks 😄

Since then I have used feincms3-forms more often than form-designer, for building simple forms and also to build multi-step form wizards with custom fields, custom validation, configurable steps etc. The README now actually explains why the project exists and how it could be used.

It still doesn’t come close to WPForms in terms of included functionality; a big feature which is missing is conditional logic because I haven’t yet had a use for it.

The feincms3-forms forms support all types of content between form fields (basically everything django-content-editor supports). Plugins for form fields are more flexible and can add as many input fields to the form as they want, you’re not restricted to single values or single input fields.

Packages

Weeknotes (2024 week 14)

2024-04-06T12:00:00Z

Weeknotes (2024 week 14)

I’m having a bit of a slow week with the easter weekend and a wisdom tooth extraction. I’m recovering quite quickly it seems and I’m glad about it.

This weeknotes entry is short and quick. I’m trying to get back into the habit of writing them after a mediocre start this year.

20th Anniversary Celebration of Young Greens Switzerland

I have attended the celebration of Young Greens Switzerland. I have been a founding member and have been active for close to ten years. A lot of time has passed since then. It has been great to reminisce about old times with friends and, more importantly, to see how the torch is carried on.

Releases

blacknoise 0.0.5: blacknoise is an ASGI app for static file serving inspired by whitenoise. It only supports a very limited subset of whitenoise’s functionality, but it supports async.
html-sanitizer 2.4.1: The lxml library moved the HTML cleaner into its own package, lxml-html-clean; this release adds support for that. I didn’t know that the HTML cleaner is viewed as being problematic by the lxml maintainers. I’m having another look at nh3 and will maybe switch html-sanitizer’s guts from lxml to nh3 in the future.
django-tree-queries 0.18: django-tree-queries now supports ordering siblings by multiple fields and even allows descending orderings.
django-cabinet 0.14.2: This release fixes the CKEditor 4 filebrowser popup when using Django 5 or better.

The Django admin is a CMS

2024-03-27T12:00:00Z

The Django admin is a CMS

The post Why is the Django Admin “Ugly”? and the discussion on Mastodon around it finally motivated me to write down my thoughts regarding the recurring theme in Django land that the Django administration interface isn’t a CMS (Content Management System).

I think that this is misguided and needlessly limits the discourse around what the admin’s current functionality is and the ideas what it could be and already is.

A web content management system is about website authoring for users who do not need to be web programming experts in their own rights. Django was created at the Lawrence Journal-World newspaper. The admin itself was created to allow quickly spinning up new websites, where the admin interface was used by content managers to fill in the content while programmers finalized the rest of the website. So obviously the admin interface was a system used to manage content¹ from the beginning.

Sure, the Django admin site documentation states:

One of the most powerful parts of Django is the automatic admin interface. It reads metadata from your models to provide a quick, model-centric interface where trusted users can manage content on your site. The admin’s recommended use is limited to an organization’s internal management tool. It’s not intended for building your entire front end around. [emphasis added]

In other words, the Django documentation also points out that the admin is powerful and that it allows trusted users to manage content¹.

Yes, it will be very painful if you try to do everything on top of the Django admin site. The warnings against using the Django admin for more than it was designed to are necessary and I totally support them. As soon as you’re getting into workflows, into complex permission scenarios (sad noises) or similar things the admin definitely isn’t for you. But, the admin nicely solves 90% of the problems with 10% of the effort. And it’s very good at that.

And sure, if you try building your own frontend on top of the Django admin you’re in for a bumpy ride, but that much should be obvious.

Many third party apps for Django actually target the Django admin interface itself, and not one of the (excellent!) Django-based CMS such as Wagtail. This means that by building on the Django admin instead of one of the CMS you’re running less code, by using more libraries instead of frameworks (on top of frameworks) you’re keeping maintenance lower, and you’re a part of a larger community², which brings the potential benefit of being able to profit more from the general Django packages ecosystem.

Since you’re depending on smaller pieces of additional software it will generally be possible to upgrade to new Django versions quicker. This isn’t true for all packages of course, and I’m a reluctant maintainer of some of them. Anecdotes aren’t data, but I see that some larger CMS systems are definitely having a hard time keeping up with Django’s release schedule.

I’m not trying to say that the Django admin is a better CMS than other Django-based CMS, or any other CMS. I’m saying it’s a trade off and you should be mindful of the downsides of choosing a larger system. And I’m saying that the people who tell you that you shouldn’t be using the Django admin interface are wrong in the first approximation.

The fact that it’s so easy to spin up an additional site and with minimal effort and still be able to work with clean database schemas and all the great tools Django (and Python) offers is important for those of us who are working on many different projects with limited financial resources, because the website often is for example just a small part of a campaign.

Sorry-not-sorry for my choice of words. ↩↩
The assumption that the communities of these Django-based CMS projects are a subset of the Django community itself shouldn’t be too controversial. ↩

blacknoise – ASGI app for static file serving

2024-03-20T12:00:00Z

blacknoise – ASGI app for static file serving

Note

This blog post consists of the blacknoise README at the time of publishing.

blacknoise is an ASGI app for static file serving inspired by whitenoise and following the principles of low maintenance software.

This is pre-alpha software and everything is subject to change. I’m not even sure if blacknoise should exist at all or if the energy wouldn’t be better spent improving whitenoise or other tools. Feedback and contributions are very welcome though!

Using blacknoise with Django to serve static files

Install blacknoise into your Python environment:

pip install blacknoise

Wrap your ASGI application with the BlackNoise app:

from blacknoise import BlackNoise
from django.core.asgi import get_asgi_application
from pathlib import Path

BASE_DIR = Path(__file__).parent

application = BlackNoise(get_asgi_application())
application.add(BASE_DIR / "static", "/static")

BlackNoise will automatically handle all paths below the prefixes added, and either return the files or return 404 errors if files do not exist. The files are added on server startup, which also means that BlackNoise only knows about files which existed at that particular point in time.

Improving performance

BlackNoise has worse performance than when using an optimized webserver such as nginx and others. Sometimes it doesn’t matter much if the app is behind a caching reverse proxy or behind a content delivery network anyway. To further support this use case BlackNoise can be configured to serve media files with far-future expiry headers and has support for serving compressed assets.

Compressing is possible by running:

python -m blacknoise.compress static/

BlackNoise will try compress non-binary files using gzip or brotli (if the Brotli library is available), and will serve the compressed version if the compression actually results in (significantly) smaller files and if the client also supports it.

Far-future expiry headers can be enabled by passing the immutable_file_test callable to the BlackNoise constructor:

def immutable_file_test(path):
    return True  # Enable far-future expiry headers for all files

application = BlackNoise(
    get_asgi_application(),
    immutable_file_test=immutable_file_test,
)

Maybe you want to add some other logic, for example check if the path contains a hash based upon the contents of the static file. Such hashes can be added by Django’s ManifestStaticFilesStorage or by appropriately configuring bundlers such as webpack and others.

License

blacknoise is distributed under the terms of the MIT license.

Podcasts I like listening to

2024-03-18T12:00:00Z

Podcasts I like listening to

I discovered listening to podcasts about one year ago. Previously, I never knew why anyone would want to listen to people talk when they could listen to music or nothing, but that has changed a bit.

So, here’s a list of podcasts I’m currently listening to on a regular basis.

Tech Won’t Save Us

I have recently stumbled over Tech Won’t Save Us, a Podcast which is critical of the technological “progress” offered by Silicon Valley elites. It’s a great antidote for the generative AI hype.

I still have some stupid hope that AGI will solve our problems because maybe people will trust a computer more than scientists that something has to be done about the combined crisis we’re facing as humans. Who knows. I do not want to be too negative about it though, there are positive news if you’re looking in the right places. I myself do not write about those bigger issues as much as I used to. I support those more important issues elsewhere.

Generative AI is certainly not helping me in my work. I do not want a computer to generate code which I have to review and maintain when I still enjoy writing code myself. Maybe that will change at some point in the future. When that happens I’ll probably retire and become a gardener or something.

The Ezra Klein Show

This show seems to be a favorite of many people. I joined for the (relatively) critical perspectives on AI and stayed for the insights into politics and into the near east conflict. I think it’s great that people from different sides get a voice on the show, even if Ezra doesn’t agree with them.

I sometimes wish the questions they ask on the show were a little bit more critical.

Hard Fork

I like the perspectives and the bantering on this podcast. It’s a good way for me to stay informed about what’s going on in AI/ML land, among other things. It’s a lighthearted listen I often look forward to.

Django Chat

I work with Django all the time and Django Chat is a great way to learn more about Django, about the people involved and also about the surrounding ecosystem. A wholehearted recommendation!

Talk Python To Me

Good interviews, interesting guests and topics. Just a great way to learn about libraries, tools etc. which I cannot wait to use in my own projects.

Datenschutzplaudereien

Required listening for people working with customer data in Switzerland. Swiss german knowledge required.

Others

A few other podcasts I’m listening to more or less regularly:

Wir müssen reden. Public Eye spricht Klartext: The podcast from public eye, an organization which uncovers human right violations perpetrated by Swiss companies around the globe. I’m not yet sure if I like the podcast, but I like and support the organization very much.
Hanselminutes: The more polifor the wide and diverse array of guests.tical podcasts have displaced Hanselminutes in my listening schedule, but I’d definitely recommend it to anyone who wants to learn more about tech. Recommended because of the diverse array of guests, among many other reasons.
Syntax: At times very interesting, at times not much new for me. I still recommend it.
Weird Studies: Really weird. I liked the discussion of Hellraiser and The Thing, and have listened to some of the other episodes. Religiosity and spiritualism was always around me when I was a child, and it’s interesting to revisit some of the ideas about reality from a different angle after years and years of not engaging with these questions at all. (Writing this down does sound a bit like a midlife crisis on the horizon, but I don’t think that’s it. I have studied a little bit of philosophy and ethics and have stayed interested in these topics ever since.)
I don’t think the german “Welt” is a good newspaper at all. I do like the «Aha! History» and «Aha! Zehn Minuten Alltagswissen» podcasts because they are short and often examine topics I don’t know much about.
Sternstude Philosophie: It’s a bit slow and definitely more geared towards viewers than listeners, but definitely interesting.

Weeknotes (2024 week 11)

2024-03-16T12:00:00Z

Weeknotes (2024 week 11)

Estimates

Jacob wrote an excellent post on breaking down tasks. I did like the post a lot. Maybe I’ll write a longer reply later, but for now just this. There definitely are good reasons for the pushback against estimation, and it’s really not just that some people lack professionalism.

Releases

django-cabinet 0.14.1: Mini release containing a Turkish translation. It’s always nice if software is used.
feincms3 4.6: Fixed a bug where the move form wouldn’t use a potentially overridden ModelAdmin.get_queryset method.
form-designer 0.24: Updated the package for django-recaptcha 4.0.
html-sanitizer 2.3.1: Fixed an edge case sanitization bug (luckily without security implications).
django-content-editor 6.4.2: django-content-editor now again supports transitioning plugin fieldsets when opening and closing thanks to CSS grid’s ability to animate the maximum height of an element. Also, the initialization in 6.4 was badly broken.
django-prose-editor 0.2: See the announcement blog post from Wednesday.

django-prose-editor – Prose-editing component for the Django admin

2024-03-13T12:00:00Z

django-prose-editor – Prose-editing component for the Django admin

During the last few days I have been working on a prose-editing component for the Django admin which will replace the basically dead django-ckeditor in all of my projects. It is based on ProseMirror, in my opinion the greatest toolkit for building prose editors for the web.

Here’s a screenshot:

django-prose-editor is in active development; it’s available on PyPI and is developed in the open here on GitHub. The version at the time of writing is 0.2, and it’s not yet used in production environments, only in staging/preview environments. That will soon change though.

Researching alternatives to django-ckeditor

I have spent a lot of time evaluating alternatives. All of them are great choices, and I don’t want to bash any of them. But, I didn’t feel good betting on any of the choices from the WYSIWYG editors grid on Django Packages.

A few packages have potential licensing issues. CKEditor 5’s change to the GPL is what basically killed django-ckeditor. The upcoming TinyMCE 7 version will also change to the GPL according to the TinyMCE GitHub repository. Froala only has a free trial. The django-summernote app doesn’t have a dedicated maintainer, so I wouldn’t bet on it. django-prose uses Trix; there are various reasons why I didn’t want to bet on Trix, among them my personal experience that it sometimes gets into a buggy state where only a full page reload unfreezes the editor.

Other packages are basically Markdown editors. I like Markdown and use it for my blog. I don’t think Markdown is a good choice for a CMS which is used by people of many different skill levels. I don’t want to teach people to use the Markdown link syntax or the heading syntax even if the editor helps out a bit.

Some of the remaining choices are using JavaScript widgets which haven’t been updated for a really long time or they are really code editors, not WYSIWYG editors, also a no go.

Deciding on going with ProseMirror

I have worked with ProseMirror on and off since October 2015, soon after the crowdfunding ended. It is used in a project where people can write their own book with a standardized pipeline and process, where the technical side of the project is implemented using Django, ProseMirror and LaTeX. A hacked prosemirror-example-setup still is good enough for this project.

The ProseMirror deep dive came much later, only a few years back, when implementing an editor with more custom functionality such as annotations, different ways of marking up text and even interactive elements within the text, for example to use it as a cloze in teaching materials.

The learning curve is steep. I haven’t worked with another library which was so hard to get started with. It is my conviction that the reason for this is that rich-text editing is actually a hard problem. The ProseMirror architecture and implementation definitely makes sense when it finally clicks.

As additional bonuses the ProseMirror community is nice and ProseMirror is used in a few large software projects which make me believe that the software has a non-zero probability of being maintained in the long run.

Current plans

The only thing which is configurable right now is the server-side sanitizing of submitted HTML content. I plan on allowing some configurability, for example to disable links when the content will only be used inside a card teaser which itself is already a link. Too much configurability isn’t a good thing though in my mind, so I’ll probably be slow to add features and rather keep complexity low.

I’m definitely interested to hear from people who want to use the package but cannot do so right now or who would want some additional features. Issues and pull requests are welcome!

Weeknotes (2024 week 07)

2024-02-16T12:00:00Z

Weeknotes (2024 week 07)

This is a short weeknotes entry which mainly contains a large list of releases. The reason for the large list is that I haven’t published a weeknotes entry in weeks.

Releases

form-designer 0.23: Only small changes, mainly updated the package for current Django and Python versions.
feincms3-cookiecontrol 1.4.6: A minor change: Swallow exceptions which happen during startup when clobbering the scripts data fails. As an aside: I find it funny that I have discovered the .f3cc class in some cookie banner blocklists. It feels good to be recognized even if this maybe isn’t the nicest way, but it works for me since I actually do not like cookie banners either. At least feincms3-cookiecontrol doesn’t inject anything without users’ consent, and doesn’t require a third party service to run.
django-simple-redirects 2.2.0: Minor release which adds a search field to the admin changelist. django-simple-redirects is a repackaged version of django.contrib.redirects without the django.contrib.sites dependency.
speckenv 6.2: django_cache_url now supports parsing redis configuration for a leader-replica redis installation with a read-write leader host and read-only replica hosts. I use the same configuration format as django-cache-url does.
django-debug-toolbar 4.3: I haven’t done much here, just some reviewing here and there. I enjoy the Djangonaut Space contributions a lot.
django-cabinet 0.14: I have removed the constraint which enforces unique names for subfolders. Enforcing the uniqueness does make sense, but it also makes bulk-updating the media library using serialized data more painful than it should be. It’s a clear case of worse is better for me. If people want to confuse themselves I’m not going to stop them (anymore, in this case) but it makes the rest of the code so much easier to write that it’s not even funny.
html-sanitizer 2.3: This release contains a nice contribution which removes some whitespace which has been added by the sanitizer when merging adjacent tags of the same type, e.g. <strong>abc</strong><strong>def</strong>.
django-ckeditor 6.7.1: See above.
django-json-schema-editor 0.0.11: Fixed a crash which happened when not providing the optional (!) configuration. Shit happens. I should really have a test suite for this package.
feincms3 4.5.2: Disables the CKEditor version check.
django-content-editor 6.4: The first release since December 2022! Very stable software. The editor now restores the collapsed state of inlines and the scroll position when using “Save and continue editing”. This is especially useful if editing an object with many content blocks.

django-ckeditor

2024-02-14T12:00:00Z

django-ckeditor

It has finally happened. The open source version of CKEditor 4 does not contain fixes for known problems, see the CKEditor 4.24.0 LTS announcement.

I totally get why the CKEditor developers did this and can only thank them for all the work that went into the editor.

I wish I didn’t have to do the migration work to move basically everything to a different editor. The CKEditor 4 LTS version is only expected to be supported until the end of 2026 and I have a few projects which will be around far longer than this (or at least I hope so). Therefore, buying the LTS package would only delay the inevitable. CKEditor 5 is a completely different editor and uses the GPL license, so that’s not really an option either. TinyMCE is well known and I have been using it much earlier in my career, but reimplementing plugins isn’t fun to do.

I would prefer moving everything to ProseMirror or some other structured editor, but we have so much legacy content contained in HTML blobs which do not use any schema at all that this isn’t workable unfortunately.

Stay tuned for updates – they will come since I unfortunately cannot just ignore this problem.

Which brings me to django-ckeditor. CKEditor shows a very annoying popup to users when it detects a newer version with security fixes, see the GitHub issue. It’s not a bad idea but users cannot do much about it, so I opted to disable the version check (and warning) in django-ckeditor and replaced it with a Django system check which annoys developers instead.

The release containing these changes is available as django-ckeditor 6.7.1 on PyPI.

Weeknotes (2024 week 03)

2024-01-17T12:00:00Z

Weeknotes (2024 week 03)

Djangonaut Space

I wish all participants a good time and much success. I do not have anything to do with it really but I enjoy the idea a lot and maybe there will be a pull request or two to review.

Kubernetes

After years and years of hosting all sites on VPS I have finally reached the point where the old setup is more annoying to work with than switching to a new one. I have searched long for a solution which wasn’t as limited as some PaaS and as complex as going full Kubernetes, and where I can still delegate the responsibility of actually keeping things up and running to other people. In the end I have now accepted that such a thing doesn’t exist; either you have the limitations of a ready made solution, the limitation of having to open many many support tickets or the problem of having to learn Kubernetes (or something similar) with its extremely steep learning curve.

After spending days with it I’m slowly getting to the point where setting up local development environments and deploying changes is fun again. I’m using the GitOps paradigm; while I’m still building and uploading Docker (podman) images from the local development environment everything else is automated and goes through a Git based process. That’s much nicer than clicking around in some interface or copy pasting obscure commands into the console.

The biggest problem I encountered was (perhaps unsurprisingly) managing secrets as a team. It seems to me that while sealed secrets work great as an individual developer they don’t really offer straightforward solutions to avoid different people overwriting and resetting each others secrets when updating them. I’m a happy user of the external secrets operator and using some cloud service to actually store those secrets.

I have started using granian in production. I like the idea of a Rust-based ASGI/WSGI server. Nothing mission critical yet. My idea is to build confidence in the software stack.

Compulsory social measures

The more I learn about how Switzerland treats its citizens the more I wonder about the ways in which humans can mistreat other humans in a so called civilized and peaceful society.

It’s not exactly a new topic for me, but working on platforms which help remember and which help introducing people to the history certainly causes a heightened awareness for issues such as these.

Releases

feincms3-sites 0.20.2: It previously wasn’t possible to filter the list of sites to only show those sites which do not have a default language. This has been fixed.
django-json-schema-editor 0.0.10: It’s now actually possible to use the JSON editor outside inlines! That was a fun bug… not. Apart from the mentioned bug this new release mostly contains fixes to the styles. We’re slowly getting there.
django-mptt 0.16: I didn’t do anything except for the changelog and the release. That’s alright. The release contains a few minor fixes.

Weeknotes (2024 week 01)

2024-01-03T12:00:00Z

Weeknotes (2024 week 01)

First weeknotes post for 2024! Happy new year!

Looking back on 2023

Writing

I have published almost 40 posts last year. That’s almost as many posts as I published in the time period from 2014 to 2023. Coworking to write more does work.

I already had a quite active blog from 2005 to 2008 with a few posts after that; everything before 2014 was in german and mainly concerned with green politics and climate change. I’m still very interested in these topics but I don’t feel as if I have much to add to the conversation, even though it’s the more important issue.

Open Source

Not much changed here. I enjoy basically everything I do in open source land, and co-maintaining the Django Debug Toolbar with Tim is a joy.

I still wish that some of my projects had more impact in Django land, especially those who augment the Django administration interface to be a lightweight CMS which requires very little maintenance and work in the long run. I think it’s great that one of the core components, django-content-editor, hasn’t required a release in more than one year. It doesn’t have to be expanded because it just works. It would be great if there was a good way to distinguish between software which basically doesn’t require any updates and software which is abandoned.

The only project which doesn’t bring me much joy is django-mptt. Most people accept that there are no guarantees, but some people are just rude in the way in which they expect others to do the work. The first reaction was to return the blow and I’m glad that I didn’t give in to the temptation to do that. It’s basically never worth it to do that in writing.

Family

We had a good 2023 together and I’m very much looking forward to a just as good 2024.

Plans for 2024

I have bought a ticket for the DjangoCon Europe 2024 in Spain and I’m very much looking forward to that.

Maybe we’ll visit a music festival again this summer. After listening to a lot of metal music in the last ten or more years I rediscovered the dark side of D’n’B. Good times.

Advent of Code

I have finished the Advent of Code with some help from the Subreddit. Almost all of the puzzles were fun to think about. I didn’t have fun solving each and everyone of them, especially not the second part of a few of the later days. I feel good checking out solutions from other people in the subreddit, and maybe adding newly gained ideas to my code or even running someone else’s code 1:1 on my data after studying the algorithms used and hopefully learning something.

I had a good time and enjoyed shutting down the computer after that.

Releases

FeinCMS 23.12: A few minor changes to the thumbnailing code. It’s always nice to hear from people who are still using this project.

Weeknotes (2023 week 50)

2023-12-15T12:00:00Z

Weeknotes (2023 week 50)

django-imagefield

The path building scheme used by django-imagefield has proven problematic: It’s too likely that processed images will have the same path.

I have changed the strategy used for generating paths to use more data from the source; it’s now possible (and recommended!) to set IMAGEFIELD_BIN_DEPTH to a value greater than 1; 2 or 3 should be sufficient. The default value is 1 which corresponds to the old default so that the change won’t be backwards incompatible. However, you’ll always get a deprecation warning if you don’t set a bigger value yourself. The default will probably change in the future.

Advent of Code

I have always felt a bit as an imposter because I do not have any formal CS education; not so much in the last few years but certainly earlier in my career. I have enjoyed participating in the Advent of Code 2022 a lot and I have definitely learned to know when to use and how to use a few algorithms I didn’t even know before. I’m again working through the puzzles in my own pace and have managed to solve almost all of them up to today this year. There still are some puzzles where I don’t even know how to start the second part 😅.

Hosting

We’re still hosting most sites on virtualized servers, without any containers or any of the new stuff. I’m finally reaching the point where the downsides of this approach start to drag new projects down and the workarounds start looking worse than maybe switching to containers or even Kubernetes. Wish me luck, I’m more confused than I’ve been in years.

Health

To absolutely nobody’s surprise the family and myself have continued to be sick in the last two weeks. Nothing really bad happened, so we’re still lucky.

There’s unfortunately no way to solve a societal problem individually, so that will probably continue to be our life for now.

Releases

django-imagefield 0.18: See above.
feincms3-sites 0.20.1: Added additional validation (cleaning) checks. Showing error messages is preferrable to crashing with IntegrityError exceptions after all.
django-js-asset 2.2.0: Hatchling seems to dislike it if the project name and the Python module name do not match. I actually like django-js-asset’s Python module to be js_asset but I’m beginning to rethink this decision.
django-json-schema-editor 0.0.4: See the post from this week.

django-json-schema-editor

2023-12-13T12:00:00Z

django-json-schema-editor

I have extracted a JSON editing component based on @json-editor/json-editor from a client’s project and released it as open source. It isn’t the first JSON editing component by far but I like it a lot for the following reasons:

It works really well.
It supports editing arrays of objects using a tabular presentation. Tabular isn’t always better, but stacked definitely isn’t always better as well.
The data structure is defined as JSON schema,the data which is being entered is validated on the server using the fastjsonschema library. Having a schema and schema-based validation fixes most problems I have with less structured data than when using only Django model fields (without JSON).

Here’s a screenshot of the editing component used as a django-content-editor plugin:

Within the first few days of having released the package it has already proven useful in several other projects. A pleasant (but not totally unexpected) surprise.

Weeknotes (2023 week 48)

2023-11-30T12:00:00Z

Weeknotes (2023 week 48)

A few weeks have passed since the last update. The whole family was repeatedly sick with different viruses etc… I hope that the worst is over now. Who knows.

12-factor Django storage configuration

I should maybe write a longer and separate post about this, but speckenv has gained support for the Django STORAGES setting. No documentation yet, but it supports two storage backends for now, the file system storage and django-s3-storage, my go-to library for S3-compatible services.

Using it looks something like this:

from speckenv import env
from speckenv_django import django_storage_url

STORAGES = {
    "default": django_storage_url(
        env(
            "STORAGE_URL",
            default="file:./media/?base_url=/media/",
            warn=True,
        ),
        base_dir=BASE_DIR,
    ),
    "staticfiles": {
        "BACKEND": "django.contrib.staticfiles.storage.ManifestStaticFilesStorage",
    },
}

Then, if you want to use S3 you can put something like this in your .env file:

STORAGE_URL=s3://access-key:secret@bucket.name.s3.eu-central-1.amazonaws.com/media/

Or maybe something like this, if you want to serve media files without authentication:

STORAGE_URL=s3://access-key:secret@bucket.name.s3.eu-central-1.amazonaws.com/media/?aws_s3_public_auth=False&aws_s3_max_age_seconds=31536000

Releases

speckenv 6.1.1: See above.
feincms3-meta 4.6: York has contributed support for emitting structured data records. Looks nice. No documentation yet.
django-tree-queries 0.16.1: .values() and .values_list() queries are now handled better and more consistently than before.

Weeknotes (2023 week 44)

2023-11-02T12:00:00Z

Weeknotes (2023 week 44)

Unmaintained but maintained packages

There’s a discussion going on in the django-mptt issue tracker about the maintenance state of django-mptt. I have marked the project as unmaintained in March 2021 and haven’t regretted this decision at all. I haven’t had to fix inconsistencies in the tree structure once since switching to django-tree-queries. And if that wasn’t enough, I get little but only warm and thankful feedback for the latter, so that’s extra nice.

Despite marking django-mptt as unmaintained I seem to be doing a little bit of maintenance still. I’m still using it in old paid projects and so the things I do to make the package work for me is paid work. I’m not personally invested in the package anymore, so I’m able to tell people that there are absolutely no guarantees about the maintenance, and that feels good.

Read the Docs

I do understand why the .readthedocs.yaml file is now necessary. I wish that I wouldn’t have to do all the busywork of adding one to projects. I have just resubscribed to the Read the Docs Gold Membership which probably has expired at some point in the past. Read the Docs is excellent and everybody who can should support them.

Releases

feincms3 4.5, feincms3-sites 0.20 and feincms3-language-sites 0.3: Fixed the check which only allows adding an application through the CMS to the page tree (yes, that’s right) once; feincms3 worked fine, feincms3-language-sites by accident but feincms3-sites didn’t.
towel 0.31: Towel is one of my oldest packages which is still being used in real-world projects. Towel is a tool for building CRUD-type applications and is designed to keep you DRY while doing that. The project has been heavily inspired by a Django-based agency software I built many years back. The package even has docs! I’m still quite proud of the mostly transparent support for multitenancy, but apart from that I haven’t used it in many new projects.

Customize the Django admin to differentiate environments

2023-10-19T12:00:00Z

Customize the Django admin to differentiate environments

We often have the same website running in different configurations:

Once as a production site.
Once as a place where editors update and preview the content. The content is later automatically (and maybe partially) transferred from this environment to the production environment.
Once as a stage environment to stabilize the code.
And maybe additional environments for local development.

The Django admin panel mainly uses CSS variables for styling since theming support was introduced in Django 3.2 (by yours truly with a lot of help from others). This makes it simple and fun to customize the colors of all interface elements in a straightforward way without having to write loads of CSS.

If you have a ENVIRONMENT context variable available (as we do) you could add the following template as admin/base.html to your project, giving you a red color scheme for the production environment (to discourage people from updating content) and a nice scheme for the preproduction environment which clearly deviates from the standard color scheme used everywhere else:

{% block extrahead %}
  {{ block.super }}
  <style>
#site-name::after {
  content: " ({{ ENVIRONMENT }})";
  font-size: 60%;
}
  </style>
  {% if ENVIRONMENT == 'production' %}
    <style>
:root {
  --primary: #aa0000;
  --secondary: #810000;
  --accent: yellow;
}
    </style>
  {% elif ENVIRONMENT == 'preproduction' %}
    <style>
:root {
  --primary: #30b181;
  --secondary: #1f7957;
  --accent: #cdffea;
}
    </style>
  {% endif %}
{% endblock %}

Weeknotes (2023 week 42)

2023-10-18T12:00:00Z

Weeknotes (2023 week 42)

Vacation in Italy

We have spent a wonderful family week in Italy. The voyage by train was very comfortable and we had a great time there. I have lived close to lakes all my life but the sea is always something else. Now I enjoy the cold temperatures of fall.

Going back (forward) to GitJournal

I have tried several note taking apps but I’m now back using GitJournal with a Git repository filled with Markdown notes. It works well enough. I just wish that there was a way to make notes more distinguishable and I wish that the editor was more forgiving when encountering badly formatted checklists.

Analog blogging

I have long wanted to write about our switch from Slack to Discord. I have started to write this post with pen and paper. I find that I think better when using pen and paper than when using the computer keyboard. One factor is certainly that the computer offers more distractions, but I suspect that another, more important factor is that as a fast typist the fingers and the thinking are always getting out of step, and this happens less when using a slower method of writing. This actually isn’t an idea I had myself, but I don’t remember where I got it from.

Zero-based versioning: Good or bad?

I discovered ZeroVer sometime in the last few days. I have many many Django packages with zero-based versions. Some of them have been used in production for years now. I sometimes wonder if staying with 0. is unprofessional and I should just release 1.0 and be done with it or if it doesn’t really matter at all.

If I evaluate software packages more often than not I don’t look at the version number or the version numbering scheme (except when a package is still using 0.0.) when deciding whether to rely on it or not. The documentation and the code itself are much more important to me.

Releases

I haven’t uploaded any releases in the last 14 days. That’s good: I’m one of those people who have made their passion their job (which is great) but that sometimes makes it hard to not work at all since I can always tell myself that I’m not working, that it’s just a hobby.

Weeknotes (2023 week 40)

2023-10-04T12:00:00Z

Weeknotes (2023 week 40)

More work on hosting several websites from a single Django application server using feincms3-sites

I have mentioned feincms3-sites last week in my last weeknotes entry; I have again given this package a lot of attention in the last days, so another update is in order.

It is now possible to override the list of languages available on each site. That’s especially useful for an upcoming campaign site where the umbrella group’s site is available in three languages, but (most?) individual group sites (hosted on subdomains) will only have a subset of languages. Since I live in a country with four national languages (english isn’t one of them, but is spoken by many!) supporting more than one language, or even many languages is totally commonplace. It’s great that Django has good support for internationalization. For the sake of an example, I have the following sites:

example.com: The default. The host has to match exactly.
subdomain.example.com: One individual group’s site. The host has to match the regex ^subdomain\. (sorry, I actually do like regexes).

Overriding configured hosts for local development

One thing which always annoyed me when using django.contrib.sites was that “just” pulling the database from production to the local development environment always produced links pointing back to the remote host instead of working locally (when producing absolute URLs). This problem was shared by feincms3-sites as well. I have now found a very ugly but perfectly workable solution: Overwrite Site.get_host() locally:

if DEBUG:
    domain = "example.com"  # Or whatever
    _get_host = lambda site: site.host.replace(domain, "localhost:8000")
    FEINCMS3_SITES_SITE_GET_HOST = _get_host

This works especially well when using example.com and maybe subdomains of example.com: All absolute links will point to localhost:8000 or subdomain.localhost:8000. Since *.localhost always resolves to the local IP the browser knows where it should connect to, and since subdomain.localhost:8000 also matches the ^subdomain\. regex mentioned above, the site selection logic works as well.

Of course if you have more domains, not just subdomains, you could adapt the get_host override and the relevant regexes to those use cases.

Closing words

We’re at 100% code coverage now when running the test suite. That’s really nice.

Logging into the Django admin using your Google account

This functionality has long been provided by django-admin-sso; however, as mentioned a long time ago this package still uses a deprecated OAuth2 library. django-authlib supports using a Google account to authenticate with the Django admin since 2017. I have now fixed a small problem with it: If you are logged into a single Google account, and this account’s email address doesn’t match the configured admin login rule, you were out of luck: There was no way to add another account at that time because the library didn’t request the account selection. That has changed now, if the first login attempt doesn’t work, it now explicitly tells Google to let the user select their Google account. A small quality of life improvement for those using more than one Google account (voluntarily or not).

Releases

feincms3 4.4.3: Polished the CKEditor integration a little bit. Re-enabled the source button now that we’re back to using the classic iframe-based editor again.
feincms3-sites 0.19.3: See above.
django-authlib 0.16.4: See above.

Weeknotes (2023 week 39)

2023-09-28T12:00:00Z

Weeknotes (2023 week 39)

Again a few weeks have passed since the last weeknotes entry :-)

Moving feincms3 repositories into the feincms organization

The feincms GitHub organization has seen more active days when FeinCMS 1.x was still actively developed. Since my interest has moved to feincms3 some years ago I haven’t kept the organization up to date. That has changed this week, and I have moved most feincms3-related repositories into the organization.

This move doesn’t change much though, but it certainly feels more official now.

Adding scheduled tests

I have started using the cronjob schedule feature of GitHub actions to ensure that tests run at least once a month in a few important projects. I want to get notified of changes in Django@main affecting my packages not only when actively working on them. I try to keep up with Django@main in all packages I maintain.

Releases

feincms3-sites 0.18.2: Many releases in the last weeks. Stopped using permanent redirects in DEBUG mode. Avoid migrations when Django adds more languages. Added utilities which allow restricting model relations to objects in the same site (trickier than it sounds). Added utilities for building full URLs to other sites without taxing the database as much.
feincms3-language-sites 0.2.0: No biggie. No permanent redirects in DEBUG mode anymore.
feincms3-cookiecontrol 1.4.5: Reduced the byte size of the CSS and JavaScript some more. Added spanish translations.
django-authlib 0.16.3: I have published a post last week describing the new role-based permissions feature.
django-imagefield 0.17: The process_imagefields management command now allows specifying globs. If you wanted to prerender all imagefields in the pages app you can use ./manage.py process_imagefields pages.* now instead of listing all image fields’ labels explicitly.
feincms3 4.4.1: I’m enormously unhappy but I had to go back to the classic CKEditor instead of using the inline editor. The latter looked much nicer but overriding the Django admin CSS was very very painful. Also, I can totally understand why CKEditor 5 is completely different and why CKEditor 4 is only maintained in a paid LTS plan. It still is making me look for alternatives.
django-mptt 0.15: I unfortunately am still using this despite the fact that I have marked it as officially unmaintained since march 2021. I did a mediocre job of making the library run on Django@main again. Parts of the library do not work, but since I’m not using them I don’t care too much. I’m still wondering if someone wants to take over maintenance of the library since it still seems to be actively used in projects of others as well. When I don’t have to use django-mptt I’m still really happy with django-tree-queries.
form-designer 0.22: This is probably my oldest actively developed project these days. 13 years! (Except for django-content-editor of course.) I have modernized the package, switched to hatchling and put out a new release.

Keep content managers' admin access up-to-date with role-based permissions

2023-09-20T12:00:00Z

Keep content managers’ Django admin access up-to-date with role-based permissions

Django’s built-in permissions system is great if you want fine-grained control of the permissions content managers should have. The allowlist-based approach where users have no permissions by default and must be granted each permission individually makes a lot of sense to me and is easy to understand.

When we build a CMS at Feinheit we often use the Django administration panel as a CMS. Unfortunately, Django doesn’t provide a way to specify that content managers should have all permissions in the pages and articles app (just as an example). Adding all current permissions in a particular app is straightforward when using the filter_horizontal interface but keeping the list up-to-date later isn’t. When we add an additional content block plugin we always have to remember to also update the permissions after deploying the change – and often, deployment happens some time after the code has been written, e.g. because clients want to approve the change first. What happens all too often is that the manual step of updating permissions gets forgotten.

This has annoyed me (intermittently) for a long time and my preferred solution has always been to give superuser permissions to everyone and trust them to not make changes which they aren’t supposed to according to the Trusted Users Editing Structured Content principle which was mentioned in a Django book I read early in my Django journey.

The basic ideas of my role-based permissions implementation

A recent project has resurfaced this annoyance and I did finally bite the bullet and implement a solution for this in the form of a django-authlib extension. The basic ideas are:

All users are assigned a single role: Single roles sound inflexible, but is good enough for my default use case. Examples for roles could be default (no additional permissions granted), content managers (grant access to the pages and articles apps) or maybe deny auth (deny access to users, groups and permissions).

The permission check is implemented using a single callable: A custom backend is provided whose only job is to call the correct callable for the user’s current role.

The callable either returns a boolean or raises PermissionDenied to prevent other backends from granting access: No new ideas here, it’s exactly what Django’s authentication backends are supposed to do.

Permission checkers for the most common scenarios are bundled: django-authlib only ships one permission checker right now, allow_deny_globs, which allows specifying a list of permission name globs to allow and to deny. Deny overrides allow as is probably expected.

Using roles in your own project

Specify the available roles in your settings and add the authentication backend:

from functools import partial
from authlib.roles import allow_deny_globs
from django.utils.translation import gettext_lazy as _

AUTHLIB_ROLES = {
    "default": {"title": _("default")},
    "staff": {
        "title": _("editorial staff"),
        "callback": partial(
            allow_deny_globs,
            allow={
                "pages.*",
                "articles.*",
            },
        ),
    },
}

AUTHENTICATION_BACKENDS = (
    # This is the necessary additional backend
    "authlib.backends.PermissionsBackend",
    # Maybe you want to use authlib's email authentication ...
    "authlib.backends.EmailBackend",
    # ... or the standard username & password combination:
    "django.contrib.auth.backends.ModelBackend",
)

You have to extend your user model (you have to use a custom user model if you’re not using django-authlib’s little_user.User):

from authlib.roles import RoleField

class User(AbstractUser):
    # ...
    role = RoleField()

And that’s basically it.

Of course the globbing is flexible, you could also allow users to view all objects:

partial(allow_deny_globs, allow={"*.view_*"})

Or you could block users from deleting anything:

partial(allow_deny_globs, deny={"*.delete_*"})

And as mentioned above, you can also combine allow and deny (deny wins over allow) or even provide your own callables. If you provide your own callable it must accept user, perm and obj (which may be None) as keyword arguments. Implementing such a callable is probably less work than implementing an authentication backend yourself; I had to do more work than initially expected because only implementing .has_perm isn’t sufficient if you want to see any apps and models in the admin index page. The current allow_deny_globs implementation is nice and short:

def allow_deny_globs(user, perm, obj, allow=(), deny=()):
    for rule in deny:
        if fnmatch(perm, rule):
            raise PermissionDenied
    return any(fnmatch(perm, rule) for rule in allow)

Matthias Kestenholz: Posts about Django

Building forms with the Django admin

Building forms with the Django admin

Why?

The early days: form-designer

More flexibility needed: feincms3-forms

feincms3-forms – A new forms builder for the Django admin interface

Packages

Weeknotes (2024 week 14)

Weeknotes (2024 week 14)

20th Anniversary Celebration of Young Greens Switzerland

Releases

The Django admin is a CMS

The Django admin is a CMS

blacknoise – ASGI app for static file serving

blacknoise – ASGI app for static file serving

Using blacknoise with Django to serve static files

Improving performance

License

Podcasts I like listening to

Podcasts I like listening to

Tech Won’t Save Us

The Ezra Klein Show

Hard Fork

Django Chat

Talk Python To Me

Datenschutzplaudereien

Others

Weeknotes (2024 week 11)

Weeknotes (2024 week 11)

Estimates

Releases

django-prose-editor – Prose-editing component for the Django admin

django-prose-editor – Prose-editing component for the Django admin

Researching alternatives to django-ckeditor

Deciding on going with ProseMirror

Current plans

Weeknotes (2024 week 07)

Weeknotes (2024 week 07)

Releases

django-ckeditor

django-ckeditor

Weeknotes (2024 week 03)

Weeknotes (2024 week 03)

Djangonaut Space

Kubernetes

Compulsory social measures

Releases

Weeknotes (2024 week 01)

Weeknotes (2024 week 01)

Looking back on 2023

Writing

Open Source

Family

Plans for 2024

Advent of Code

Releases

Weeknotes (2023 week 50)

Weeknotes (2023 week 50)

django-imagefield

Advent of Code

Hosting

Health

Releases

django-json-schema-editor

django-json-schema-editor

Links:

Weeknotes (2023 week 48)

Weeknotes (2023 week 48)

12-factor Django storage configuration

Releases

Weeknotes (2023 week 44)

Weeknotes (2023 week 44)

Unmaintained but maintained packages

Read the Docs

Releases

Customize the Django admin to differentiate environments

Customize the Django admin to differentiate environments

Weeknotes (2023 week 42)

Weeknotes (2023 week 42)