Matthias Kestenholz: Posts about Django

Weeknotes (2024 week 47)

2024-11-20T12:00:00Z

Weeknotes (2024 week 47)

I missed a single co-writing session and of course that lead to four weeks of no posts at all to the blog. Oh well.

Debugging

I want to share a few debugging stories from the last weeks.

Pillow 11 and Django’s `get_image_dimensions`

The goal of django-imagefield was to deeply verify that Django and Pillow are able to work with uploaded files; some files can be loaded, their dimensions can be inspected, but problems happen later when Pillow actually tries resizing or filtering files. Because of this django-imagefield does more work when images are added to the system instead of working around it later. (Django doesn’t do this on purpose because doing all this work up-front could be considered a DoS factor.)

In the last weeks I suddenly got recurring errors from saved files again, something which shouldn’t happen, but obviously did.

Django wants to read image dimensions when accessing or saving image files (by the way, always use height_field and width_field, otherwise Django will open and inspect image files even when you’re only loading Django models from the database…!) and it uses a smart and wonderful¹ hack to do this: It reads a few hundred bytes from the image file, instructs Pillow to inspect the file and if an exception happens it reads more bytes and tries again. This process relies on the exact type of exceptions raised internally though, and the release of Pillow 11 changed the types… for some file types only. Fun times.

The issue had already been reported as #33240 and is now tracked as #8530 on the Pillow issue tracker. Let’s see what happens. For now, django-imagefield declares itself to be incompatible with Pillow 11.0.0 so that this error cannot happen.

rspack and lightningcss shuffled CSS properties

rspack 1.0 started reordering CSS properties which of course lead to CSS properties overriding each other in the incorrect order. That was a fun one to debug. I tracked the issue down to the switch from the swc CSS minimizer to lightningcss and submitted a reproduction to the issue tracker. My rust knowledge wasn’t up to the task of attempting to submit a fix myself. Luckily, it has been fixed in the meantime.

rspack problems

I have another problem with rspack where I haven’t yet tracked down the issue. rspack produces a broken bundle starting with 1.0.0-beta.2 when compiling a particular project of mine. I have the suspicion that I have misconfigured some stuff related to import paths and yarn workspaces. I have no idea how anyone could have a complete understanding of these things…

Bundlers are complex beasts, and I’m happy that I mostly can just use them.

Closing thoughts

Debugging is definitely a rewarding activity for me. I like tracking stuff down like this. Unfortunately, problems always tend to crop up when time is scarce already, but what can you do.

Releases

Quite a few releases, many of them verifying Python 3.13 and Django 5.1 support (if it hasn’t been added already in previous releases). The nicest part: If I remember correctly I didn’t have to change anything anywhere, everything just continues to work.

django-admin-ordering 0.19: I added support for automatically renumbering objects on page load. This is mostly useful if you already have existing data which isn’t ordered yet.
feincms3-data 0.7: Made sure that objects are dumped in a deterministic order when dumping. I wanted to compare JSON dumps by hand before and after a big data migration in a customer project and differently ordered dumps made the comparison impossible.
django-prose-editor 0.9: I updated the ProseMirror packages, and put the editor into read-only mode for <textarea disabled> elements.
feincms3-language-sites 0.4: Finally released the update containing the necessary hook to validate page trees and their unique paths before moving produces integrity errors. Error messages are nicer than internal server errors.
django-authlib 0.17.2: The value of the cookie which is used to save the URL where users should be redirected to after authentication wasn’t checked for validity when setting it, only when reading it. This meant that attackers could produce invalid header errors in application servers. No real security problem here when using authlib’s code.
feincms3 5.3: Minor update which mostly removes support for outdated Python and Django versions.
django-imagefield 0.20: See above.

wonderfully ugly ↩

Weeknotes (2024 week 43)

2024-10-23T12:00:00Z

Weeknotes (2024 week 43)

I had some much needed time off, so this post isn’t that long even though four weeks have passed since the last entry.

From webpack to rspack

I’ve been really happy with rspack lately. Converting webpack projects to rspack is straightforward since it mostly supports the same configuration, but it’s much much faster since it’s written in Rust. Rewriting things in Rust is a recurring theme, but in this case it really helps a lot. Building the frontend of a larger project of ours consisting of several admin tools and complete frontend implementations for different teaching materials only takes 10 seconds now instead of several minutes. That’s a big and relevant difference.

Newcomers should probably still either use rsbuild, Vite or maybe no bundler at all. Vanilla JS and browser support for ES modules is great. That being said, I like cache busting, optimized bundling and far-future expiry headers in production and hot module reloading in development a lot, so learning to work with a frontend bundler is definitely still worth it.

Dark mode and light mode

I have been switching themes in my preferred a few times per year in the past. The following ugly bit of vimscript helps switch me the theme each time the sun comes out when working outside:

let t:light = 0
function! FiatLux()
    if t:light == 0
        :set background=light
        let t:light = 1
    else
        :set background=dark
        let t:light = 0
    endif
endfunction
nnoremap <F12> :call FiatLux()<CR>

I’m using the Ptyxis terminal emulator currently, I haven’t investigated yet if there’s a shortcut to toggle dark and light mode for it as well. Using F10 to open the main menu works fine though, and using the mouse wouldn’t be painful either.

Helping out in the Django forum and the Discord

I have found some pleasure in helping out in the Django Forum and in the official Django Discord. I sometimes wonder why more people aren’t reading the Django source code when they hit something which looks like a bug or something which they do not understand. I find Django’s source code very readable and I have found many nuggets within it. I’d always recommend checking the documentation or maybe official help channels first, but the code is also out there and that fact should be taken advantage of.

Releases

django-content-editor 7.1: Fixed a bug where the ordering and region fields were handled incorrectly when they appear on one line in the fieldset. Also improved the presentation of inlines in unknown regions and clarified the meaning of the move to region dropdown. Also, released the improvements from previous patch releases as a new minor release because that’s what I should have been doing all along.
form-designer 0.27: A user has been bitten by slugify removing cyrillic characters because it only keeps ASCII characters around. Here’s the wontfixed bug in the Django issue tracker: #8391. I fixed the issue by removing the slugification (is that even a word?) when generating choices.

Weeknotes (2024 week 39)

2024-09-25T12:00:00Z

Weeknotes (2024 week 39)

CSS for Django forms

Not much going on in OSS land. I have been somewhat active in the official Django forum, discussing ways to add Python-level hooks to allow adding CSS classes around form fields and their labels. The discussion on the forum and on the pull request goes in the direction of allowing using custom BoundField classes per form or even per project (instead of only per field as is already possible today). This would allow overriding css_classes, e.g. to add a simple class="field". Together with :has() this would probably allow me to skip using custom HTML templates in 99% of all cases.

I have also been lurking in the Discord, but more to help and less to promote my packages and ideas :-)

Releases

django-user-messages 1.1: Added Django 5.0, 5.1 to the CI, and fixed the migrations to no longer mention index_together at all. It seems that squashing the migrations wasn’t sufficient, I also had to actually delete the old migrations.
blacknoise 1.1: Starlette’s FileResponse has gained support for the HTTP Range header, allowing me to remove my homegrown implementation from the package. The blacknoise implementation is now half as long as it was in 1.0.
django-fhadmin 2.3: No new features, only tweaks to the styling and behavior prompted by updates to Django’s admin interface.
django-cabinet 0.17: I have pruned the CI matrix and accepted a pull request adding a ru translation. I feel conflicted about that since I strongly believe that everything is political, but I don’t know if rejecting translations helps anyone.

django-content-editor now supports nested sections

2024-09-13T12:00:00Z

django-content-editor now supports nested sections

django-content-editor (and it’s ancestor FeinCMS) has been the Django admin extension for editing content consisting of reusable blocks since 2009. In the last years we have more and more often started automatically grouping related items, e.g. for rendering a sequence of images as a gallery. But, sometimes it’s nice to give editors more control. This has been possible by using blocks which open a subsection and blocks which close a subsection for a long time, but it hasn’t been friendly to content managers, especially when using nested sections.

The content editor now has first-class support for such nested sections. Here’s a screenshot showing the nesting:

Finally it’s possible to visually group blocks into sections, collapse those sections as once and drag and drop whole sections into their place instead of having to select the involved blocks individually.

The best part about it is that the content editor still supports all Django admin widgets, as long as those widgets have support for the Django administration interface’s inline form events! Moving DOM nodes around breaks attached JavaScript behaviors, but we do not actually move DOM nodes around after the initialization – instead, we use Flexbox ordering to visually reorder blocks. It’s a bit more work than using a ready-made sortable plugin, but – as mentioned – the prize is that we don’t break any other Django admin extensions.

Simple patterns

I previously already reacted to a blog post by Lincoln Loop here in my post My reaction to the block-driven CMS blog post.

The latest blog post, Solving the Messy Middle: a Simple Block Pattern for Wagtail CMS was interesting as well. It dives into the configuration of a Wagtail stream field which allows composing content out of reusable blocks of content (sounds familiar!). The result is saved in a JSON blob in the database with all the advantages and disadvantages that entails.

Now, django-content-editor is a worthy competitor when you do not want to add another interface to your website besides the user-facing frontend and the Django administration interface.

The example from the Lincoln Loop blog post can be replicated quite closely with django-content-editor by using sections. I’m using the django-json-schema-editor package for the section plugin since it easily allows adding more fields if some section type needs it.

Here’s an example model definition:

# Models
from content_editor.models import Region, create_plugin_base
from django_json_schema_editor.plugins import JSONPluginBase
from feincms3 import plugins

class Page(models.Model):
    # You have to define regions; each region gets a tab in the admin interface
    regions = [Region(key="content", title="Content")]

    # Additional fields for the page...

PagePlugin = create_plugin_base(Page)

class RichText(plugins.richtext.RichText, PagePlugin):
    pass

class Image(plugins.image.Image, PagePlugin):
    pass

class Section(JSONPluginBase, PagePlugin):
    pass

AccordionSection = Section.proxy(
    "accordion",
    schema={"type": "object", {"properties": {"title": {"type": "string"}}}},
)
CloseSection = Section.proxy(
    "close",
    schema={"type": "object", {"properties": {}}},
)

Here’s the corresponding admin definition:

# Admin
from content_editor.admin import ContentEditor
from django_json_schema_editor.plugins import JSONPluginInline
from feincms3 import plugins

@admin.register(models.Page)
class PageAdmin(ContentEditor):
    inlines = [
        plugins.richtext.RichTextInline.create(models.RichText),
        plugins.image.ImageInline.create(models.Image),
        JSONPluginInline.create(models.AccordionSection, sections=1),
        JSONPluginInline.create(models.CloseSection, sections=-1),
    ]

The somewhat cryptic sections= argument says how many levels of sections the individual blocks open or close.

To render the content including accordions I’d probably use a feincms3 renderer. At the time of writing the renderer definition for sections is a bit tricky.

from feincms3.renderer import RegionRenderer, render_in_context, template_renderer

class PageRenderer(RegionRenderer):
    def handle(self, plugins, context):
        plugins = deque(plugins)
        yield from self._handle(plugins, context)

    def _handle(self, plugins, context, *, in_section=False):
        while plugins:
            if isinstance(plugins[0], models.Section):
                section = plugins.popleft()
                if section.type == "close":
                    if in_section:
                        return
                    # Ignore close section plugins when not inside section
                    continue

                if section.type == "accordion":
                    yield render_in_context("accordion.html", {
                        "title": accordion.data["title"],
                        "content": self._handle(plugins, context, in_section=True),
                    })

            else:
                yield self.render_plugin(plugin, context)

renderer = PageRenderer()
renderer.register(models.RichText, template_renderer("plugins/richtext.html"))
renderer.register(models.Image, template_renderer("plugins/image.html"))
renderer.register(models.Section, "")

Closing thoughts

Sometimes, I think to myself, I’ll “just” write a “simple” blog post. I get what I deserve when using those forbidden words. This blog post is neither short or simple. That being said, the rendering code is a bit tricky, the rest is quite straightforward. The amount of code in django-content-editor and feincms3 is reasonable as well. Even though it may look like a lot you’ll still be running less code in production than when using comparable solutions built using Django.

Weeknotes (2024 week 37)

2024-09-11T12:00:00Z

Weeknotes (2024 week 37)

I have helped mentoring Aman Pandey who has worked all summer to add async support to django-debug-toolbar. Tim has released an alpha which contains all of the work up to a few days ago. Test it! Let’s find the breakages before the final release.

Dropping Python 3.9 from my projects

I have read Carlton’s post about the only green Python release and have started dropping Python 3.9 support from many of the packages I maintain. This is such a good point:

[…] I’m also thinking about it in terms of reducing the number of Python versions we support in CI. It feels like a lot of trees to support 5 full versions of Python for their entire life. 🌳

Releases

django-debug-toolbar 5.0.0a0: See above.
form-designer 0.26.2: The values of choice fields are now returned as-is when sending mails or exporting form submissions instead of only returning the slugified version.
django-authlib 0.17.1: The role-based permissions backend had a bug where it wouldn’t return all available permissions in all circumstances, leading to empty navigation sidebars in the Django administration. This has been fixed.
feincms3 5.2.3: Bugfix release, the page moving interface is no longer hidden by an expanded navigation sidebar. I almost always turn off the sidebar in my projects so I haven’t noticed this.
django-prose-editor 0.8.1: Contains the most recent ProseMirror updates and bugfixes.
django-content-editor 7.0.10: Now supports sections. Separate blog post coming up!

Weeknotes (2024 week 35)

2024-08-28T12:00:00Z

Weeknotes (2024 week 35)

Getting deep into htmx and django-template-partials

I have been skeptical about htmx for some time because basically everything the library does is straightforward to do myself with a few lines of JavaScript. I am a convert now because, really, adding a few HTML attributes is nicer than copy pasting a few lines of JavaScript. Feels good.

The combination of htmx with django-template-partials is great as well. I didn’t know I had been missing template partials until I started using them. Includes are still useful, but replacing some of them with partials makes working on the project much more enjoyable.

I haven’t yet had a use for django-htmx but I may yet surprise myself.

Releases

django-authlib 0.17: django-authlib bundles authlib.little_auth which offers an user model which uses the email address as the username. I have also introduced the concept of roles instead of permissions; now I have reorganized the user admin fieldset to hide user permissions altogether. Group permissions are still available as are roles. I’m personally convinced that user permissions were a mistake.
feincms3-forms 0.5: Allowed setting a maximum length for the bundled URL and email fields through the Django administration interface.
django-content-editor 7.0.7: Fixed a bug where plugins with several fieldsets weren’t collapsed completely.
django-imagefield 0.19.1: Allowed deactivating autogeneration of thumbnails completely through the IMAGEFIELD_AUTOGENERATE setting. This is very useful for batch processing. Also, documented all available settings.
django-prose-editor 0.8: Added support for translating the interface elements and for restricting the available heading levels in the UI.
form-designer 0.25: Fixed the type of the author field for the send-to-author processing action.
feincms3 5.2.2: Added support for embedding SRF play external content. They do not support oEmbed unfortunately.
feincms3-cookiecontrol 1.5.3: Added support for SRF play embeddings as well. The difference is that the feincms3-cookiecontrol embedding requires consent before embedding external content.

How I handle versioning

2024-08-20T12:00:00Z

How I handle versioning

I have been reading up on versioning methods a bit, and I noticed that I never shared my a bit unorthodox versioning method. I previously wrote about my rules for releasing open source software but skipped everything related to versioning.

I use something close to the ideas of semantic versioning but it’s not quite that. Note that the versioning scheme has nothing to do with production readiness, it’s more about communicating the state of the project.

0.0.x – Everything breaks

I don’t trust my choices a lot. Everything may radically change, and I may also abandon the project with no hard feelings.

0.1.0 – Changelogging

I generally start writing release notes or rather a Changelog, since writing full release notes is much more work. You absolutely have to test the software and I do not guarantee anything related to backwards compatibility – just that you’ll know about breaking changes when you read the CHANGELOG.

?.?.x – Bugfixes and pure additions

Strictly speaking, patch version increments are only for bugfixes. However, when I add new features which are purely additional to the package or if there’s no way (famous last words) that anything will break, I’ll upload a patch release.

Sometimes it might be better to increment the minor version, yes. But, being very lazy, I sometimes only want to add a line to the CHANGELOG instead of adding a new heading for the new release. Also, if I don’t quickly release new additions there’s a real danger that I’ll only come back to the project weeks or months later, and I don’t want anyone (myself included) waiting on these updates. Also, if I inadvertently introduced new bugs it’s better to fix them quickly instead of forgetting about it first and having to rediscover the buggy code later.

1.0.0 – I’m happy

I’m happy with the package and I commit to start using semver more properly. If I have to (or want to) break backwards compatibility you get a migration path, especially if I need it myself. Which is often the case since I’m maintaining dozens of Django-based websites and webapps, and I’m always dogfooding my stuff.

The pure additions case from the last section still applies, since slapping a 1.0 version on something doesn’t suddenly make me less lazy.

Outliers

FeinCMS uses a variant of calendar versioning; the version at the time of writing is 24.8.2 which means the second release in August 2024.

There are probably other outliers I forgot about.

Weeknotes (2024 week 33)

2024-08-14T12:00:00Z

Weeknotes (2024 week 33)

Partying

It’s summer, it’s hot, and it’s dance week. Lethargy is over, Jungle Street Groove is coming up. Good times.

Releases

django-json-schema-editor 0.1: I have finally left the alpha versioning. I’m still not committing to backwards compatibility, but I have started writing a CHANGELOG.
django-prose-editor 0.7.1: Thanks to Carlton’s pull request I have finally cleaned up the CSS somewhat and made overriding the styles more agreeable when using the editor outside the Django administration. The confusing active state of menubar buttons has also been rectified. Docs are now available on Read the Docs.
django-imagefield 0.19: django-imagefield can now be used with proxy models. Previously, thumbnails weren’t generated or deleted when saving proxy models because the signal handlers would only be called if the sender matches exactly. I have already debugged this before, but have forgotten about it again. The ticket is really old for this, and fixing it isn’t easy since it’s unclear what should happen (#9318).
django-canonical-domain 0.11: django-canonical-domain has gained support for excluding additional domains from the canonical domain redirect. django-canonical-domain is used to redirect users to HTTPS (optionally) and to a particular canonical domain (as the name says). But sometimes you have auxiliary domains, e.g. for an API service, which shouldn’t be redirected. The package can now be used in these scenarios as well.
FeinCMS 24.8.2: The venerable FeinCMS, now more than 15 years old. The thumbnailing support had a bug where it tried saving JPEGs using RGBA (which obviously doesn’t work). This has been fixed.

Weeknotes (2024 week 31)

2024-07-31T12:00:00Z

Weeknotes (2024 week 31)

I have missed almost two months of weeknotes. I’ve got some catching up to do. I have tried writing a larger piece on my thoughts about CMS, but with everything going on in my personal and work life I haven’t made much progress.

This weeknotes entry is me trying to get back into the groove of writing (and publishing!) regularly.

django-prose-editor

I have previously written about the ProseMirror-based editor for Django websites here. I have continued working on the project in the meantime. Apart from bugfixes the big new feature is the support for showing typographic characters. For now the editor supports showing non-breaking spaces and soft hyphens. The project seems to get a little more interest after the deprecation of django-ckeditor has become more well known and the project has even received a contribution by someone else. It’s always a lovely moment when this happens.

django-json-schema-editor

Still alpha. Updated the vendorized JSON editor and fixed the integration into Django to not throw errors with the newer version.

Foreign key fields now support describing the referenced value similar to the raw ID fields functionality. Added optional support for using "format": "prose" to use the django-prose-editor to edit individual fields. JSON plugins for the content editor are now downcasted into their proxy models automatically. This is especially useful with the feincms3 changes mentioned below. (You do not have to use either django-content-editor or feincms3 to use this package!)

The following screenshot shows the prose editor integration; the mentioned foreign key field description isn’t visible yet here.

Traduire

Traduire (french for «translate») is a web-based platform for editing gettext translations.

It is intended as a replacement for Transifex, Weblate and comparable products. It is geared towards small teams or agencies which want to allow their customers and their less technical team members to update translations.

Traduire profits from the great work done on django-rosetta. I would still be using Rosetta if it would work when used with a container orchestator such as Kubernetes. Since all application storage is ephemeral that doesn’t work, translation editing and deployment have to be separated.

It is built using Django and relies on polib to do the heavy lifting.

This is a project which might also be interesting for others. I would especially appreciate it if someone could contribute an easier way to get it up and running, e.g. using a Docker Compose configuration or something. I am using Kubernetes and GitOps to host it, but that’s not straightforward at all. Really, all that’s needed to run it is a Django host with any database which is supported by Django. I prefer using PostgreSQL because I have it, but sqlite etc. work just as well.

Releases since the second week of June

django-translated-fields 0.13: Nothing much except for CI and pre-commit updates. The implementation continues to be rock-solid and basically unchanged.
django-content-editor 7.0.6: Tweaks and fixes to the new interface. Added better scrolling behavior when dragging content around. The editor now also supports colorized icons which helps quickly understanding the structure of some content when there are many plugins.
blacknoise 1.0.2: Fixed a few bugs in the blacknoise.compress utility and started running the testsuite on GitHub actions. whitenoise has been friendly-forked as ServeStatic and I’m definitely having a close look at this project as well, but blacknoise is simple and works well, so I’m not convinced that switching back to the much larger project (in terms of amounts of code) is an improvement now.
django-authlib: Minor bugfixes.
feincms3: Allowed registering plugin models with the renderer which aren’t supposed to be fetched from the database. This is especially useful when used together with JSON plugins, where the individual JSON plugins are created as proxies for the underlying Django model and fetched all at once. Disabled the version check on our CKEditor plugin. Still, really stop using CKEditor 4 if you want to use maintained software.
FeinCMS 24.7.1: Small bugfixes, and made the Read the Docs build work correctly.
django-debug-toolbar 4.4.x: The toolbar continues to be a nice project to work on. Fixed a few edge cases in the new alerts panel.
django-admin-ordering 0.18.2: The value of ordering_field now has additional sanity checks.
django-cabinet 0.16: cabinet now supports exporting a folder as a ZIP file while preserving the structure you see in the CMS instead of the structure on the file system. The inline upload form has been dropped from the CabinetForeignKey widget because the folder dropdown slowed down the page a lot when used on a site with many folders. Using the raw ID fields popup isn’t that bad.
django-prose-editor 0.6.2: See above.
django-json-schema-editor 0.0.28: See above.

Weeknotes (2024 week 23)

2024-06-07T12:00:00Z

Weeknotes (2024 week 23)

Switching everything from pip to uv

Enough said. I’m always astonished how fast computers can be.

Releases

django-admin-ordering 0.18: Added a database index to the ordering field since we’re always sorting by it.
django-prose-editor 0.4: Dropped the jQuery dependency making it possible to use the editor outside the Django administration interface without annoying JavaScript errors. Allowed additional heading levels and moved the block type buttons into a popover.
django-debug-toolbar 4.4.2: I enjoy working on this important piece of software very much.
django-email-hosts 0.2.1: Added a command analogous to ./manage.py sendtestemail so that it’s possible to easily test the different configured email backends.
feincms3 5.0: I completely reworked the move node action; previously it opened a new page where you could see all possible targets; now you can cut a page and paste it somewhere else. The advantages of the new interface is that you don’t leave the changelist and can still profit from all its features while moving pages around.
feincms3-sites 0.21: A new release taking advantage of a new hook in feincms3 7.0 so that the new moving interface works.
django-authlib 0.16.5: authlib now shows a welcome message when authenticating using admin OAuth2. It’s nice and helps with debugging strange authentication failures.
django-content-editor 7.0: I reworked the UI. The sidebar is gone, instead there are nice buttons in the place where you can add new plugins; the plugins appear in a nice grid instead of a list, which looks much better once you have more than just a few plugin types available. Also, plugin type icons are now shown in the plugin forms. I think it looks much better than before.
feincms3-cookiecontrol 1.5.2: I didn’t contribute anything to this release which is also a nice experience for a change. The Google consent mode integration has been improved and simplified.
django-json-schema-editor 0.0.22: Various small-ish improvements. I should really start using higher version numbers, but not having to commit to anything also feels great. That being said, the editor is in active use in several projects, so maybe I’m deceiving myself.

Workbench: Coffee time!

2024-05-24T12:00:00Z

Workbench: Coffee time!

I have written about the Workbench agency software a few weeks back.

Back when we were using Slack at Feinheit we used a Donut bot to generate randomized invites for a coffee break. We lost that bot when we switched to Discord. I searched some time for an equivalent bot for Discord but couldn’t find any, so like any self-respecting nerd, break lover and NIH-sufferer I reimplemented the functionality as a part of our agency software.

The first version generated totally random pairings without any history; I thought that maybe this would be good enough, but of course I received an invite for a coffee with the exact same person in the first two invitations. Even though I did enjoy the break both times this motivated me to put some effort into a better solution :-)

The result of this work is the coffee_invites() function

There are two main parts to the generator:

It randomly generates twenty pairings from all participants; it prefers groups of two except for the last group which may contain three people.
It loads old pairings from the database and gives higher penalties to equal pairings depending on the time which has passed since the last time.

So, it’s not mathematically perfect, but all imbalances will cancel out over a long time. I experimented with the discounting factor and the number of random pairings it generates upfront. The values in the code seem to work fine.

Sometimes it’s simple stuff like this which gets me going and reignites the love for programming. Like a small puzzle.

Weeknotes (2024 week 21)

2024-05-22T12:00:00Z

Weeknotes (2024 week 21)

There have been times when work has been more enjoyable than in the last few weeks. It feels more stressful than at other times, and this mostly has to do with particular projects. I hope I’ll be able to move on soon.

blacknoise

I have released blacknoise 1.0. It’s an ASGI app for static file serving inspired by whitenoise.

The 1.0 version number is only a big step in versioning terms, not much has happened with the code. It’s a tiny little well working piece of software which has been running in production for some time without any hickups. The biggest recent change is that I have parallelized the gzip and brotli compression step; this makes building images using whitenoise more painful because there the wait is really really long sometimes. A pull request fixing this exists, but it hasn’t moved forwards in months.

I have written a longer post about it earlier this year here.

Releases

feincms3-cookiecontrol 1.5: Code golfing. Added backwards compatibility with old Django versions so that I can use it for old projects. Also includes optional support for Google consent management.
django-fast-export 0.1.1: This is basically a repackaging of the streaming CSV view from Django’s documentation as a reusable class. I have switched to using an iterator so that I can export even larger datasets.
django-json-schema-editor 0.0.18: Still alpha versioned but used in production in various projects. I should really release an 1.0 version, but there are no integration tests at all. Mainly visual tweaks in this update.
django-content-editor 6.5: Better handling of templates and regions when a particular editor instance only shows a subset of configured templates. Disallowed adding plugins when in an unknown region. It’s funny how many edge cases exist in software as old as this.
blacknoise 1.0: See above.
html-sanitizer 2.4.4: Fixed edge cases with whitespace handling when merging elements.
feincms3-data 0.6.1: Allowed ./manage.py f3loaddata - to load JSON data from stdin.

Weeknotes (2024 week 18)

2024-05-03T12:00:00Z

Weeknotes (2024 week 18)

Google Summer of Code has begun

We have a student helping out with adding async support to the Django Debug Toolbar. It’s great that someone can spend some concentrated time to work on this. Tim and others have done all the necessary preparation work, I’m only helping from the sidelines so don’t thank me.

Bike to Work

Two teams from my company are participating in the Bike to Work Challenge 2024. It’s what I do anyway (if I’m not working from home) but maybe it helps build others some motivation to get on the bicycle once more. Public transports in the city where I live are great but I’ll always take the bike when I can. I also went on my first mountain bike ride in a few months yesterday, good fun.

JSON blobs and referential integrity

The django-json-schema-editor has gained support for referencing Django models. Here’s an example schema excerpt:

{
    ...
    "articles": {
        "type": "array",
        "format": "table",
        "title": _("articles"),
        "minItems": 1,
        "maxItems": 3,
        "items": {
            "type": "string",
            "title": _("article"),
            "format": "foreign_key",
            "options": {
                "url": "/admin/articles/article/?_popup=1&_to_field=id",
            },
        },
    },
    ...
}

The ID field is stringly typed; using an integer directly wouldn’t work because the empty string isn’t a valid integer.

The problem with referencing models in this way is that there’s no way to know if the referenced object is still around or not, or even to protect it against deletion. The bundled django-content-editor JSONPlugin now supports automatically generating a ManyToManyField with a through model which protects articles from deletion as long as they are referenced from a JSONPlugin instance. The register_reference line creates the mentioned model with an on_delete=models.PROTECT foreign key to articles and a post_save handler which updates said references.

from django_json_schema_editor.plugins import JSONPluginBase, register_reference
from articles.models import Article

class JSONPlugin(JSONPluginBase, ...):
    pass

register_reference(JSONPlugin, "articles", Article)

Releases since the beginning of April

django-json-schema-editor 0.0.14: See above. Also, some styling work and a patch update to the vendorized json-editor.
django-content-editor 6.4.6: Many small stylistic fixes. The target indicator when dragging plugins is now also shown when plugins are collapsed. It’s now possible to directly drag a plugin to the end, and not just to the second to last position.
django-prose-editor 0.3.4: Switched to the nh3 sanitizer because it’s faster and because ProseMirror never emits HTML which has to be cleaned up first. Stopped generating menu items for nodes and marks which aren’t in the schema. Added the possibility to reduce the functionality per editor instance. Small tweaks and fixes.
django-tree-queries 0.19: Added support for pre-filtering the tree (much more efficient when only querying a part of the tree). Added support for adding additional fields to the CTE so that you can collect values from ancestors for other fields than the default fields too.
FeinCMS 24.4.2: Added support for webp images. Fixed a few of the admin list filters to work with Django 5.
django-cabinet 0.14.3: Fixed the support for the extra_context argument to our changelist_view implementation.

Workbench, the Django-based agency software

2024-04-24T12:00:00Z

Workbench, the Django-based agency software

I get the impression that there’s a lot of interesting but unknown software in Django land. I don’t know if there’s any interest in some of the packages I have been working on; if not this blog post is for myself only.

(Hi)story time

As people may know I work at Feinheit, an agency which specializes in digital communication services for SMEs, campaigns for referendums, and website and webapp development. At the time of writing we are a team of about 20-25 communication experts, graphic designers, programmers and project managers.

We have many different clients and are working on many different projects at the same time and are billing by the hour¹. Last year my own work has been billed to more than 50 different customers. In the early days we used a shared file server with spreadsheet files to track our working hours. Luckily we didn’t often overwrite the edits others made but that was definitely something which happened from time to time.

We knew of another agency who had the same problems and used a FileMaker-based software. Their solution had several problems, among them the fact that it became hard to evolve and that it got slower and slower as more and more data was entered into it over the years. They had the accounting know how and we had the software engineering know how so we wrote a webapp based on the Django framework. As always, it was much more work than the initial estimate, but if we as programmers didn’t underestimate the effort needed we wouldn’t have started many of the great projects we’re now getting much value and/or enjoyment from, hopefully both. The product of that work was Metronom. The first release happened a little bit later than harvest but it already came with full time tracking including an annual working time calculator, absence management, offers, invoices including PDF generation etc, so it was quite a bit more versatile while still being easier to use than “real” business software solutions.

I personally was of the opinion that the product was good enough to try selling it, but for a variety of reasons (which I don’t want to go into here) this never happened and we decided that we didn’t want to be involved anymore.

However, this meant that we were dead-end street with a software that didn’t belong to us anymore, which wasn’t evolving to our changing requirements. I also didn’t enjoy working on it anymore. Over the years I have tried replacing it several times but that never came to pass until some time after the introduction of Holacracy at our company. I noticed that I didn’t have to persuade everyone but that I, as the responsible person for this particular decision, could “just”² move ahead with a broad interpretation of the purpose and accountabilities of one of my roles.

Workbench

Workbench is the product of a few long nights of hacking. The project was started as an experiment in 2015 and was used for sending invoices but that wasn’t really the intended purpose. After long periods of lying dormant I have brought the project to a good enough state and switched Feinheit away from Metronom in 2019.

I have thought long and hard about switching to one of the off-the-shelf products and it could very well be that one of them would work well for us. Also, we wouldn’t have to pay (in form of working hours) for the maintenance and for enhancements ourselves. On the other hand, we can use a tool which is tailored to our needs. Is it worth the effort? That’s always hard to answer. The tool certainly works well for the few companies which are using it right now, so there’s no reason to agonize over that.

At the time of writing, Workbench offers projects and services, offers and invoices incl. PDF generation, recurring invoices, an address book, a logbook for rendered services, annual working time reports, an acquisition funnel, a stupid project planning and resource management tool and various reports. It has not only replaced Metronom but also a selection of SaaS (for example Pipedrive and TeamGantt) we were using previously.

The whole thing is open source because I don’t want to try making agency software into a business anymore, I only want to solve the problems we have. That being said, if someone else finds it useful then that’s certainly alright as well.

The license has been MIT for the first few years but I have switched to the GPL because I wanted to integrate the excellent qrbill module which is also licensed under the GPL. As I wrote elsewhere, I have released almost everything under the GPL in my first few open source years but have switched to BSD/MIT later when starting to work mainly with Python and Django because I thought that this license is a better fit³ for the ecosystem. That being said, for a product such as this the GPL is certainly an excellent fit.

Final words?

There are a few things I could write about to make this into a series. I’m putting a few ideas here, not as an announcement, just as a reminder for myself.

Better time tracking
Patterns for creating recurring invoices
Feature switches and configurability
Coffee time

Rather, in six minute increments. It’s even worse. ↩
Of course it’s never that simple, because the responsibility is a lot. The important point is: It would have been a lot of work anyways, the big difference is that it was sufficient to get consent from people; no consensus required. ↩
That’s not meant as a criticism in any way! ↩

Building forms with the Django admin

2024-04-12T12:00:00Z

Building forms with the Django admin

The title of this post was shamelessly copied from Jeff Triplett’s post on Mastodon.

Why?

Many websites need a simple way of embedding forms, for example as a contact form or for simple surveys to collect some data or inputs from visitors. Django’s forms library makes building such forms straightforward but changing those forms requires programming skills and programmer time. Both of them may not be readily available. More importantly, sometimes it’s just nice to give more tools to web publishers.

The simple way to build something like this is to use a form builder such as Google Forms, Typeform, Paperform or anything of the sort. Those options work nicely. The downsides are that embedded forms using those services load slowly, look differently, cost a lot or collect a lot of data on users, or all of those options. Because of that there’s still a place for building such functionality locally.

If I wanted to use PHP and WordPress I could just use WPForms and call it a day. Since I do not actually want that this blog post is a bit longer.

The early days: form-designer

One of the first Django-based third party apps I published was the form-designer. The first version was uploaded to PyPI in 2012 but it had already been used in production for more than two years at that point in time. I had used Git submodules for the deployment back then, before switching to Python virtualenvs some time later (and never looking back!)

The form-designer is still maintained actively. Because of Django’s stability and because of the fact that the app doesn’t do all that much it doesn’t require much development at all.

The form designer supports a selection of standard HTML5 input fields out of the box and also has an optional django-recaptcha integration. All fields support some basic configuration such as setting a title, a help text, marking the field as required etc. Submissions can be sent to a configurable email address and can be saved in the database and later exported as an XLSX file. It’s also possible to define your own actions.

More flexibility needed: feincms3-forms

A few years back I mentioned feincms3-forms in a weeknotes entry. The reasons why form-designer wasn’t sufficient for a project back then are outlined in the blog post:

feincms3-forms – A new forms builder for the Django admin interface

For a current project we needed a forms builder with the following constraints:

Simple fields (text, email, checkboxes, dropdowns etc.)

Custom validation and processing logic

It should be possible to add other content, e.g. headings and explanations between form fields

The form_designer fulfilled a few of these requirements but not all. It still works well but I wanted a forms builder based on django-content-editor for a long time already. Also, I really like the feincms3 pattern where the third party app only provides abstract models. Yeah, it is much more work to start with but the flexibility and configurability is worth it – especially since it’s possible to write straightforward code to handle special cases[^2] instead of configuring even more settings.

The humble beginnings are here in the feincms3-forms repository. The test suite already shows how things work together but as of now no documentation exists and no release has been made yet. I hope it will be ready for a first beta release in the next few weeks 😄

Since then I have used feincms3-forms more often than form-designer, for building simple forms and also to build multi-step form wizards with custom fields, custom validation, configurable steps etc. The README now actually explains why the project exists and how it could be used.

It still doesn’t come close to WPForms in terms of included functionality; a big feature which is missing is conditional logic because I haven’t yet had a use for it.

The feincms3-forms forms support all types of content between form fields (basically everything django-content-editor supports). Plugins for form fields are more flexible and can add as many input fields to the form as they want, you’re not restricted to single values or single input fields.

Packages

Weeknotes (2024 week 14)

2024-04-06T12:00:00Z

Weeknotes (2024 week 14)

I’m having a bit of a slow week with the easter weekend and a wisdom tooth extraction. I’m recovering quite quickly it seems and I’m glad about it.

This weeknotes entry is short and quick. I’m trying to get back into the habit of writing them after a mediocre start this year.

20th Anniversary Celebration of Young Greens Switzerland

I have attended the celebration of Young Greens Switzerland. I have been a founding member and have been active for close to ten years. A lot of time has passed since then. It has been great to reminisce about old times with friends and, more importantly, to see how the torch is carried on.

Releases

blacknoise 0.0.5: blacknoise is an ASGI app for static file serving inspired by whitenoise. It only supports a very limited subset of whitenoise’s functionality, but it supports async.
html-sanitizer 2.4.1: The lxml library moved the HTML cleaner into its own package, lxml-html-clean; this release adds support for that. I didn’t know that the HTML cleaner is viewed as being problematic by the lxml maintainers. I’m having another look at nh3 and will maybe switch html-sanitizer’s guts from lxml to nh3 in the future.
django-tree-queries 0.18: django-tree-queries now supports ordering siblings by multiple fields and even allows descending orderings.
django-cabinet 0.14.2: This release fixes the CKEditor 4 filebrowser popup when using Django 5 or better.

The Django admin is a CMS

2024-03-27T12:00:00Z

The Django admin is a CMS

The post Why is the Django Admin “Ugly”? and the discussion on Mastodon around it finally motivated me to write down my thoughts regarding the recurring theme in Django land that the Django administration interface isn’t a CMS (Content Management System).

I think that this is misguided and needlessly limits the discourse around what the admin’s current functionality is and the ideas what it could be and already is.

A web content management system is about website authoring for users who do not need to be web programming experts in their own rights. Django was created at the Lawrence Journal-World newspaper. The admin itself was created to allow quickly spinning up new websites, where the admin interface was used by content managers to fill in the content while programmers finalized the rest of the website. So obviously the admin interface was a system used to manage content¹ from the beginning.

Sure, the Django admin site documentation states:

One of the most powerful parts of Django is the automatic admin interface. It reads metadata from your models to provide a quick, model-centric interface where trusted users can manage content on your site. The admin’s recommended use is limited to an organization’s internal management tool. It’s not intended for building your entire front end around. [emphasis added]

In other words, the Django documentation also points out that the admin is powerful and that it allows trusted users to manage content¹.

Yes, it will be very painful if you try to do everything on top of the Django admin site. The warnings against using the Django admin for more than it was designed to are necessary and I totally support them. As soon as you’re getting into workflows, into complex permission scenarios (sad noises) or similar things the admin definitely isn’t for you. But, the admin nicely solves 90% of the problems with 10% of the effort. And it’s very good at that.

And sure, if you try building your own frontend on top of the Django admin you’re in for a bumpy ride, but that much should be obvious.

Many third party apps for Django actually target the Django admin interface itself, and not one of the (excellent!) Django-based CMS such as Wagtail. This means that by building on the Django admin instead of one of the CMS you’re running less code, by using more libraries instead of frameworks (on top of frameworks) you’re keeping maintenance lower, and you’re a part of a larger community², which brings the potential benefit of being able to profit more from the general Django packages ecosystem.

Since you’re depending on smaller pieces of additional software it will generally be possible to upgrade to new Django versions quicker. This isn’t true for all packages of course, and I’m a reluctant maintainer of some of them. Anecdotes aren’t data, but I see that some larger CMS systems are definitely having a hard time keeping up with Django’s release schedule.

I’m not trying to say that the Django admin is a better CMS than other Django-based CMS, or any other CMS. I’m saying it’s a trade off and you should be mindful of the downsides of choosing a larger system. And I’m saying that the people who tell you that you shouldn’t be using the Django admin interface are wrong in the first approximation.

The fact that it’s so easy to spin up an additional site and with minimal effort and still be able to work with clean database schemas and all the great tools Django (and Python) offers is important for those of us who are working on many different projects with limited financial resources, because the website often is for example just a small part of a campaign.

Sorry-not-sorry for my choice of words. ↩↩
The assumption that the communities of these Django-based CMS projects are a subset of the Django community itself shouldn’t be too controversial. ↩

blacknoise – ASGI app for static file serving

2024-03-20T12:00:00Z

blacknoise – ASGI app for static file serving

Note

This blog post consists of the blacknoise README at the time of publishing.

I have released blacknoise 1.0 in the meantime and believe that it’s actually good.

blacknoise is an ASGI app for static file serving inspired by whitenoise and following the principles of low maintenance software.

This is pre-alpha software and everything is subject to change. I’m not even sure if blacknoise should exist at all or if the energy wouldn’t be better spent improving whitenoise or other tools. Feedback and contributions are very welcome though!

Using blacknoise with Django to serve static files

Install blacknoise into your Python environment:

pip install blacknoise

Wrap your ASGI application with the BlackNoise app:

from blacknoise import BlackNoise
from django.core.asgi import get_asgi_application
from pathlib import Path

BASE_DIR = Path(__file__).parent

application = BlackNoise(get_asgi_application())
application.add(BASE_DIR / "static", "/static")

BlackNoise will automatically handle all paths below the prefixes added, and either return the files or return 404 errors if files do not exist. The files are added on server startup, which also means that BlackNoise only knows about files which existed at that particular point in time.

Improving performance

BlackNoise has worse performance than when using an optimized webserver such as nginx and others. Sometimes it doesn’t matter much if the app is behind a caching reverse proxy or behind a content delivery network anyway. To further support this use case BlackNoise can be configured to serve media files with far-future expiry headers and has support for serving compressed assets.

Compressing is possible by running:

python -m blacknoise.compress static/

BlackNoise will try compress non-binary files using gzip or brotli (if the Brotli library is available), and will serve the compressed version if the compression actually results in (significantly) smaller files and if the client also supports it.

Far-future expiry headers can be enabled by passing the immutable_file_test callable to the BlackNoise constructor:

def immutable_file_test(path):
    return True  # Enable far-future expiry headers for all files

application = BlackNoise(
    get_asgi_application(),
    immutable_file_test=immutable_file_test,
)

Maybe you want to add some other logic, for example check if the path contains a hash based upon the contents of the static file. Such hashes can be added by Django’s ManifestStaticFilesStorage or by appropriately configuring bundlers such as webpack and others.

License

blacknoise is distributed under the terms of the MIT license.

Podcasts I like listening to

2024-03-18T12:00:00Z

Podcasts I like listening to

I discovered listening to podcasts about one year ago. Previously, I never knew why anyone would want to listen to people talk when they could listen to music or nothing, but that has changed a bit.

So, here’s a list of podcasts I’m currently listening to on a regular basis.

Tech Won’t Save Us

I have recently stumbled over Tech Won’t Save Us, a Podcast which is critical of the technological “progress” offered by Silicon Valley elites. It’s a great antidote for the generative AI hype.

I still have some stupid hope that AGI will solve our problems because maybe people will trust a computer more than scientists that something has to be done about the combined crisis we’re facing as humans. Who knows. I do not want to be too negative about it though, there are positive news if you’re looking in the right places. I myself do not write about those bigger issues as much as I used to. I support those more important issues elsewhere.

Generative AI is certainly not helping me in my work. I do not want a computer to generate code which I have to review and maintain when I still enjoy writing code myself. Maybe that will change at some point in the future. When that happens I’ll probably retire and become a gardener or something.

The Ezra Klein Show

This show seems to be a favorite of many people. I joined for the (relatively) critical perspectives on AI and stayed for the insights into politics and into the near east conflict. I think it’s great that people from different sides get a voice on the show, even if Ezra doesn’t agree with them.

I sometimes wish the questions they ask on the show were a little bit more critical.

Wir müssen reden. Public Eye spricht Klartext: The podcast from public eye, an organization which uncovers human right violations perpetrated by Swiss companies around the globe. I’m not yet sure if I like the podcast, but I like and support the organization very much.
Hanselminutes: The more polifor the wide and diverse array of guests.tical podcasts have displaced Hanselminutes in my listening schedule, but I’d definitely recommend it to anyone who wants to learn more about tech. Recommended because of the diverse array of guests, among many other reasons.
Syntax: At times very interesting, at times not much new for me. I still recommend it.
Weird Studies: Really weird. I liked the discussion of Hellraiser and The Thing, and have listened to some of the other episodes. Religiosity and spiritualism was always around me when I was a child, and it’s interesting to revisit some of the ideas about reality from a different angle after years and years of not engaging with these questions at all. (Writing this down does sound a bit like a midlife crisis on the horizon, but I don’t think that’s it. I have studied a little bit of philosophy and ethics and have stayed interested in these topics ever since.)
I don’t think the german “Welt” is a good newspaper at all. I do like the «Aha! History» and «Aha! Zehn Minuten Alltagswissen» podcasts because they are short and often examine topics I don’t know much about.
Sternstude Philosophie: It’s a bit slow and definitely more geared towards viewers than listeners, but definitely interesting.

Weeknotes (2024 week 11)

2024-03-16T12:00:00Z

Weeknotes (2024 week 11)

Estimates

Jacob wrote an excellent post on breaking down tasks. I did like the post a lot. Maybe I’ll write a longer reply later, but for now just this. There definitely are good reasons for the pushback against estimation, and it’s really not just that some people lack professionalism.

Releases

django-cabinet 0.14.1: Mini release containing a Turkish translation. It’s always nice if software is used.
feincms3 4.6: Fixed a bug where the move form wouldn’t use a potentially overridden ModelAdmin.get_queryset method.
form-designer 0.24: Updated the package for django-recaptcha 4.0.
html-sanitizer 2.3.1: Fixed an edge case sanitization bug (luckily without security implications).
django-content-editor 6.4.2: django-content-editor now again supports transitioning plugin fieldsets when opening and closing thanks to CSS grid’s ability to animate the maximum height of an element. Also, the initialization in 6.4 was badly broken.
django-prose-editor 0.2: See the announcement blog post from Wednesday.