Matthias Kestenholz: Posts about Django

Weeknotes (2024 week 51)

2024-12-20T12:00:00Z

Weeknotes (2024 week 51)

Building forms using Django

I last wrote about this topic in April. It has resurfaced on Mastodon this week. I’m thinking about writing a feincms3-forms demo app, but I already have too much on my plate. I think composing a forms builder on top of django-content-editor is the way to go, instead of replacing the admin interface altogether – sure, you can always do that, but it’s so much less composable…

Releases

blacknoise 1.2: No real changes, added support for Python 3.13 basically without changing anything. It’s always nice when this happens.
django-imagefield 0.21
django-prose-editor 0.10: I rebuilt django-prose-editor from the ground up and wrote about that two weeks ago. The 0.10 release marks the final point of this particular rewrite.
django-js-asset 3.0: See the blog post from this week

Object-based assets for Django's forms.Media

2024-12-18T12:00:00Z

Object-based assets for Django’s forms.Media

The pull request for adding object-based script media assets into Django is in a good state and I hope it will be merged soon. I have been using object-based assets long before Django actually added support for them in 4.1 (since 2016, that’s before Django 1.10!) by using a gross hack. Luckily I have been able to clean up the code when Django 4.1 landed.

I have been asking myself at times why I haven’t proposed the change to Django myself despite having been a user of something like this for such a long time. After all, I have been happily contributing issue reports, bug fixes and tests to Django. The process of adding new features sometimes is terribly frustrating though even when looking (and cheering) from the sidelines. It feels bad that adding another package to the list of packages I maintain so clearly seems to be the better way to get things done compared to proposing a new feature for Django itself. I hope processes change somewhat.

But I digress.

The ProseEditorWidget in django-prose-editor wants to ship CSS, JavaScript and some JSON to the browser for the widget. So, of course I used object-based media assets for this instead of widget HTML templates. Media assets are deduplicated and sorted by Django. If different editor presets use differing lists of assets they are smartly merged by forms.Media using a topological sort. You get those niceties for free when using forms.Media and everything just works, so what’s not to like?

The only thing which isn’t to like is that Django, at the time of writing, doesn’t provide any classes helping with this. You can put strings into forms.Media or you can put objects with a __html__() method in there. The latter of course is all that’s needed to support more advanced use cases – and that’s exactly what django-js-asset now provides, and what django-prose-editor uses.

django-js-asset has long supported a JS class with support for additional attributes, for example:

from js_asset import JS

forms.Media(js=[
    JS("asset.js", {"id": "asset-script", "data-answer": "42"}),
])

Since 3.0 the package also comes with a CSS and JSON class:

from js_asset import CSS, JS, JSON

forms.Media(js=[
    JSON({"cfg": 42}, id="widget-cfg"),
    CSS("widget/style.css"),
    CSS("p{color:red;}", inline=True),
    JS("widget/script.js", {"type": "module"}),
])

This produces the following HTML:

<script id="widget-cfg" type="application/json">{"cfg": 42}</script>
<link rel="stylesheet" href="/static/widget/style.css">
<style>p{color:red;}</style>
<script src="/static/widget/script.js" type="module"></script>

The code which is proposed for Django supports the JavaScript use case but with a slightly different API:

forms.Media(js=[
    Script("widget/script.js", type="module"),
])

This looks slightly nicer as long as you don’t use e.g. data attributes, because then you have to do:

forms.Media(js=[
    Script("widget/script.js", **{"data-cfg": ...}),
])

I always forget that Python supports passing keyword arguments names which aren’t valid Python identifiers (but only when using **kwargs). I personally don’t care much either way, and when my packages can finally drop compatibility with Django versions which do not support all these functionalities yet I’ll finally be able to retire django-js-asset. That won’t happen any time soon though, if only because I like supporting old versions of Django because I have so many Django-based websites running somewhere.

Weeknotes (2024 week 49)

2024-12-06T12:00:00Z

Weeknotes (2024 week 49)

Django Steering Council elections

I have been thinking long and hard about running for the Django Steering Council. I think there are a few things I could contribute since I’ve been using Django for 16 or more years, and have been working on, maintaining and publishing third-party apps almost all this time. I have also contributed a few small features to Django core itself, and contributed my fair share of tests and bugfixes. The reason why I haven’t been more involved was always that I feared the review process with what I perceive to be a too unrestrained perfectionism. Teaching people is good, but I fear that those who teach are self-selected survivors of the process, which come to appreciate the perfectionism a bit too much. It’s somewhat the same as with the Swiss naturalization process – the hurdles are very high, and some of those who weather the process maybe are or grow to be too fond of it.

An important point is that this has nothing to do with being nice (or not). Everybody has always been great, maybe with the exception of myself back when I didn’t understand that the problem wasn’t the individuals but the way everyone has agreed things should be done.

I’m not the only one who thinks that we should improve the process somewhat. So, I’m definitely going to look out for candidates who think this is important.

There are a few reasons why I’m not running myself at this time. A somewhat important reason is that my candidacy wouldn’t help diversity at all. This shouldn’t discourage anyone else with the same background from running – we cannot change the world all at once. More importantly, I have more personal reasons for being hesitant to accept new commitments. That being said, I’m looking forward to be more involved in the community in other ways. And also, it’s not now or never.

Releases

feincms3-cookiecontrol 1.5.4: No functional changes, only code golfing. It’s nice to have a working cookie banner with a solution for embedding third party content only when people consent in less than 4KiB of minified (not compressed!) JavaScript.
django-admin-ordering 0.20: Objects can now be reordered using arrow buttons instead of drag drop or manually changing the ordering field’s value. This should make the package more accessible. It’s always a joy when people contribute such useful improvements.
django-prose-editor 0.10a?: See the recent blog post.

Rebuilding django-prose-editor from the ground up

2024-12-04T12:00:00Z

Rebuilding django-prose-editor from the ground up

The django-prose-editor package provides a HTML editor based upon the ProseMirror toolkit for the Django administration interface and for the frontend.

The package has been extracted from a customer project and open sourced so that it could be used in other projects as well. It followed a very restricted view of how rich text editors should work, which I have initially added to the FeinCMS repository when documenting the design decisions more than 15 years ago (Note that I didn’t edit the paragraph, it’s reproduced here as it was back then, with all the errors and heedlessness.)

All of this convinced me that offering the user a rich text editor with too much capabilites is a really bad idea. The rich text editor in FeinCMS only has bold, italic, bullets, link and headlines activated (and the HTML code button, because that’s sort of inevitable – sometimes the rich text editor messes up and you cannot fix it other than going directly into the HTML code. Plus, if someone really knows what he’s doing, I’d still like to give him the power to shot his own foot).

My personal views are unchanged. I have to recognize though that forcing this idea upon everyone isn’t workable and that this would mean that I’d have to find a different editor for most projects just because people really want or need more rope. Going back to an editor which allows everything was out of the question, so I had to look around for a way to allow project-specific extensions for the editor.

Of course that’s problematic, since Django packages and Python virtualenvs do not offer a good way of shipping CSS and JavaScript which should be available for a frontend bundler to process. The existing Django staticfiles app is great, works well, but it’s not a bundler – and it shouldn’t be.

So, I started shopping around for ways to make ProseMirror extensible while keeping extensions clean and well localized. Instead of inventing another plugin ecosystem I settled on Tiptap which uses ProseMirror under the hood. The abstractions are pleasantly leaky – if you know how to work with ProseMirror’s API, you can use Tiptap’s API without any issues. That was important for me, since I already have a somewhat large selection of plugins which I do not want to reimplement from the ground up.

I had already looked at Tiptap a few years back, but ultimately stayed with ProseMirror because I liked some behaviors better (such as not including trailing spaces in marks) and because I didn’t need the extensibility which at the time only made the resulting bundle much bigger.

Now, things have improved a lot, and I’m really happy with Tiptap and the development version of django-prose-editor. Writing an editor extension in project code is great, and my editor core stays nice. Also the list of readily available extensions is large, and most of the things just work.

Weeknotes (2024 week 47)

2024-11-20T12:00:00Z

Weeknotes (2024 week 47)

I missed a single co-writing session and of course that lead to four weeks of no posts at all to the blog. Oh well.

Debugging

I want to share a few debugging stories from the last weeks.

Pillow 11 and Django’s `get_image_dimensions`

The goal of django-imagefield was to deeply verify that Django and Pillow are able to work with uploaded files; some files can be loaded, their dimensions can be inspected, but problems happen later when Pillow actually tries resizing or filtering files. Because of this django-imagefield does more work when images are added to the system instead of working around it later. (Django doesn’t do this on purpose because doing all this work up-front could be considered a DoS factor.)

In the last weeks I suddenly got recurring errors from saved files again, something which shouldn’t happen, but obviously did.

Django wants to read image dimensions when accessing or saving image files (by the way, always use height_field and width_field, otherwise Django will open and inspect image files even when you’re only loading Django models from the database…!) and it uses a smart and wonderful¹ hack to do this: It reads a few hundred bytes from the image file, instructs Pillow to inspect the file and if an exception happens it reads more bytes and tries again. This process relies on the exact type of exceptions raised internally though, and the release of Pillow 11 changed the types… for some file types only. Fun times.

The issue had already been reported as #33240 and is now tracked as #8530 on the Pillow issue tracker. Let’s see what happens. For now, django-imagefield declares itself to be incompatible with Pillow 11.0.0 so that this error cannot happen.

rspack and lightningcss shuffled CSS properties

rspack 1.0 started reordering CSS properties which of course lead to CSS properties overriding each other in the incorrect order. That was a fun one to debug. I tracked the issue down to the switch from the swc CSS minimizer to lightningcss and submitted a reproduction to the issue tracker. My rust knowledge wasn’t up to the task of attempting to submit a fix myself. Luckily, it has been fixed in the meantime.

rspack problems

I have another problem with rspack where I haven’t yet tracked down the issue. rspack produces a broken bundle starting with 1.0.0-beta.2 when compiling a particular project of mine. I have the suspicion that I have misconfigured some stuff related to import paths and yarn workspaces. I have no idea how anyone could have a complete understanding of these things…

Update: The problem was #8027, experiments.css is quite broken for now.

Bundlers are complex beasts, and I’m happy that I mostly can just use them.

Closing thoughts

Debugging is definitely a rewarding activity for me. I like tracking stuff down like this. Unfortunately, problems always tend to crop up when time is scarce already, but what can you do.

Releases

Quite a few releases, many of them verifying Python 3.13 and Django 5.1 support (if it hasn’t been added already in previous releases). The nicest part: If I remember correctly I didn’t have to change anything anywhere, everything just continues to work.

django-admin-ordering 0.19: I added support for automatically renumbering objects on page load. This is mostly useful if you already have existing data which isn’t ordered yet.
feincms3-data 0.7: Made sure that objects are dumped in a deterministic order when dumping. I wanted to compare JSON dumps by hand before and after a big data migration in a customer project and differently ordered dumps made the comparison impossible.
django-prose-editor 0.9: I updated the ProseMirror packages, and put the editor into read-only mode for <textarea disabled> elements.
feincms3-language-sites 0.4: Finally released the update containing the necessary hook to validate page trees and their unique paths before moving produces integrity errors. Error messages are nicer than internal server errors.
django-authlib 0.17.2: The value of the cookie which is used to save the URL where users should be redirected to after authentication wasn’t checked for validity when setting it, only when reading it. This meant that attackers could produce invalid header errors in application servers. No real security problem here when using authlib’s code.
feincms3 5.3: Minor update which mostly removes support for outdated Python and Django versions.
django-imagefield 0.20: See above.

wonderfully ugly ↩

Weeknotes (2024 week 43)

2024-10-23T12:00:00Z

Weeknotes (2024 week 43)

I had some much needed time off, so this post isn’t that long even though four weeks have passed since the last entry.

From webpack to rspack

I’ve been really happy with rspack lately. Converting webpack projects to rspack is straightforward since it mostly supports the same configuration, but it’s much much faster since it’s written in Rust. Rewriting things in Rust is a recurring theme, but in this case it really helps a lot. Building the frontend of a larger project of ours consisting of several admin tools and complete frontend implementations for different teaching materials only takes 10 seconds now instead of several minutes. That’s a big and relevant difference.

Newcomers should probably still either use rsbuild, Vite or maybe no bundler at all. Vanilla JS and browser support for ES modules is great. That being said, I like cache busting, optimized bundling and far-future expiry headers in production and hot module reloading in development a lot, so learning to work with a frontend bundler is definitely still worth it.

Dark mode and light mode

I have been switching themes in my preferred a few times per year in the past. The following ugly bit of vimscript helps switch me the theme each time the sun comes out when working outside:

let t:light = 0
function! FiatLux()
    if t:light == 0
        :set background=light
        let t:light = 1
    else
        :set background=dark
        let t:light = 0
    endif
endfunction
nnoremap <F12> :call FiatLux()<CR>

I’m using the Ptyxis terminal emulator currently, I haven’t investigated yet if there’s a shortcut to toggle dark and light mode for it as well. Using F10 to open the main menu works fine though, and using the mouse wouldn’t be painful either.

Helping out in the Django forum and the Discord

I have found some pleasure in helping out in the Django Forum and in the official Django Discord. I sometimes wonder why more people aren’t reading the Django source code when they hit something which looks like a bug or something which they do not understand. I find Django’s source code very readable and I have found many nuggets within it. I’d always recommend checking the documentation or maybe official help channels first, but the code is also out there and that fact should be taken advantage of.

Releases

django-content-editor 7.1: Fixed a bug where the ordering and region fields were handled incorrectly when they appear on one line in the fieldset. Also improved the presentation of inlines in unknown regions and clarified the meaning of the move to region dropdown. Also, released the improvements from previous patch releases as a new minor release because that’s what I should have been doing all along.
form-designer 0.27: A user has been bitten by slugify removing cyrillic characters because it only keeps ASCII characters around. Here’s the wontfixed bug in the Django issue tracker: #8391. I fixed the issue by removing the slugification (is that even a word?) when generating choices.

Weeknotes (2024 week 39)

2024-09-25T12:00:00Z

Weeknotes (2024 week 39)

CSS for Django forms

Not much going on in OSS land. I have been somewhat active in the official Django forum, discussing ways to add Python-level hooks to allow adding CSS classes around form fields and their labels. The discussion on the forum and on the pull request goes in the direction of allowing using custom BoundField classes per form or even per project (instead of only per field as is already possible today). This would allow overriding css_classes, e.g. to add a simple class="field". Together with :has() this would probably allow me to skip using custom HTML templates in 99% of all cases.

I have also been lurking in the Discord, but more to help and less to promote my packages and ideas :-)

Releases

django-user-messages 1.1: Added Django 5.0, 5.1 to the CI, and fixed the migrations to no longer mention index_together at all. It seems that squashing the migrations wasn’t sufficient, I also had to actually delete the old migrations.
blacknoise 1.1: Starlette’s FileResponse has gained support for the HTTP Range header, allowing me to remove my homegrown implementation from the package. The blacknoise implementation is now half as long as it was in 1.0.
django-fhadmin 2.3: No new features, only tweaks to the styling and behavior prompted by updates to Django’s admin interface.
django-cabinet 0.17: I have pruned the CI matrix and accepted a pull request adding a ru translation. I feel conflicted about that since I strongly believe that everything is political, but I don’t know if rejecting translations helps anyone.

django-content-editor now supports nested sections

2024-09-13T12:00:00Z

django-content-editor now supports nested sections

django-content-editor (and it’s ancestor FeinCMS) has been the Django admin extension for editing content consisting of reusable blocks since 2009. In the last years we have more and more often started automatically grouping related items, e.g. for rendering a sequence of images as a gallery. But, sometimes it’s nice to give editors more control. This has been possible by using blocks which open a subsection and blocks which close a subsection for a long time, but it hasn’t been friendly to content managers, especially when using nested sections.

The content editor now has first-class support for such nested sections. Here’s a screenshot showing the nesting:

Finally it’s possible to visually group blocks into sections, collapse those sections as once and drag and drop whole sections into their place instead of having to select the involved blocks individually.

The best part about it is that the content editor still supports all Django admin widgets, as long as those widgets have support for the Django administration interface’s inline form events! Moving DOM nodes around breaks attached JavaScript behaviors, but we do not actually move DOM nodes around after the initialization – instead, we use Flexbox ordering to visually reorder blocks. It’s a bit more work than using a ready-made sortable plugin, but – as mentioned – the prize is that we don’t break any other Django admin extensions.

Simple patterns

I previously already reacted to a blog post by Lincoln Loop here in my post My reaction to the block-driven CMS blog post.

The latest blog post, Solving the Messy Middle: a Simple Block Pattern for Wagtail CMS was interesting as well. It dives into the configuration of a Wagtail stream field which allows composing content out of reusable blocks of content (sounds familiar!). The result is saved in a JSON blob in the database with all the advantages and disadvantages that entails.

Now, django-content-editor is a worthy competitor when you do not want to add another interface to your website besides the user-facing frontend and the Django administration interface.

The example from the Lincoln Loop blog post can be replicated quite closely with django-content-editor by using sections. I’m using the django-json-schema-editor package for the section plugin since it easily allows adding more fields if some section type needs it.

Here’s an example model definition:

# Models
from content_editor.models import Region, create_plugin_base
from django_json_schema_editor.plugins import JSONPluginBase
from feincms3 import plugins

class Page(models.Model):
    # You have to define regions; each region gets a tab in the admin interface
    regions = [Region(key="content", title="Content")]

    # Additional fields for the page...

PagePlugin = create_plugin_base(Page)

class RichText(plugins.richtext.RichText, PagePlugin):
    pass

class Image(plugins.image.Image, PagePlugin):
    pass

class Section(JSONPluginBase, PagePlugin):
    pass

AccordionSection = Section.proxy(
    "accordion",
    schema={"type": "object", {"properties": {"title": {"type": "string"}}}},
)
CloseSection = Section.proxy(
    "close",
    schema={"type": "object", {"properties": {}}},
)

Here’s the corresponding admin definition:

# Admin
from content_editor.admin import ContentEditor
from django_json_schema_editor.plugins import JSONPluginInline
from feincms3 import plugins

@admin.register(models.Page)
class PageAdmin(ContentEditor):
    inlines = [
        plugins.richtext.RichTextInline.create(models.RichText),
        plugins.image.ImageInline.create(models.Image),
        JSONPluginInline.create(models.AccordionSection, sections=1),
        JSONPluginInline.create(models.CloseSection, sections=-1),
    ]

The somewhat cryptic sections= argument says how many levels of sections the individual blocks open or close.

To render the content including accordions I’d probably use a feincms3 renderer. At the time of writing the renderer definition for sections is a bit tricky.

from feincms3.renderer import RegionRenderer, render_in_context, template_renderer

class PageRenderer(RegionRenderer):
    def handle(self, plugins, context):
        plugins = deque(plugins)
        yield from self._handle(plugins, context)

    def _handle(self, plugins, context, *, in_section=False):
        while plugins:
            if isinstance(plugins[0], models.Section):
                section = plugins.popleft()
                if section.type == "close":
                    if in_section:
                        return
                    # Ignore close section plugins when not inside section
                    continue

                if section.type == "accordion":
                    yield render_in_context("accordion.html", {
                        "title": accordion.data["title"],
                        "content": self._handle(plugins, context, in_section=True),
                    })

            else:
                yield self.render_plugin(plugin, context)

renderer = PageRenderer()
renderer.register(models.RichText, template_renderer("plugins/richtext.html"))
renderer.register(models.Image, template_renderer("plugins/image.html"))
renderer.register(models.Section, "")

Closing thoughts

Sometimes, I think to myself, I’ll “just” write a “simple” blog post. I get what I deserve when using those forbidden words. This blog post is neither short or simple. That being said, the rendering code is a bit tricky, the rest is quite straightforward. The amount of code in django-content-editor and feincms3 is reasonable as well. Even though it may look like a lot you’ll still be running less code in production than when using comparable solutions built using Django.

Weeknotes (2024 week 37)

2024-09-11T12:00:00Z

Weeknotes (2024 week 37)

I have helped mentoring Aman Pandey who has worked all summer to add async support to django-debug-toolbar. Tim has released an alpha which contains all of the work up to a few days ago. Test it! Let’s find the breakages before the final release.

Dropping Python 3.9 from my projects

I have read Carlton’s post about the only green Python release and have started dropping Python 3.9 support from many of the packages I maintain. This is such a good point:

[…] I’m also thinking about it in terms of reducing the number of Python versions we support in CI. It feels like a lot of trees to support 5 full versions of Python for their entire life. 🌳

Releases

django-debug-toolbar 5.0.0a0: See above.
form-designer 0.26.2: The values of choice fields are now returned as-is when sending mails or exporting form submissions instead of only returning the slugified version.
django-authlib 0.17.1: The role-based permissions backend had a bug where it wouldn’t return all available permissions in all circumstances, leading to empty navigation sidebars in the Django administration. This has been fixed.
feincms3 5.2.3: Bugfix release, the page moving interface is no longer hidden by an expanded navigation sidebar. I almost always turn off the sidebar in my projects so I haven’t noticed this.
django-prose-editor 0.8.1: Contains the most recent ProseMirror updates and bugfixes.
django-content-editor 7.0.10: Now supports sections. Separate blog post coming up!

Weeknotes (2024 week 35)

2024-08-28T12:00:00Z

Weeknotes (2024 week 35)

Getting deep into htmx and django-template-partials

I have been skeptical about htmx for some time because basically everything the library does is straightforward to do myself with a few lines of JavaScript. I am a convert now because, really, adding a few HTML attributes is nicer than copy pasting a few lines of JavaScript. Feels good.

The combination of htmx with django-template-partials is great as well. I didn’t know I had been missing template partials until I started using them. Includes are still useful, but replacing some of them with partials makes working on the project much more enjoyable.

I haven’t yet had a use for django-htmx but I may yet surprise myself.

Releases

django-authlib 0.17: django-authlib bundles authlib.little_auth which offers an user model which uses the email address as the username. I have also introduced the concept of roles instead of permissions; now I have reorganized the user admin fieldset to hide user permissions altogether. Group permissions are still available as are roles. I’m personally convinced that user permissions were a mistake.
feincms3-forms 0.5: Allowed setting a maximum length for the bundled URL and email fields through the Django administration interface.
django-content-editor 7.0.7: Fixed a bug where plugins with several fieldsets weren’t collapsed completely.
django-imagefield 0.19.1: Allowed deactivating autogeneration of thumbnails completely through the IMAGEFIELD_AUTOGENERATE setting. This is very useful for batch processing. Also, documented all available settings.
django-prose-editor 0.8: Added support for translating the interface elements and for restricting the available heading levels in the UI.
form-designer 0.25: Fixed the type of the author field for the send-to-author processing action.
feincms3 5.2.2: Added support for embedding SRF play external content. They do not support oEmbed unfortunately.
feincms3-cookiecontrol 1.5.3: Added support for SRF play embeddings as well. The difference is that the feincms3-cookiecontrol embedding requires consent before embedding external content.

How I handle versioning

2024-08-20T12:00:00Z

How I handle versioning

I have been reading up on versioning methods a bit, and I noticed that I never shared my a bit unorthodox versioning method. I previously wrote about my rules for releasing open source software but skipped everything related to versioning.

I use something close to the ideas of semantic versioning but it’s not quite that. Note that the versioning scheme has nothing to do with production readiness, it’s more about communicating the state of the project.

0.0.x – Everything breaks

I don’t trust my choices a lot. Everything may radically change, and I may also abandon the project with no hard feelings.

0.1.0 – Changelogging

I generally start writing release notes or rather a Changelog, since writing full release notes is much more work. You absolutely have to test the software and I do not guarantee anything related to backwards compatibility – just that you’ll know about breaking changes when you read the CHANGELOG.

?.?.x – Bugfixes and pure additions

Strictly speaking, patch version increments are only for bugfixes. However, when I add new features which are purely additional to the package or if there’s no way (famous last words) that anything will break, I’ll upload a patch release.

Sometimes it might be better to increment the minor version, yes. But, being very lazy, I sometimes only want to add a line to the CHANGELOG instead of adding a new heading for the new release. Also, if I don’t quickly release new additions there’s a real danger that I’ll only come back to the project weeks or months later, and I don’t want anyone (myself included) waiting on these updates. Also, if I inadvertently introduced new bugs it’s better to fix them quickly instead of forgetting about it first and having to rediscover the buggy code later.

1.0.0 – I’m happy

I’m happy with the package and I commit to start using semver more properly. If I have to (or want to) break backwards compatibility you get a migration path, especially if I need it myself. Which is often the case since I’m maintaining dozens of Django-based websites and webapps, and I’m always dogfooding my stuff.

The pure additions case from the last section still applies, since slapping a 1.0 version on something doesn’t suddenly make me less lazy.

Outliers

FeinCMS uses a variant of calendar versioning; the version at the time of writing is 24.8.2 which means the second release in August 2024.

There are probably other outliers I forgot about.

Weeknotes (2024 week 33)

2024-08-14T12:00:00Z

Weeknotes (2024 week 33)

Partying

It’s summer, it’s hot, and it’s dance week. Lethargy is over, Jungle Street Groove is coming up. Good times.

Releases

django-json-schema-editor 0.1: I have finally left the alpha versioning. I’m still not committing to backwards compatibility, but I have started writing a CHANGELOG.
django-prose-editor 0.7.1: Thanks to Carlton’s pull request I have finally cleaned up the CSS somewhat and made overriding the styles more agreeable when using the editor outside the Django administration. The confusing active state of menubar buttons has also been rectified. Docs are now available on Read the Docs.
django-imagefield 0.19: django-imagefield can now be used with proxy models. Previously, thumbnails weren’t generated or deleted when saving proxy models because the signal handlers would only be called if the sender matches exactly. I have already debugged this before, but have forgotten about it again. The ticket is really old for this, and fixing it isn’t easy since it’s unclear what should happen (#9318).
django-canonical-domain 0.11: django-canonical-domain has gained support for excluding additional domains from the canonical domain redirect. django-canonical-domain is used to redirect users to HTTPS (optionally) and to a particular canonical domain (as the name says). But sometimes you have auxiliary domains, e.g. for an API service, which shouldn’t be redirected. The package can now be used in these scenarios as well.
FeinCMS 24.8.2: The venerable FeinCMS, now more than 15 years old. The thumbnailing support had a bug where it tried saving JPEGs using RGBA (which obviously doesn’t work). This has been fixed.

Weeknotes (2024 week 31)

2024-07-31T12:00:00Z

Weeknotes (2024 week 31)

I have missed almost two months of weeknotes. I’ve got some catching up to do. I have tried writing a larger piece on my thoughts about CMS, but with everything going on in my personal and work life I haven’t made much progress.

This weeknotes entry is me trying to get back into the groove of writing (and publishing!) regularly.

django-prose-editor

I have previously written about the ProseMirror-based editor for Django websites here. I have continued working on the project in the meantime. Apart from bugfixes the big new feature is the support for showing typographic characters. For now the editor supports showing non-breaking spaces and soft hyphens. The project seems to get a little more interest after the deprecation of django-ckeditor has become more well known and the project has even received a contribution by someone else. It’s always a lovely moment when this happens.

django-json-schema-editor

Still alpha. Updated the vendorized JSON editor and fixed the integration into Django to not throw errors with the newer version.

Foreign key fields now support describing the referenced value similar to the raw ID fields functionality. Added optional support for using "format": "prose" to use the django-prose-editor to edit individual fields. JSON plugins for the content editor are now downcasted into their proxy models automatically. This is especially useful with the feincms3 changes mentioned below. (You do not have to use either django-content-editor or feincms3 to use this package!)

The following screenshot shows the prose editor integration; the mentioned foreign key field description isn’t visible yet here.

Traduire

Traduire (french for «translate») is a web-based platform for editing gettext translations.

It is intended as a replacement for Transifex, Weblate and comparable products. It is geared towards small teams or agencies which want to allow their customers and their less technical team members to update translations.

Traduire profits from the great work done on django-rosetta. I would still be using Rosetta if it would work when used with a container orchestator such as Kubernetes. Since all application storage is ephemeral that doesn’t work, translation editing and deployment have to be separated.

It is built using Django and relies on polib to do the heavy lifting.

This is a project which might also be interesting for others. I would especially appreciate it if someone could contribute an easier way to get it up and running, e.g. using a Docker Compose configuration or something. I am using Kubernetes and GitOps to host it, but that’s not straightforward at all. Really, all that’s needed to run it is a Django host with any database which is supported by Django. I prefer using PostgreSQL because I have it, but sqlite etc. work just as well.

Releases since the second week of June

django-translated-fields 0.13: Nothing much except for CI and pre-commit updates. The implementation continues to be rock-solid and basically unchanged.
django-content-editor 7.0.6: Tweaks and fixes to the new interface. Added better scrolling behavior when dragging content around. The editor now also supports colorized icons which helps quickly understanding the structure of some content when there are many plugins.
blacknoise 1.0.2: Fixed a few bugs in the blacknoise.compress utility and started running the testsuite on GitHub actions. whitenoise has been friendly-forked as ServeStatic and I’m definitely having a close look at this project as well, but blacknoise is simple and works well, so I’m not convinced that switching back to the much larger project (in terms of amounts of code) is an improvement now.
django-authlib: Minor bugfixes.
feincms3: Allowed registering plugin models with the renderer which aren’t supposed to be fetched from the database. This is especially useful when used together with JSON plugins, where the individual JSON plugins are created as proxies for the underlying Django model and fetched all at once. Disabled the version check on our CKEditor plugin. Still, really stop using CKEditor 4 if you want to use maintained software.
FeinCMS 24.7.1: Small bugfixes, and made the Read the Docs build work correctly.
django-debug-toolbar 4.4.x: The toolbar continues to be a nice project to work on. Fixed a few edge cases in the new alerts panel.
django-admin-ordering 0.18.2: The value of ordering_field now has additional sanity checks.
django-cabinet 0.16: cabinet now supports exporting a folder as a ZIP file while preserving the structure you see in the CMS instead of the structure on the file system. The inline upload form has been dropped from the CabinetForeignKey widget because the folder dropdown slowed down the page a lot when used on a site with many folders. Using the raw ID fields popup isn’t that bad.
django-prose-editor 0.6.2: See above.
django-json-schema-editor 0.0.28: See above.

Weeknotes (2024 week 23)

2024-06-07T12:00:00Z

Weeknotes (2024 week 23)

Switching everything from pip to uv

Enough said. I’m always astonished how fast computers can be.

Releases

django-admin-ordering 0.18: Added a database index to the ordering field since we’re always sorting by it.
django-prose-editor 0.4: Dropped the jQuery dependency making it possible to use the editor outside the Django administration interface without annoying JavaScript errors. Allowed additional heading levels and moved the block type buttons into a popover.
django-debug-toolbar 4.4.2: I enjoy working on this important piece of software very much.
django-email-hosts 0.2.1: Added a command analogous to ./manage.py sendtestemail so that it’s possible to easily test the different configured email backends.
feincms3 5.0: I completely reworked the move node action; previously it opened a new page where you could see all possible targets; now you can cut a page and paste it somewhere else. The advantages of the new interface is that you don’t leave the changelist and can still profit from all its features while moving pages around.
feincms3-sites 0.21: A new release taking advantage of a new hook in feincms3 7.0 so that the new moving interface works.
django-authlib 0.16.5: authlib now shows a welcome message when authenticating using admin OAuth2. It’s nice and helps with debugging strange authentication failures.
django-content-editor 7.0: I reworked the UI. The sidebar is gone, instead there are nice buttons in the place where you can add new plugins; the plugins appear in a nice grid instead of a list, which looks much better once you have more than just a few plugin types available. Also, plugin type icons are now shown in the plugin forms. I think it looks much better than before.
feincms3-cookiecontrol 1.5.2: I didn’t contribute anything to this release which is also a nice experience for a change. The Google consent mode integration has been improved and simplified.
django-json-schema-editor 0.0.22: Various small-ish improvements. I should really start using higher version numbers, but not having to commit to anything also feels great. That being said, the editor is in active use in several projects, so maybe I’m deceiving myself.

Workbench: Coffee time!

2024-05-24T12:00:00Z

Workbench: Coffee time!

I have written about the Workbench agency software a few weeks back.

Back when we were using Slack at Feinheit we used a Donut bot to generate randomized invites for a coffee break. We lost that bot when we switched to Discord. I searched some time for an equivalent bot for Discord but couldn’t find any, so like any self-respecting nerd, break lover and NIH-sufferer I reimplemented the functionality as a part of our agency software.

The first version generated totally random pairings without any history; I thought that maybe this would be good enough, but of course I received an invite for a coffee with the exact same person in the first two invitations. Even though I did enjoy the break both times this motivated me to put some effort into a better solution :-)

The result of this work is the coffee_invites() function

There are two main parts to the generator:

It randomly generates twenty pairings from all participants; it prefers groups of two except for the last group which may contain three people.
It loads old pairings from the database and gives higher penalties to equal pairings depending on the time which has passed since the last time.

So, it’s not mathematically perfect, but all imbalances will cancel out over a long time. I experimented with the discounting factor and the number of random pairings it generates upfront. The values in the code seem to work fine.

Sometimes it’s simple stuff like this which gets me going and reignites the love for programming. Like a small puzzle.

Weeknotes (2024 week 21)

2024-05-22T12:00:00Z

Weeknotes (2024 week 21)

There have been times when work has been more enjoyable than in the last few weeks. It feels more stressful than at other times, and this mostly has to do with particular projects. I hope I’ll be able to move on soon.

blacknoise

I have released blacknoise 1.0. It’s an ASGI app for static file serving inspired by whitenoise.

The 1.0 version number is only a big step in versioning terms, not much has happened with the code. It’s a tiny little well working piece of software which has been running in production for some time without any hickups. The biggest recent change is that I have parallelized the gzip and brotli compression step; this makes building images using whitenoise more painful because there the wait is really really long sometimes. A pull request fixing this exists, but it hasn’t moved forwards in months.

I have written a longer post about it earlier this year here.

Releases

feincms3-cookiecontrol 1.5: Code golfing. Added backwards compatibility with old Django versions so that I can use it for old projects. Also includes optional support for Google consent management.
django-fast-export 0.1.1: This is basically a repackaging of the streaming CSV view from Django’s documentation as a reusable class. I have switched to using an iterator so that I can export even larger datasets.
django-json-schema-editor 0.0.18: Still alpha versioned but used in production in various projects. I should really release an 1.0 version, but there are no integration tests at all. Mainly visual tweaks in this update.
django-content-editor 6.5: Better handling of templates and regions when a particular editor instance only shows a subset of configured templates. Disallowed adding plugins when in an unknown region. It’s funny how many edge cases exist in software as old as this.
blacknoise 1.0: See above.
html-sanitizer 2.4.4: Fixed edge cases with whitespace handling when merging elements.
feincms3-data 0.6.1: Allowed ./manage.py f3loaddata - to load JSON data from stdin.

Weeknotes (2024 week 18)

2024-05-03T12:00:00Z

Weeknotes (2024 week 18)

Google Summer of Code has begun

We have a student helping out with adding async support to the Django Debug Toolbar. It’s great that someone can spend some concentrated time to work on this. Tim and others have done all the necessary preparation work, I’m only helping from the sidelines so don’t thank me.

Bike to Work

Two teams from my company are participating in the Bike to Work Challenge 2024. It’s what I do anyway (if I’m not working from home) but maybe it helps build others some motivation to get on the bicycle once more. Public transports in the city where I live are great but I’ll always take the bike when I can. I also went on my first mountain bike ride in a few months yesterday, good fun.

JSON blobs and referential integrity

The django-json-schema-editor has gained support for referencing Django models. Here’s an example schema excerpt:

{
    ...
    "articles": {
        "type": "array",
        "format": "table",
        "title": _("articles"),
        "minItems": 1,
        "maxItems": 3,
        "items": {
            "type": "string",
            "title": _("article"),
            "format": "foreign_key",
            "options": {
                "url": "/admin/articles/article/?_popup=1&_to_field=id",
            },
        },
    },
    ...
}

The ID field is stringly typed; using an integer directly wouldn’t work because the empty string isn’t a valid integer.

The problem with referencing models in this way is that there’s no way to know if the referenced object is still around or not, or even to protect it against deletion. The bundled django-content-editor JSONPlugin now supports automatically generating a ManyToManyField with a through model which protects articles from deletion as long as they are referenced from a JSONPlugin instance. The register_reference line creates the mentioned model with an on_delete=models.PROTECT foreign key to articles and a post_save handler which updates said references.

from django_json_schema_editor.plugins import JSONPluginBase, register_reference
from articles.models import Article

class JSONPlugin(JSONPluginBase, ...):
    pass

register_reference(JSONPlugin, "articles", Article)

Releases since the beginning of April

django-json-schema-editor 0.0.14: See above. Also, some styling work and a patch update to the vendorized json-editor.
django-content-editor 6.4.6: Many small stylistic fixes. The target indicator when dragging plugins is now also shown when plugins are collapsed. It’s now possible to directly drag a plugin to the end, and not just to the second to last position.
django-prose-editor 0.3.4: Switched to the nh3 sanitizer because it’s faster and because ProseMirror never emits HTML which has to be cleaned up first. Stopped generating menu items for nodes and marks which aren’t in the schema. Added the possibility to reduce the functionality per editor instance. Small tweaks and fixes.
django-tree-queries 0.19: Added support for pre-filtering the tree (much more efficient when only querying a part of the tree). Added support for adding additional fields to the CTE so that you can collect values from ancestors for other fields than the default fields too.
FeinCMS 24.4.2: Added support for webp images. Fixed a few of the admin list filters to work with Django 5.
django-cabinet 0.14.3: Fixed the support for the extra_context argument to our changelist_view implementation.

Workbench, the Django-based agency software

2024-04-24T12:00:00Z

Workbench, the Django-based agency software

I get the impression that there’s a lot of interesting but unknown software in Django land. I don’t know if there’s any interest in some of the packages I have been working on; if not this blog post is for myself only.

(Hi)story time

As people may know I work at Feinheit, an agency which specializes in digital communication services for SMEs, campaigns for referendums, and website and webapp development. At the time of writing we are a team of about 20-25 communication experts, graphic designers, programmers and project managers.

We have many different clients and are working on many different projects at the same time and are billing by the hour¹. Last year my own work has been billed to more than 50 different customers. In the early days we used a shared file server with spreadsheet files to track our working hours. Luckily we didn’t often overwrite the edits others made but that was definitely something which happened from time to time.

We knew of another agency who had the same problems and used a FileMaker-based software. Their solution had several problems, among them the fact that it became hard to evolve and that it got slower and slower as more and more data was entered into it over the years. They had the accounting know how and we had the software engineering know how so we wrote a webapp based on the Django framework. As always, it was much more work than the initial estimate, but if we as programmers didn’t underestimate the effort needed we wouldn’t have started many of the great projects we’re now getting much value and/or enjoyment from, hopefully both. The product of that work was Metronom. The first release happened a little bit later than harvest but it already came with full time tracking including an annual working time calculator, absence management, offers, invoices including PDF generation etc, so it was quite a bit more versatile while still being easier to use than “real” business software solutions.

I personally was of the opinion that the product was good enough to try selling it, but for a variety of reasons (which I don’t want to go into here) this never happened and we decided that we didn’t want to be involved anymore.

However, this meant that we were dead-end street with a software that didn’t belong to us anymore, which wasn’t evolving to our changing requirements. I also didn’t enjoy working on it anymore. Over the years I have tried replacing it several times but that never came to pass until some time after the introduction of Holacracy at our company. I noticed that I didn’t have to persuade everyone but that I, as the responsible person for this particular decision, could “just”² move ahead with a broad interpretation of the purpose and accountabilities of one of my roles.

Workbench

Workbench is the product of a few long nights of hacking. The project was started as an experiment in 2015 and was used for sending invoices but that wasn’t really the intended purpose. After long periods of lying dormant I have brought the project to a good enough state and switched Feinheit away from Metronom in 2019.

I have thought long and hard about switching to one of the off-the-shelf products and it could very well be that one of them would work well for us. Also, we wouldn’t have to pay (in form of working hours) for the maintenance and for enhancements ourselves. On the other hand, we can use a tool which is tailored to our needs. Is it worth the effort? That’s always hard to answer. The tool certainly works well for the few companies which are using it right now, so there’s no reason to agonize over that.

At the time of writing, Workbench offers projects and services, offers and invoices incl. PDF generation, recurring invoices, an address book, a logbook for rendered services, annual working time reports, an acquisition funnel, a stupid project planning and resource management tool and various reports. It has not only replaced Metronom but also a selection of SaaS (for example Pipedrive and TeamGantt) we were using previously.

The whole thing is open source because I don’t want to try making agency software into a business anymore, I only want to solve the problems we have. That being said, if someone else finds it useful then that’s certainly alright as well.

The license has been MIT for the first few years but I have switched to the GPL because I wanted to integrate the excellent qrbill module which is also licensed under the GPL. As I wrote elsewhere, I have released almost everything under the GPL in my first few open source years but have switched to BSD/MIT later when starting to work mainly with Python and Django because I thought that this license is a better fit³ for the ecosystem. That being said, for a product such as this the GPL is certainly an excellent fit.

Final words?

There are a few things I could write about to make this into a series. I’m putting a few ideas here, not as an announcement, just as a reminder for myself.

Better time tracking
Patterns for creating recurring invoices
Feature switches and configurability
Coffee time

Rather, in six minute increments. It’s even worse. ↩
Of course it’s never that simple, because the responsibility is a lot. The important point is: It would have been a lot of work anyways, the big difference is that it was sufficient to get consent from people; no consensus required. ↩
That’s not meant as a criticism in any way! ↩

Building forms with the Django admin

2024-04-12T12:00:00Z

Building forms with the Django admin

The title of this post was shamelessly copied from Jeff Triplett’s post on Mastodon.

Why?

Many websites need a simple way of embedding forms, for example as a contact form or for simple surveys to collect some data or inputs from visitors. Django’s forms library makes building such forms straightforward but changing those forms requires programming skills and programmer time. Both of them may not be readily available. More importantly, sometimes it’s just nice to give more tools to web publishers.

The simple way to build something like this is to use a form builder such as Google Forms, Typeform, Paperform or anything of the sort. Those options work nicely. The downsides are that embedded forms using those services load slowly, look differently, cost a lot or collect a lot of data on users, or all of those options. Because of that there’s still a place for building such functionality locally.

If I wanted to use PHP and WordPress I could just use WPForms and call it a day. Since I do not actually want that this blog post is a bit longer.

The early days: form-designer

One of the first Django-based third party apps I published was the form-designer. The first version was uploaded to PyPI in 2012 but it had already been used in production for more than two years at that point in time. I had used Git submodules for the deployment back then, before switching to Python virtualenvs some time later (and never looking back!)

The form-designer is still maintained actively. Because of Django’s stability and because of the fact that the app doesn’t do all that much it doesn’t require much development at all.

The form designer supports a selection of standard HTML5 input fields out of the box and also has an optional django-recaptcha integration. All fields support some basic configuration such as setting a title, a help text, marking the field as required etc. Submissions can be sent to a configurable email address and can be saved in the database and later exported as an XLSX file. It’s also possible to define your own actions.

More flexibility needed: feincms3-forms

A few years back I mentioned feincms3-forms in a weeknotes entry. The reasons why form-designer wasn’t sufficient for a project back then are outlined in the blog post:

feincms3-forms – A new forms builder for the Django admin interface

For a current project we needed a forms builder with the following constraints:

Simple fields (text, email, checkboxes, dropdowns etc.)

Custom validation and processing logic

It should be possible to add other content, e.g. headings and explanations between form fields

The form_designer fulfilled a few of these requirements but not all. It still works well but I wanted a forms builder based on django-content-editor for a long time already. Also, I really like the feincms3 pattern where the third party app only provides abstract models. Yeah, it is much more work to start with but the flexibility and configurability is worth it – especially since it’s possible to write straightforward code to handle special cases[^2] instead of configuring even more settings.

The humble beginnings are here in the feincms3-forms repository. The test suite already shows how things work together but as of now no documentation exists and no release has been made yet. I hope it will be ready for a first beta release in the next few weeks 😄

Since then I have used feincms3-forms more often than form-designer, for building simple forms and also to build multi-step form wizards with custom fields, custom validation, configurable steps etc. The README now actually explains why the project exists and how it could be used.

It still doesn’t come close to WPForms in terms of included functionality; a big feature which is missing is conditional logic because I haven’t yet had a use for it.

The feincms3-forms forms support all types of content between form fields (basically everything django-content-editor supports). Plugins for form fields are more flexible and can add as many input fields to the form as they want, you’re not restricted to single values or single input fields.

Packages

Weeknotes (2024 week 14)

2024-04-06T12:00:00Z

Weeknotes (2024 week 14)

I’m having a bit of a slow week with the easter weekend and a wisdom tooth extraction. I’m recovering quite quickly it seems and I’m glad about it.

This weeknotes entry is short and quick. I’m trying to get back into the habit of writing them after a mediocre start this year.

20th Anniversary Celebration of Young Greens Switzerland

I have attended the celebration of Young Greens Switzerland. I have been a founding member and have been active for close to ten years. A lot of time has passed since then. It has been great to reminisce about old times with friends and, more importantly, to see how the torch is carried on.

Releases

blacknoise 0.0.5: blacknoise is an ASGI app for static file serving inspired by whitenoise. It only supports a very limited subset of whitenoise’s functionality, but it supports async.
html-sanitizer 2.4.1: The lxml library moved the HTML cleaner into its own package, lxml-html-clean; this release adds support for that. I didn’t know that the HTML cleaner is viewed as being problematic by the lxml maintainers. I’m having another look at nh3 and will maybe switch html-sanitizer’s guts from lxml to nh3 in the future.
django-tree-queries 0.18: django-tree-queries now supports ordering siblings by multiple fields and even allows descending orderings.
django-cabinet 0.14.2: This release fixes the CKEditor 4 filebrowser popup when using Django 5 or better.