Poor Performance for Form Rendering In Django 1.11

[John]

My site <https://www.morphmarket.com/> has a navbar with an advanced search
widget (beside the search field), which renders on every page. For each
request, a context_processor creates the form so it can be available on
that page in the navbar. This form has about a dozen selects with a total
of several hundred options. Most of those options are for the currency and
country selects, along with about 80 other options. There is an even larger
list for "stores" but it is loaded via AJAX so it should not be a factor
here.

Performance was fine on Django 1.8, but after upgrading to 1.11 I noticed
with NewRelic that over 500 ms are now being used on my most frequent
request between the following:

- Render/django/forms/widgets/select_option.html
- Render/django/forms/widgets/select.html
- Render/django/forms/widgets/attrs.html

[image: xDE2A.png]

This seems to be related to 1.11's change to Template-based Widget Rendering
<https://code.djangoproject.com/ticket/15667> (docs
<https://docs.djangoproject.com/en/1.11/ref/forms/renderers/#djangotemplates>),
however the only pages I could find talking about related problems were
about Django Toolbar which I do not run in production.

I am and already using the Cached Template Loader (which is now default),
however I don't know if this helps here. I cannot easily cache this form
because as you can see in the code, I set a number of defaults based on the
request.

Why is my form suffering so badly from this change? Eliminating two of the
bigger selects helps, but surely several hundred options should not take
this long to render so it seems to me there is an underlying problem that
the quantity is merely exacerbating.

Here are links to to code for the full form and html. (I will include
snippets in the question later when we identify the problem, for future
readers).

- Search Form
<https://gist.github.com/jplehmann/b75379c12a37a6064f1face208d7dc88#file-search-py-L223>
- Search HTML
<https://gist.github.com/jplehmann/44872ab2c02da89080e9a199aed349ae>
- Live Site <http://www.morphmarket.com/>

Thank you in advance for your help!

[Simon Charette]

Hello John,

Did you try switching to the Jinja2 form renderer[0] to see if it helps?
I've not
tried it myself but I heard it makes a significant difference.

Cheers,
Simon

[0] https://docs.djangoproject.com/en/2.1/ref/forms/renderers/#jinja2

Post by John
My site <https://www.morphmarket.com/> has a navbar with an advanced
search widget (beside the search field), which renders on every page. For
each request, a context_processor creates the form so it can be available
on that page in the navbar. This form has about a dozen selects with a
total of several hundred options. Most of those options are for the
currency and country selects, along with about 80 other options. There is
an even larger list for "stores" but it is loaded via AJAX so it should not
be a factor here.
Performance was fine on Django 1.8, but after upgrading to 1.11 I noticed
with NewRelic that over 500 ms are now being used on my most frequent
- Render/django/forms/widgets/select_option.html
- Render/django/forms/widgets/select.html
- Render/django/forms/widgets/attrs.html
[image: xDE2A.png]
This seems to be related to 1.11's change to Template-based Widget
Rendering <https://code.djangoproject.com/ticket/15667> (docs
<https://docs.djangoproject.com/en/1.11/ref/forms/renderers/#djangotemplates>),
however the only pages I could find talking about related problems were
about Django Toolbar which I do not run in production.
I am and already using the Cached Template Loader (which is now default),
however I don't know if this helps here. I cannot easily cache this form
because as you can see in the code, I set a number of defaults based on the
request.
Why is my form suffering so badly from this change? Eliminating two of the
bigger selects helps, but surely several hundred options should not take
this long to render so it seems to me there is an underlying problem that
the quantity is merely exacerbating.
Here are links to to code for the full form and html. (I will include
snippets in the question later when we identify the problem, for future
readers).
- Search Form
<https://gist.github.com/jplehmann/b75379c12a37a6064f1face208d7dc88#file-search-py-L223>
- Search HTML
<https://gist.github.com/jplehmann/44872ab2c02da89080e9a199aed349ae>
- Live Site <http://www.morphmarket.com/>
Thank you in advance for your help!

[John Lehmann]

Hi James,

Thank you for the input!

So my takeaway from what you are saying is that no one is running a
production site with a Django select field to render a country or currency
option (e.g., django-countries or django-money), because either of these
would have well over a hundred entries (and my form happens to have both).
Instead they would be loading these options via AJAX or using a different
widget all together like an auto-complete. (I get it that a long list is
not the best UX but that's a bit of a different discussion).

It seems like a regression of this magnitude should be called out in the
Django documentation for the default rendering option? I'm also surprised
neither of those mainstream libraries would mention something about this.
Adding 500+ms isn't in the realm of "tweaking" performance, for most
serious sites it would be a complete fail. So I guess I'm just a little
skeptical that the real answer here is that I shouldn't expect this to work
anyway. I feel like this kind of change would have affected a lot of other
folks in similar significant fashion, which makes me think there is
something else going on (some other kind of boneheaded thing I may be
doing).

thanks,
John

I am still hoping however for someone to explain to me why the default
renderer cannot handle my use case, such as that a few hundred inputs is
too many, or that I am doing something else improperly. Surely this kind
of a change would not be made to the framework that would cause such an
unacceptable performance (500+ms added). Kind of disappointing to spend a
whole week upgrading only to end up wanting to downgrade again.

The Django template language is not really optimized for speed of
rendering. Even with a caching loader, rendering a template hundreds of
times will be an expensive enough operation to show up in your profiling.
The solutions are to use something faster (Jinja) if you absolutely must
be rendering hundreds of things, or find a way to make the form more
efficient (generally, when a form has hundreds of options to display in a
select, that's a sign of issues with the design of the form).
--
You received this message because you are subscribed to a topic in the
Google Groups "Django users" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/django-users/uhe7ExBJoxg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/CAL13Cg9amLPan9-pj%2BHpHc7vns6_5EJcvNHvF0VPakf651%3DT1g%40mail.gmail.com
<https://groups.google.com/d/msgid/django-users/CAL13Cg9amLPan9-pj%2BHpHc7vns6_5EJcvNHvF0VPakf651%3DT1g%40mail.gmail.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.

[John]

Yes, the form has about a dozen selects, with two in particular being
larger - the country and the currency fields - that probably have a couple
hundred options each. I have since removed those two fields which leaves
about 10 selects with about 80 options total. While these numbers
fluctuate, the 12 field form was taking a little over 750ms, and the new
form with 10 fields totals 375ms. So the two larger fields accounted for
about half the time, but even the reduced form is still a major problem.

Regarding the reduced version, I cannot say how much the performance issue
relates to having 10 selects, or to have 80 options. But it seems like
375ms is an unreasonable amount of time for this amount of work. (Just as
it seems unreasonable to spend 400ms drawing those other two fields).

You can see the form by visiting the site <https://www.morphmarket.com/us/>
and clicking on "options" next to the search bar.

I'm sorry and didn't mean to sound combative, was trying to point out that
if this is the performance we expect for the default renderer, then I would
have expected to see a lot more people reacting to 1.11. Which is why I
believe something else must be going on.




[image: Screen Shot 2018-12-10 at 8.59.32 AM.png]

Post by John Lehmann
So my takeaway from what you are saying is that no one is running a
production site with a Django select field to render a country or currency
option (e.g., django-countries or django-money), because either of these
would have well over a hundred entries (and my form happens to have both).
Instead they would be loading these options via AJAX or using a different
widget all together like an auto-complete. (I get it that a long list is
not the best UX but that's a bit of a different discussion).

Have you profiled a country select on its own?
Your original post suggested you had multiple large (hundreds of options)
selects in a single form that you were rendering. Whether any specific
individual select in the form is the sole culprit, or whether it's the
combination of them, is something you still haven't pinned down, and it's a
bit combative to jump to "nobody must have a country select" from that.
In general, a select with a large set of options is going to be slower to
render. It may turn out that this has nothing whatsoever to do with your
problem, but we don't have enough information to precisely diagnose your
problem; all we can do here is give you general advice like "avoid
rendering a template hundreds of times if you can, or use a
faster-rendering template option if you can't".