Discussion:
Odd problem: some database updates do not appear on other pages until server restart
bobhaugen
2016-08-18 16:54:15 UTC
Permalink
I'm running django 1.8.14, I have an odd problem, that I have reproduced
both locally using runserver and sqlite, and also using Apache with
Postgres.

I create a new model instance on one page, and go to another page where
that same object should appear in a form selection list. It does not.

After I restart the server (both with runserver and Apache) it appears.

I also see other places where the same odd behavior with template variables
showing foreign key relationships, like {{ object.foreign_key_field }}.

This is not on all pages, just some. It happens in both of the situations I
mentioned above (form selection choices and template variables for foreign
key fields).

I grepped for cache. None in my code, a lot in site-packages, but I have no
idea where to look and what to look for.

I would gratefully follow any clues.

Thanks.
--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/e3d375e3-dffd-43f7-a7ea-409a15e5381c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Tim Graham
2016-08-18 18:34:29 UTC
Permalink
I'd guess you're doing a query for the form field's choices at the module
level which will be executed once and cached for as long as the server
runs. See if
https://docs.djangoproject.com/en/stable/ref/forms/fields/#fields-which-handle-relationships
helps, otherwise please post the code for the form in question.
Post by bobhaugen
I'm running django 1.8.14, I have an odd problem, that I have reproduced
both locally using runserver and sqlite, and also using Apache with
Postgres.
I create a new model instance on one page, and go to another page where
that same object should appear in a form selection list. It does not.
After I restart the server (both with runserver and Apache) it appears.
I also see other places where the same odd behavior with template
variables showing foreign key relationships, like {{
object.foreign_key_field }}.
This is not on all pages, just some. It happens in both of the situations
I mentioned above (form selection choices and template variables for
foreign key fields).
I grepped for cache. None in my code, a lot in site-packages, but I have
no idea where to look and what to look for.
I would gratefully follow any clues.
Thanks.
--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/44c72a65-4082-408a-adf2-f1662e768563%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
bobhaugen
2016-08-18 18:57:40 UTC
Permalink
Post by Tim Graham
I'd guess you're doing a query for the form field's choices at the module
level which will be executed once and cached for as long as the server
runs. See if
https://docs.djangoproject.com/en/stable/ref/forms/fields/#fields-which-handle-relationships
helps, otherwise please post the code for the form in question.
Ooooohhh, Tim! You might just have nailed it! Yes I am.
Here's the relevant code for the form field:

```
from_agent = forms.ModelChoiceField(
required=False,
queryset=EconomicAgent.objects.with_user(),
```

with_user() is a method of
class AgentManager(models.Manager)


executed once and cached for as long as the server runs.
Is that behavior documented anywhere? Regardless, got any ideas how to
avoid it?

But thank you very much for the likely suspect.
--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/b55a8386-fffa-44ca-8543-f512cc4fbc6a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
bobhaugen
2016-08-18 19:11:21 UTC
Permalink
Looks like it works if I "specify queryset=None when declaring the form
field and then populate the queryset in the form’s__init__() method:"

Does that make sense to you?
--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/47647de8-b119-4235-b862-ea2170877246%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
bobhaugen
2016-08-18 19:14:17 UTC
Permalink
Also, how pervasive is this behavior? Does it affect all querysets
generated by model methods? I do that all over the place. This could be bug
heaven!
--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/16924d52-c9e7-4666-b80d-7baaa49e59e7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Sergiy Khohlov
2016-08-18 19:17:39 UTC
Permalink
Hello,
This is trivial mistake. Use form.__init__ if you would like to change it
dynamically
Post by bobhaugen
Also, how pervasive is this behavior? Does it affect all querysets
generated by model methods? I do that all over the place. This could be bug
heaven!
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/
msgid/django-users/16924d52-c9e7-4666-b80d-7baaa49e59e7%40googlegroups.com
<https://groups.google.com/d/msgid/django-users/16924d52-c9e7-4666-b80d-7baaa49e59e7%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CADTRxJOrnvSacsL%2Bseok_PP-QnGdFo%2B-%3DMNU-ESd_ugpzSdUyQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
bobhaugen
2016-08-18 19:23:30 UTC
Permalink
Yes, that's what I did. It worked for the form field.

But, still, how pervasive is this behavior? (That was the question in the
message you answered).
Post by Sergiy Khohlov
Hello,
This is trivial mistake. Use form.__init__ if you would like to change it
dynamically
Post by bobhaugen
Also, how pervasive is this behavior? Does it affect all querysets
generated by model methods? I do that all over the place. This could be bug
heaven!
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an
<javascript:>.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/16924d52-c9e7-4666-b80d-7baaa49e59e7%40googlegroups.com
<https://groups.google.com/d/msgid/django-users/16924d52-c9e7-4666-b80d-7baaa49e59e7%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/6b847f04-6112-4512-848c-57eb95ad8185%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Michal Petrucha
2016-08-19 10:19:15 UTC
Permalink
Post by bobhaugen
Post by Tim Graham
I'd guess you're doing a query for the form field's choices at the module
level which will be executed once and cached for as long as the server
runs. See if
https://docs.djangoproject.com/en/stable/ref/forms/fields/#fields-which-handle-relationships
helps, otherwise please post the code for the form in question.
Ooooohhh, Tim! You might just have nailed it! Yes I am.
```
from_agent = forms.ModelChoiceField(
required=False,
queryset=EconomicAgent.objects.with_user(),
```
with_user() is a method of
class AgentManager(models.Manager)
Could you show us the code of with_user? Maybe it does not return an
unevaluated queryset?

Cheers,

Michal
--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/20160819101915.GI27882%40koniiiik.org.
For more options, visit https://groups.google.com/d/optout.
bobhaugen
2016-08-19 12:02:39 UTC
Permalink
Post by Michal Petrucha
Could you show us the code of with_user? Maybe it does not return an
unevaluated queryset?
def with_user(self):
all_agents = EconomicAgent.objects.all()
ua_ids = []
for agent in all_agents:
if agent.users.all():
ua_ids.append(agent.id)
return EconomicAgent.objects.filter(id__in=ua_ids)

Moving the call to with_user to form.__init__ solved the problem in the
form ModelChoiceField.

These questions remain unanswered, although I intend to do a bunch more
testing:

1. How pervasive is this problem? Does it affect template variables like
{{ object.foreign_key_method }} where the foreign_key_method returns a
queryset?
2. Is this behavior clearly documented anywhere?
--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/e21e3ec0-dfba-4d0f-a4f0-828a552ee0af%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
ludovic coues
2016-08-19 12:36:27 UTC
Permalink
You were calling the method in the class definition. The class is
"defined" when the module is imported. That's why things where
"cached". Module is imported only once.
The init method on the other hand is called every time an instance of
the class is created. I believe that method will be called in template
everytime the template is rendered.

I don't think this is clearly documented, but that's how python work.
there is nothing magic happening.
Post by bobhaugen
Post by Michal Petrucha
Could you show us the code of with_user? Maybe it does not return an
unevaluated queryset?
all_agents = EconomicAgent.objects.all()
ua_ids = []
ua_ids.append(agent.id)
return EconomicAgent.objects.filter(id__in=ua_ids)
Moving the call to with_user to form.__init__ solved the problem in the form
ModelChoiceField.
These questions remain unanswered, although I intend to do a bunch more
How pervasive is this problem? Does it affect template variables like {{
object.foreign_key_method }} where the foreign_key_method returns a
queryset?
Is this behavior clearly documented anywhere?
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/e21e3ec0-dfba-4d0f-a4f0-828a552ee0af%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
Cordialement, Coues Ludovic
+336 148 743 42
--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAEuG%2BTZibykPMT6TiKHQpLvcAxPsTMu2YTXpDsHP5t4XWU_WWw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Michal Petrucha
2016-08-19 12:38:02 UTC
Permalink
Post by bobhaugen
Post by Michal Petrucha
Could you show us the code of with_user? Maybe it does not return an
unevaluated queryset?
all_agents = EconomicAgent.objects.all()
ua_ids = []
ua_ids.append(agent.id)
return EconomicAgent.objects.filter(id__in=ua_ids)
Moving the call to with_user to form.__init__ solved the problem in the
form ModelChoiceField.
Thanks for sharing the code. The problem is that the ``with_user``
method, which was called during import, eagerly evaluates
``EconomicAgent.objects.all()`` right away, and returns a QuerySet
filtered by the result of that. So even though the final queryset
returned by ``with_user`` is not yet evaluated, the filtering
condition in that queryset is already fixed.

You could fix this by doing something along the lines of::

def with_user(self):
return EconomicAgent.objects.filter(users__isnull=False).distinct()

The important difference is that this version of ``with_user`` does
not evaluate anything eagerly, but rather sets up the necessary
filters to get the correct result set at the time of evaluation of
that queryset.

Depending on what you do with the queryset further, you might need to
change it a little bit to remove the ``distinct()`` call, and use a
subquery instead.
Post by bobhaugen
These questions remain unanswered, although I intend to do a bunch more
1. How pervasive is this problem? Does it affect template variables like
{{ object.foreign_key_method }} where the foreign_key_method returns a
queryset?
2. Is this behavior clearly documented anywhere?
Honestly, I'm not sure what exactly you're asking here. Your
implementation of ``with_user`` was hard-wiring a queryset filter
based on the state of the database at the time ``with_user`` was
called. The rest is just standard Python behavior – since the method
was called during import (in the ModelForm definition), the result of
that call was used for the rest of the process' lifetime.

Cheers,

Michal
--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/20160819123802.GJ27882%40koniiiik.org.
For more options, visit https://groups.google.com/d/optout.
bobhaugen
2016-08-19 12:44:42 UTC
Permalink
Post by Michal Petrucha
Honestly, I'm not sure what exactly you're asking here. Your
implementation of ``with_user`` was hard-wiring a queryset filter
based on the state of the database at the time ``with_user`` was
called. The rest is just standard Python behavior – since the method
was called during import (in the ModelForm definition), the result of
that call was used for the rest of the process' lifetime.
Ok, so from your response and that of Coues Ludovic upthread, I clearly
need to study Python more deeply. Will do.

Thanks for all the responses.
--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/bde5f813-155b-438b-9850-4311be2541b7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Michal Petrucha
2016-08-19 13:12:04 UTC
Permalink
Post by Michal Petrucha
Post by bobhaugen
These questions remain unanswered, although I intend to do a bunch more
1. How pervasive is this problem? Does it affect template variables like
{{ object.foreign_key_method }} where the foreign_key_method returns a
queryset?
2. Is this behavior clearly documented anywhere?
Honestly, I'm not sure what exactly you're asking here. Your
implementation of ``with_user`` was hard-wiring a queryset filter
based on the state of the database at the time ``with_user`` was
called. The rest is just standard Python behavior – since the method
was called during import (in the ModelForm definition), the result of
that call was used for the rest of the process' lifetime.
Actually, let me rephrase that. This problem you encountered here
wasn't a case of “you need to set the ``queryset`` argument of
``ModelChoiceField`` in your form's ``__init__``,” but rather a case
of “don't do too much filtering in your Python code – let your
database do the filtering”.

Using ``__init__`` to set the queryset is not really a very standard
thing to do. It's just a workaround to make a form work if it uses a
manager or queryset method that does not behave entirely correctly.

Does this help make things at least a little bit clearer?

Michal
--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/20160819131204.GK27882%40koniiiik.org.
For more options, visit https://groups.google.com/d/optout.
bobhaugen
2016-08-19 13:52:02 UTC
Permalink
Michal, thanks for following up.

I'll try your suggested code from your other response upthread and see if
it solves the problem as well as the form.__init__.

And do more digging about how Python does things. I am a self-taught
programmer who focuses on community economic development and I usually just
learn as much about software as I need to do what I want to do. Sometimes
that is not enough learning...;-)
Post by bobhaugen
Post by Michal Petrucha
Post by bobhaugen
These questions remain unanswered, although I intend to do a bunch
more
Post by Michal Petrucha
Post by bobhaugen
1. How pervasive is this problem? Does it affect template variables
like
Post by Michal Petrucha
Post by bobhaugen
{{ object.foreign_key_method }} where the foreign_key_method
returns a
Post by Michal Petrucha
Post by bobhaugen
queryset?
2. Is this behavior clearly documented anywhere?
Honestly, I'm not sure what exactly you're asking here. Your
implementation of ``with_user`` was hard-wiring a queryset filter
based on the state of the database at the time ``with_user`` was
called. The rest is just standard Python behavior – since the method
was called during import (in the ModelForm definition), the result of
that call was used for the rest of the process' lifetime.
Actually, let me rephrase that. This problem you encountered here
wasn't a case of “you need to set the ``queryset`` argument of
``ModelChoiceField`` in your form's ``__init__``,” but rather a case
of “don't do too much filtering in your Python code – let your
database do the filtering”.
Using ``__init__`` to set the queryset is not really a very standard
thing to do. It's just a workaround to make a form work if it uses a
manager or queryset method that does not behave entirely correctly.
Does this help make things at least a little bit clearer?
Michal
--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/697f546d-b6df-4c2e-bf00-f8c5d16665f7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Fred Stluka
2017-01-17 00:38:20 UTC
Permalink
<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Bob,<br>
<br>
On 8/19/16 8:02 AM, bobhaugen wrote:<br>
&gt; On Friday, August 19, 2016 at 5:20:45 AM UTC-5, Michal Petrucha
wrote:<br>
&gt;<br>
&gt; Moving the call to with_user to form.__init__ solved the
problem in the form ModelChoiceField.<br>
&gt;<br>
&gt; These questions remain unanswered, although I intend to do a
bunch more testing:<br>
&gt;<br>
&gt;     How pervasive is this problem? Does it affect template
variables like {{ object.foreign_key_method }} where the
foreign_key_method returns a queryset?<br>
<br>
I think this is very pervasive.  That is, it happened a few times <br>
to my team until we educated all developers to never use the <br>
<b>choices</b>, <b>limit_choices_to</b>, or <b>queryset</b>
parameters of a Model<br>
field or Form field.  Instead, we always explicitly set them in <br>
__init__() instead.<br>
<br>
I assume other teams also make this mistake often.<br>
<br>
For us, this mistake caused 3 separate type of problems:<br>
<br>
1. The problem you described:<br>
    Old data from the DB was "cached" until we restarted the <br>
    Django server, because the DB query was done only <br>
    when the Model or Form class was defined, not when each <br>
    Model or Form instance was created.  So, in answer to<br>
    your question above, yes, it would also affect template <br>
    variables that refer to the data that was loaded from the <br>
    DB when the class was defined.<br>
<br>
2. Occasional import loops, which prevented the Django <br>
    server from starting the app, because the value of the <br>
    <b>choices</b>, <b>limit_choices_to</b>, or <b>queryset</b>
parameter caused<br>
    another model to be loaded.  This would trip us up badly<br>
    because the import loop would often occur in our TEST or<br>
    PROD environments (Django loaded via Apache WSGI) but <br>
    not in our DEV environments (Django DEV server), since <br>
    the order of class imports could be different in those 2 <br>
    different configurations.  See:<br>
    -
<a class="moz-txt-link-freetext" href="https://groups.google.com/forum/#!topic/django-users/ONGCSO37UWY">https://groups.google.com/forum/#!topic/django-users/ONGCSO37UWY</a><br>
<br>
3. There's a warning here about this mistake being able to <br>
    pollute your TEST execution with PROD data:<br>
    -
<a class="moz-txt-link-freetext" href="https://docs.djangoproject.com/en/dev/topics/testing/overview/#the-test-database">https://docs.djangoproject.com/en/dev/topics/testing/overview/#the-test-database</a><br>
<br>
&gt;     Is this behavior clearly documented anywhere?<br>
<br>
It didn't jump out at us.  We got burned by and it and put in<br>
some time to diagnose it.  Then we found the above link that<br>
warns of one of the problems.<br>
<br>
<div class="moz-signature">--Fred
<br>
<hr>
Fred Stluka -- <a class="moz-txt-link-freetext" href="mailto:***@bristle.com">mailto:***@bristle.com</a> --
<a class="moz-txt-link-freetext" href="http://bristle.com/~fred/">http://bristle.com/~fred/</a>
<br>
Bristle Software, Inc -- <a class="moz-txt-link-freetext" href="http://bristle.com">http://bristle.com</a> -- Glad to be of
service!
<br>
Open Source: Without walls and fences, we need no Windows or
Gates.
<hr>
<br>
</div>
<br>
</body>
</html>

<p></p>

-- <br />
You received this message because you are subscribed to the Google Groups &quot;Django users&quot; group.<br />
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="mailto:django-users+***@googlegroups.com">django-users+***@googlegroups.com</a>.<br />
To post to this group, send email to <a href="mailto:django-***@googlegroups.com">django-***@googlegroups.com</a>.<br />
Visit this group at <a href="https://groups.google.com/group/django-users">https://groups.google.com/group/django-users</a>.<br />
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/django-users/4ae3b41a-1bbb-0f9f-660a-938a80ace6d0%40bristle.com?utm_medium=email&utm_source=footer">https://groups.google.com/d/msgid/django-users/4ae3b41a-1bbb-0f9f-660a-938a80ace6d0%40bristle.com</a>.<br />
For more options, visit <a href="https://groups.google.com/d/optout">https://groups.google.com/d/optout</a>.<br />
Continue reading on narkive:
Loading...