Dynamic model, reverse foreign key not working

[Roman Akopov]

Hello,

My problem is very rare so I'll try ad add as much useful detail as
possible.

I am using python 3.5 and 3.6, Django 1.11.4

I am creating complex security related application and need to generate
additional models based on other applications' models. In general,
everything works fine, models are created, migrated without a problem. But
there is single problem I totally failed to solve.

Let's say there is preexisting model Alpha of some other application and I
dynamically create model Beta which references Alpha with ForeignKey. The
problem is that Beta does not get reverse relation, so I can query for
beta.alpha, but not for alpha.betas. I receive
"django.core.exceptions.FieldError: Cannot resolve keyword 'betas' into
field. Choices are: x, y, z"

Here is my code























* def _create_dynamic_model(self, model_label, fields, attributes=None,
options=None): from django.db import models class
Meta: pass setattr(Meta, 'app_label', '_talos') if
options is not None: for key, value in
options.items(): setattr(Meta, key, value) attrs =
{'__module__': '_talos', '_talos_dynamic': True, 'Meta': Meta} if
attributes: attrs.update(attributes) if
fields: attrs.update(fields) model = type(model_label,
(models.Model,), attrs) return model*

I call it like this (where self.model is referenced model class)

def _create_object_permission_model(self):
from django.db import models

return self._create_dynamic_model(
'o_{0}_{1}'.format(self.model._meta.app_config.label,
self.model.__name__),
fields={
'role': models.ForeignKey(
'talos.Role',
related_name='+',
on_delete=models.CASCADE),
'permission': models.ForeignKey(
'talos.ObjectPermission',
related_name='+',
on_delete=models.CASCADE),
'target': models.ForeignKey(
self.model,
related_name='talos_permissions', # self.model does not
receive reverse relation!
on_delete=models.CASCADE)
},
options={
'unique_together': [('role', 'permission', 'target')],
'index_together': [
('target', 'permission', 'role'),
('target', 'role', 'permission'),
('role', 'target', 'permission')]
})


I tried calling contribute_to_class and contribute_to_related_class methods
manually, tried to call this code at different moments (My application
ready method, class_prepared signal handler), with absolutely no luck.

I tried to read Django sources, and got that contribute_to_related_class is
called from contribute_to_class with delay, but it did not help me realize
what exactly I am doing wrong.

I am trying to use reverse relation from custom view and admin, so
everything should be pretty much initialized already.

Roman

[Michal Petrucha]

Post by Roman Akopov
Hello,
My problem is very rare so I'll try ad add as much useful detail as
possible.
I am using python 3.5 and 3.6, Django 1.11.4
I am creating complex security related application and need to generate
additional models based on other applications' models. In general,
everything works fine, models are created, migrated without a problem. But
there is single problem I totally failed to solve.
Let's say there is preexisting model Alpha of some other application and I
dynamically create model Beta which references Alpha with ForeignKey. The
problem is that Beta does not get reverse relation, so I can query for
beta.alpha, but not for alpha.betas. I receive
"django.core.exceptions.FieldError: Cannot resolve keyword 'betas' into
field. Choices are: x, y, z"

So the deal is that each model's _meta caches a bunch of structures
storing the list of fields, reverse relations, and so on, the first
time you access any of them. If you add a new field (or a new reverse
relation) after those caches have already been filled, the new field
or relation won't be reflected in them, leading to errors just like
yours.

There's an internal undocumented API that takes care of this,
Model._meta._expire_cache(), which will clear all those caches. It
should do the trick for you, but as always with private APIs, be aware
that it might break or change in the future.

Cheers,

Michal

[Roman Akopov]

Post by Michal Petrucha
So the deal is that each model's _meta caches a bunch of structures
storing the list of fields, reverse relations, and so on, the first
time you access any of them. If you add a new field (or a new reverse
relation) after those caches have already been filled, the new field
or relation won't be reflected in them, leading to errors just like
yours.
There's an internal undocumented API that takes care of this,
Model._meta._expire_cache(), which will clear all those caches. It
should do the trick for you, but as always with private APIs, be aware
that it might break or change in the future.
Cheers,
Michal

Michael,

Thanks for great hint!

Unfortulately, it did not help. I have added "model._meta._expire_cache()"
call almost everywhere, before generating dynamic model, after, between
steps, it did not help a bit, error is exactly the same.
Also, I have additionally tested my application against django 1.10 and
django 1.9 and got exactly the same result.

Roman

[Michal Petrucha]

Post by Roman Akopov
Unfortulately, it did not help. I have added "model._meta._expire_cache()"
call almost everywhere, before generating dynamic model, after, between
steps, it did not help a bit, error is exactly the same.
Also, I have additionally tested my application against django 1.10 and
django 1.9 and got exactly the same result.

On which models did you call that? You should call it on the target
model of any relationship that you create dynamically. So if you have
existing models Target1, and Target2, and create a new model Dynamic
with a ForeignKey(Target1) and ManyToManyField(Target2), you'd need to
call _expire_cache() on Target1 and Target2 right after creating the
dynamic model, but before trying to make any queries using those new
reverse relations.

If this doesn't help, then you might have to investigate if there's
perhaps some cached attribute that doesn't get cleared.

Good luck,

Michal

[Roman Akopov]

I call it on target model, the one with reverse relation missing.

Post by Roman Akopov

Post by Roman Akopov
Unfortulately, it did not help. I have added

"model._meta._expire_cache()"

Post by Roman Akopov
call almost everywhere, before generating dynamic model, after, between
steps, it did not help a bit, error is exactly the same.
Also, I have additionally tested my application against django 1.10 and
django 1.9 and got exactly the same result.

On which models did you call that? You should call it on the target
model of any relationship that you create dynamically. So if you have
existing models Target1, and Target2, and create a new model Dynamic
with a ForeignKey(Target1) and ManyToManyField(Target2), you'd need to
call _expire_cache() on Target1 and Target2 right after creating the
dynamic model, but before trying to make any queries using those new
reverse relations.
If this doesn't help, then you might have to investigate if there's
perhaps some cached attribute that doesn't get cleared.
Good luck,
Michal

[Roman Akopov]

I have investigated a bit more and looks lite it is Options._relation_tree
property, it's value is calculated by _populate_directed_relation_graph

Post by Michal Petrucha
If this doesn't help, then you might have to investigate if there's
perhaps some cached attribute that doesn't get cleared.
Good luck,
Michal