Python Multiprocessing With Django

[Vitaly Babiy]

I have a management command that that starts up a few process to process a
request: http://dpaste.com/7925/.

But every so offen I get a exception say mysql went a way, but it didn't.
Process PoolWorker-3:
Traceback (most recent call last):
File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1-py2.5-linux-i686.egg/multiprocessing/process.py",
line 236, in _bootstrap
self.run()
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1-py2.5-linux-i686.egg/multiprocessing/process.py",
line 93, in run
self._target(*self._args, **self._kwargs)
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1-py2.5-linux-i686.egg/multiprocessing/pool.py",
line 71, in worker
put((job, i, result))
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1-py2.5-linux-i686.egg/multiprocessing/queues.py",
line 353, in put
return send(obj)
PicklingError: Can't pickle <class 'http_tracker.models.DoesNotExist'>:
attribute lookup http_tracker.models.DoesNotExist failed
Traceback (most recent call last):
File "/home/vbabiy/projects/git-projects/howsthedotcom/manage.py", line 11,
in <module>
execute_manager(settings)
File "/usr/lib/python2.5/site-packages/django/core/management/__init__.py",
line 350, in execute_manager
utility.execute()
File "/usr/lib/python2.5/site-packages/django/core/management/__init__.py",
line 295, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/lib/python2.5/site-packages/django/core/management/base.py",
line 195, in run_from_argv
self.execute(*args, **options.__dict__)
File "/usr/lib/python2.5/site-packages/django/core/management/base.py",
line 222, in execute
output = self.handle(*args, **options)
File
"/home/vbabiy/projects/git-projects/howsthedotcom/tracker/management/commands/tracker.py",
line 37, in handle
res = p.map(check_site, sites)
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1-py2.5-linux-i686.egg/multiprocessing/pool.py",
line 148, in map
return self.map_async(func, iterable, chunksize).get()
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1-py2.5-linux-i686.egg/multiprocessing/pool.py",
line 422, in get
raise self._value
_mysql_exceptions.OperationalError: (2013, 'Lost connection to MySQL server
during query')

Any one got any idea why this is happening?

Vitaly Babiy

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to django-***@googlegroups.com
To unsubscribe from this group, send email to django-users+***@googlegroups.com
For more options, visit this group at http://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---

[Jirka Vejrazka]

Post by Vitaly Babiy
I have a management command that that starts up a few process to process a
request: http://dpaste.com/7925/.

Hi, I was recently debugging similar issue and came to a conclusion
(which may be wrong of course :) that multiprocessing and Django DB
connections don't play well together. I ended up closing Django DB
connection first thing in the new process. It'll recreate a new
connection when it needs one, but that one will have no references to
the connection used by the parent.

Post by Vitaly Babiy

from django.db import connection
connection.close()

This solved my problem.

Cheers

Jirka

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to django-***@googlegroups.com
To unsubscribe from this group, send email to django-users+***@googlegroups.com
For more options, visit this group at http://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---

[Vitaly Babiy]

Alright I will have to give this a try

Vitaly Babiy

Post by Jirka Vejrazka

Post by Vitaly Babiy
I have a management command that that starts up a few process to process

a

Post by Vitaly Babiy
request: http://dpaste.com/7925/.

Hi, I was recently debugging similar issue and came to a conclusion
(which may be wrong of course :) that multiprocessing and Django DB
connections don't play well together. I ended up closing Django DB
connection first thing in the new process. It'll recreate a new
connection when it needs one, but that one will have no references to
the connection used by the parent.

Post by Vitaly Babiy

from django.db import connection
connection.close()

This solved my problem.
Cheers
Jirka

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to django-***@googlegroups.com
To unsubscribe from this group, send email to django-users+***@googlegroups.com
For more options, visit this group at http://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---

[Vitaly Babiy]

Thanks Jirka, that fixed the problem. But I think I will file a bug report
that seem to be very hacky :)

Vitaly Babiy

Post by Vitaly Babiy
Alright I will have to give this a try
Vitaly Babiy

Post by Jirka Vejrazka

Post by Vitaly Babiy
I have a management command that that starts up a few process to process

a

Post by Vitaly Babiy
request: http://dpaste.com/7925/.

Hi, I was recently debugging similar issue and came to a conclusion
(which may be wrong of course :) that multiprocessing and Django DB
connections don't play well together. I ended up closing Django DB
connection first thing in the new process. It'll recreate a new
connection when it needs one, but that one will have no references to
the connection used by the parent.

Post by Vitaly Babiy

from django.db import connection
connection.close()

This solved my problem.
Cheers
Jirka

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to django-***@googlegroups.com
To unsubscribe from this group, send email to django-users+***@googlegroups.com
For more options, visit this group at http://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---

[Jirka Vejrazka]

Post by Vitaly Babiy
Thanks Jirka, that fixed the problem. But I think I will file a bug report
that seem to be very hacky :)

Great, I'm glad it solved your problem. Personally, I don't think it
is a bug, as we both pushed Django well beyond its "area of
expertise". I really believe that Django is great for serving web
content and everyone (including) me trying to implement it in
different areas must be ready for problems and gotchas here and there.
But YMMV.

Cheers

Jirka

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to django-***@googlegroups.com
To unsubscribe from this group, send email to django-users+***@googlegroups.com
For more options, visit this group at http://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---

[Peter Schmidt]

Just in case someone else stumbles across this old thread.

TLDR: The recipe may be able to be improved - I only needed to close the
connection in the parent Python process, not in the child process.

--
Best guess hypothesis as to what's going on is when you fork a process, the
OS (e.g. OSX Mountain Lion for me) will simply copy the memory, including
any current DB connection details.

Then the child process in Django sometimes tries to reuse it.
I say sometimes because Django's backend.mysql.base.py tries:

self.connection.ping()


If that fails, it closes the connection, then the Django magic of creating
a new connection to the default DB will happen later when it is needed.

Reusing a connection then sometimes leads to one or more whacky things in
the child processes:
- DatabaseError: (2013, 'Lost connection to MySQL server during query')
- DatabaseError: (2006, 'MySQL server has gone away')
- MultipleObjectsReturned: get() returned more than one <Model object -- it
returned 6! Lookup parameters were ...
- DoesNotExist: <Model object> matching query does not exist.

Perhaps someone will be motivated one day to patch something like this into
Django itself, but it feels like something that comes with the territory of
handling multiprocessing without using a meaningful subset of concurrency
such as RabbitMQ/Celery for a task queue. In other words, there's a reason
Python's multiprocessing documentation is filled with many notes and
warnings - concurrency is one of the more difficult to use constructs in
the developer toolbox and as developers we should use it only when it is
justified by specific requirements for extreme performance (or perhaps for
the personal challenge).

Post by Jirka Vejrazka

Post by Vitaly Babiy
I have a management command that that starts up a few process to process

a

Post by Vitaly Babiy
request: http://dpaste.com/7925/.

Hi, I was recently debugging similar issue and came to a conclusion
(which may be wrong of course :) that multiprocessing and Django DB
connections don't play well together. I ended up closing Django DB
connection first thing in the new process. It'll recreate a new
connection when it needs one, but that one will have no references to
the connection used by the parent.

Post by Vitaly Babiy

from django.db import connection
connection.close()

This solved my problem.
Cheers
Jirka

[Subha Bera]

My program trigger some regression file in a remote host for that I used
paramiko module.
But I want to trigger same kinda file to different machines parallel-ly.
I am using django so that people can use this as web app.
I am using url-view-template model. How can use I multi threading in the
views function? If you come up with a solution please let me know.
It requires the( if "__name__"="main": ).How can I do that?

Post by Vitaly Babiy
I have a management command that that starts up a few process to process a
request: http://dpaste.com/7925/.
But every so offen I get a exception say mysql went a way, but it didn't.
File "/usr/lib/python2.5/site-
packages/multiprocessing-2.6.1.1-py2.5-linux-i686.egg/multiprocessing/process.py",
line 236, in _bootstrap
self.run()
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1-py2.5-linux-i686.egg/multiprocessing/process.py",
line 93, in run
self._target(*self._args, **self._kwargs)
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1-py2.5-linux-i686.egg/multiprocessing/pool.py",
line 71, in worker
put((job, i, result))
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1-py2.5-linux-i686.egg/multiprocessing/queues.py",
line 353, in put
return send(obj)
attribute lookup http_tracker.models.DoesNotExist failed
File "/home/vbabiy/projects/git-projects/howsthedotcom/manage.py", line
11, in <module>
execute_manager(settings)
File
"/usr/lib/python2.5/site-packages/django/core/management/__init__.py", line
350, in execute_manager
utility.execute()
File
"/usr/lib/python2.5/site-packages/django/core/management/__init__.py", line
295, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/lib/python2.5/site-packages/django/core/management/base.py",
line 195, in run_from_argv
self.execute(*args, **options.__dict__)
File "/usr/lib/python2.5/site-packages/django/core/management/base.py",
line 222, in execute
output = self.handle(*args, **options)
File
"/home/vbabiy/projects/git-projects/howsthedotcom/tracker/management/commands/tracker.py",
line 37, in handle
res = p.map(check_site, sites)
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1-py2.5-linux-i686.egg/multiprocessing/pool.py",
line 148, in map
return self.map_async(func, iterable, chunksize).get()
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1-py2.5-linux-i686.egg/multiprocessing/pool.py",
line 422, in get
raise self._value
_mysql_exceptions.OperationalError: (2013, 'Lost connection to MySQL
server during query')
Any one got any idea why this is happening?
Vitaly Babiy