Discussion:
Channels group_send hangs in extrenal process
Yaro Kifor
2018-11-13 22:01:20 UTC
Permalink
Hello,

I am a little new to channels, but I can't seem to figure out this issue. I
am trying to use group_send from an external process and I am using Redis
as my channel backend. Here is what my process code looks like:


from channels.layers import get_channel_layer
from multiprocessing import Process
from asgiref.sync import AsyncToSync, SyncToAsync

class GenericAgent(Process):
def __init__(self, *args, **kwargs):
self.channel_layer = get_channel_layer()
self.send_message = AsyncToSync(self.channel_layer.group_send)
super().__init__(*args, **kwargs)

def run(self):
while True:
print("Sending data")
self.send_message('test', {'test': 0})
time.sleep(1)


Anytime I run the process it will print out "Sending data" once and do
nothing else. Any channel that is part of the test group will not receive
anything as well. If anyone has a clue to what the issue might be please
let know your help is greatly appreciated.

Here is my pip freeze:

aioredis==1.2.0
asgiref==2.3.2
async-timeout==3.0.1
attrs==18.2.0
autobahn==18.11.1
Automat==0.7.0
certifi==2018.4.16
channels==2.1.5
channels-redis==2.3.1
chardet==3.0.4
constantly==15.1.0
daphne==2.2.3
Django==2.0.6
easy-thumbnails==2.5
future==0.15.2
hiredis==0.2.0
hyperlink==18.0.0
idna==2.7
imageio==2.3.0
incremental==17.5.0
interop==0.0.0
LatLon23==1.0.7
monotonic==1.2
msgpack==0.5.6
numpy==1.14.4
Pillow==5.1.0
psycopg2-binary==2.7.5
PyHamcrest==1.9.0
pyproj==1.9.5.1
pytz==2018.4
rawpy==0.10.1
requests==2.19.1
six==1.11.0
sorl-thumbnail==12.4.1
Twisted==18.9.0
txaio==18.8.1
urllib3==1.23
zope.interface==4.6.0


Thanks in advance,
Yaro
--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/4916f184-9db9-4d34-a9cd-3d6ff5238aff%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Andrew Godwin
2018-11-13 22:33:44 UTC
Permalink
I've never tried using the async-to-sync stuff with multiprocessing -
first, I'd try removing that and see if it still works (just run from a
management command or script directly).

If it still fails, I'd suggest running a redis MONITOR to see if any
traffic makes it to Redis at all.

Andrew
Post by Yaro Kifor
Hello,
I am a little new to channels, but I can't seem to figure out this issue.
I am trying to use group_send from an external process and I am using Redis
from channels.layers import get_channel_layer
from multiprocessing import Process
from asgiref.sync import AsyncToSync, SyncToAsync
self.channel_layer = get_channel_layer()
self.send_message = AsyncToSync(self.channel_layer.group_send)
super().__init__(*args, **kwargs)
print("Sending data")
self.send_message('test', {'test': 0})
time.sleep(1)
Anytime I run the process it will print out "Sending data" once and do
nothing else. Any channel that is part of the test group will not receive
anything as well. If anyone has a clue to what the issue might be please
let know your help is greatly appreciated.
aioredis==1.2.0
asgiref==2.3.2
async-timeout==3.0.1
attrs==18.2.0
autobahn==18.11.1
Automat==0.7.0
certifi==2018.4.16
channels==2.1.5
channels-redis==2.3.1
chardet==3.0.4
constantly==15.1.0
daphne==2.2.3
Django==2.0.6
easy-thumbnails==2.5
future==0.15.2
hiredis==0.2.0
hyperlink==18.0.0
idna==2.7
imageio==2.3.0
incremental==17.5.0
interop==0.0.0
LatLon23==1.0.7
monotonic==1.2
msgpack==0.5.6
numpy==1.14.4
Pillow==5.1.0
psycopg2-binary==2.7.5
PyHamcrest==1.9.0
pyproj==1.9.5.1
pytz==2018.4
rawpy==0.10.1
requests==2.19.1
six==1.11.0
sorl-thumbnail==12.4.1
Twisted==18.9.0
txaio==18.8.1
urllib3==1.23
zope.interface==4.6.0
Thanks in advance,
Yaro
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/4916f184-9db9-4d34-a9cd-3d6ff5238aff%40googlegroups.com
<https://groups.google.com/d/msgid/django-users/4916f184-9db9-4d34-a9cd-3d6ff5238aff%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAFwN1upK-iXuxbZNTcPu78OzPnDAfS8-E%3DfOHnPUio0u1dppRw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Yaro Kifor
2018-11-13 22:57:28 UTC
Permalink
Huh that's weird. I've removed the async-to-sync stuff and now the script
doesn't hang. I've checked redis MONITOR and I do see traffic now which
nice, but for some reason my consumer isn't seeing any of it. Maybe I
messed up the group_add method. Here's the redis MONITOR output.

1542149506.032331 [0 172.19.0.4:45506] "BRPOPLPUSH"
"asgi:specific.eYdlWGyC!" "asgi:specific.eYdlWGyC!$inflight" "5"
1542149512.043776 [0 172.19.0.4:45506] "EVAL" "\n local
backed_up = redis.call('LRANGE', ARGV[2], 0, -1)\n for i =
#backed_up, 1, -1 do\n redis.call('LPUSH', ARGV[1],
backed_up[i])\n end\n redis.call('DEL',
ARGV[2])\n " "0" "asgi:specific.eYdlWGyC!"
"asgi:specific.eYdlWGyC!$inflight"
1542149512.043867 [0 lua] "LRANGE" "asgi:specific.eYdlWGyC!$inflight" "0"
"-1"
1542149512.043884 [0 lua] "DEL" "asgi:specific.eYdlWGyC!$inflight"
1542149512.044194 [0 172.19.0.4:45506] "BRPOPLPUSH"
"asgi:specific.eYdlWGyC!" "asgi:specific.eYdlWGyC!$inflight" "5"
1542149518.056667 [0 172.19.0.4:45506] "EVAL" "\n local
backed_up = redis.call('LRANGE', ARGV[2], 0, -1)\n for i =
#backed_up, 1, -1 do\n redis.call('LPUSH', ARGV[1],
backed_up[i])\n end\n redis.call('DEL',
ARGV[2])\n " "0" "asgi:specific.eYdlWGyC!"
"asgi:specific.eYdlWGyC!$inflight"
1542149518.056757 [0 lua] "LRANGE" "asgi:specific.eYdlWGyC!$inflight" "0"
"-1"
1542149518.056773 [0 lua] "DEL" "asgi:specific.eYdlWGyC!$inflight"
1542149518.057120 [0 172.19.0.4:45506] "BRPOPLPUSH"
"asgi:specific.eYdlWGyC!" "asgi:specific.eYdlWGyC!$inflight" "5"
1542149524.068806 [0 172.19.0.4:45506] "EVAL" "\n local
backed_up = redis.call('LRANGE', ARGV[2], 0, -1)\n for i =
#backed_up, 1, -1 do\n redis.call('LPUSH', ARGV[1],
backed_up[i])\n end\n redis.call('DEL',
ARGV[2])\n " "0" "asgi:specific.eYdlWGyC!"
"asgi:specific.eYdlWGyC!$inflight"
1542149524.068919 [0 lua] "LRANGE" "asgi:specific.eYdlWGyC!$inflight" "0"
"-1"
1542149524.068935 [0 lua] "DEL" "asgi:specific.eYdlWGyC!$inflight"
1542149524.069345 [0 172.19.0.4:45506] "BRPOPLPUSH"
"asgi:specific.eYdlWGyC!" "asgi:specific.eYdlWGyC!$inflight" "5"
1542149530.080909 [0 172.19.0.4:45506] "EVAL" "\n local
backed_up = redis.call('LRANGE', ARGV[2], 0, -1)\n for i =
#backed_up, 1, -1 do\n redis.call('LPUSH', ARGV[1],
backed_up[i])\n end\n redis.call('DEL',
ARGV[2])\n " "0" "asgi:specific.eYdlWGyC!"
"asgi:specific.eYdlWGyC!$inflight"
1542149530.080999 [0 lua] "LRANGE" "asgi:specific.eYdlWGyC!$inflight" "0"
"-1"
1542149530.081015 [0 lua] "DEL" "asgi:specific.eYdlWGyC!$inflight"
1542149530.081372 [0 172.19.0.4:45506] "BRPOPLPUSH"
"asgi:specific.eYdlWGyC!" "asgi:specific.eYdlWGyC!$inflight" "5"
I'll let you know where I get with this. Thanks for the help so far.

Yaro
I've never tried using the async-to-sync stuff with multiprocessing -
first, I'd try removing that and see if it still works (just run from a
management command or script directly).
If it still fails, I'd suggest running a redis MONITOR to see if any
traffic makes it to Redis at all.
Andrew
Post by Yaro Kifor
Hello,
I am a little new to channels, but I can't seem to figure out this issue.
I am trying to use group_send from an external process and I am using Redis
from channels.layers import get_channel_layer
from multiprocessing import Process
from asgiref.sync import AsyncToSync, SyncToAsync
self.channel_layer = get_channel_layer()
self.send_message = AsyncToSync(self.channel_layer.group_send)
super().__init__(*args, **kwargs)
print("Sending data")
self.send_message('test', {'test': 0})
time.sleep(1)
Anytime I run the process it will print out "Sending data" once and do
nothing else. Any channel that is part of the test group will not receive
anything as well. If anyone has a clue to what the issue might be please
let know your help is greatly appreciated.
aioredis==1.2.0
asgiref==2.3.2
async-timeout==3.0.1
attrs==18.2.0
autobahn==18.11.1
Automat==0.7.0
certifi==2018.4.16
channels==2.1.5
channels-redis==2.3.1
chardet==3.0.4
constantly==15.1.0
daphne==2.2.3
Django==2.0.6
easy-thumbnails==2.5
future==0.15.2
hiredis==0.2.0
hyperlink==18.0.0
idna==2.7
imageio==2.3.0
incremental==17.5.0
interop==0.0.0
LatLon23==1.0.7
monotonic==1.2
msgpack==0.5.6
numpy==1.14.4
Pillow==5.1.0
psycopg2-binary==2.7.5
PyHamcrest==1.9.0
pyproj==1.9.5.1
pytz==2018.4
rawpy==0.10.1
requests==2.19.1
six==1.11.0
sorl-thumbnail==12.4.1
Twisted==18.9.0
txaio==18.8.1
urllib3==1.23
zope.interface==4.6.0
Thanks in advance,
Yaro
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/4916f184-9db9-4d34-a9cd-3d6ff5238aff%40googlegroups.com
<https://groups.google.com/d/msgid/django-users/4916f184-9db9-4d34-a9cd-3d6ff5238aff%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to a topic in the
Google Groups "Django users" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/django-users/Y2xNv_aOVqE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/CAFwN1upK-iXuxbZNTcPu78OzPnDAfS8-E%3DfOHnPUio0u1dppRw%40mail.gmail.com
<https://groups.google.com/d/msgid/django-users/CAFwN1upK-iXuxbZNTcPu78OzPnDAfS8-E%3DfOHnPUio0u1dppRw%40mail.gmail.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CANkeueVxK0XwdUJqaXZPZYSE0r5yOc%3DFYvdPSuVnMLhTQ4q5_Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Yaro Kifor
2018-11-13 23:40:16 UTC
Permalink
I've ensured my consumer is setup properly, added a type field to the
message and added a callback method for that type. My callback method isn't
getting called. I've tested my callback method from a different source and
1542152017.290859 [0 lua] "EXPIRE" "asgi:specific.cNbsZzOu!" "60"
1542152017.290866 [0 lua] "LLEN" "asgi:specific.aqRkFBbT!"
1542152017.290874 [0 lua] "LPUSH" "asgi:specific.aqRkFBbT!"
"\x83\xa4type\xa4test\xa4test\xa4test\xb0__asgi_channel__\x91\xbespecific.aqRkFBbT!FCpiHVGLaITQ
While from my GenericAgent process, redis MONITOR doesn't show anything
like this (see my previous email). I am not sure what my process is
missing. Is this a bug or am I going about this all wrong?

Yaro
Huh that's weird. I've removed the async-to-sync stuff and now the script
doesn't hang. I've checked redis MONITOR and I do see traffic now which
nice, but for some reason my consumer isn't seeing any of it. Maybe I
messed up the group_add method. Here's the redis MONITOR output.
1542149506.032331 [0 172.19.0.4:45506] "BRPOPLPUSH"
"asgi:specific.eYdlWGyC!" "asgi:specific.eYdlWGyC!$inflight" "5"
1542149512.043776 [0 172.19.0.4:45506] "EVAL" "\n local
backed_up = redis.call('LRANGE', ARGV[2], 0, -1)\n for i =
#backed_up, 1, -1 do\n redis.call('LPUSH', ARGV[1],
backed_up[i])\n end\n redis.call('DEL',
ARGV[2])\n " "0" "asgi:specific.eYdlWGyC!"
"asgi:specific.eYdlWGyC!$inflight"
1542149512.043867 [0 lua] "LRANGE" "asgi:specific.eYdlWGyC!$inflight" "0"
"-1"
1542149512.043884 [0 lua] "DEL" "asgi:specific.eYdlWGyC!$inflight"
1542149512.044194 [0 172.19.0.4:45506] "BRPOPLPUSH"
"asgi:specific.eYdlWGyC!" "asgi:specific.eYdlWGyC!$inflight" "5"
1542149518.056667 [0 172.19.0.4:45506] "EVAL" "\n local
backed_up = redis.call('LRANGE', ARGV[2], 0, -1)\n for i =
#backed_up, 1, -1 do\n redis.call('LPUSH', ARGV[1],
backed_up[i])\n end\n redis.call('DEL',
ARGV[2])\n " "0" "asgi:specific.eYdlWGyC!"
"asgi:specific.eYdlWGyC!$inflight"
1542149518.056757 [0 lua] "LRANGE" "asgi:specific.eYdlWGyC!$inflight" "0"
"-1"
1542149518.056773 [0 lua] "DEL" "asgi:specific.eYdlWGyC!$inflight"
1542149518.057120 [0 172.19.0.4:45506] "BRPOPLPUSH"
"asgi:specific.eYdlWGyC!" "asgi:specific.eYdlWGyC!$inflight" "5"
1542149524.068806 [0 172.19.0.4:45506] "EVAL" "\n local
backed_up = redis.call('LRANGE', ARGV[2], 0, -1)\n for i =
#backed_up, 1, -1 do\n redis.call('LPUSH', ARGV[1],
backed_up[i])\n end\n redis.call('DEL',
ARGV[2])\n " "0" "asgi:specific.eYdlWGyC!"
"asgi:specific.eYdlWGyC!$inflight"
1542149524.068919 [0 lua] "LRANGE" "asgi:specific.eYdlWGyC!$inflight" "0"
"-1"
1542149524.068935 [0 lua] "DEL" "asgi:specific.eYdlWGyC!$inflight"
1542149524.069345 [0 172.19.0.4:45506] "BRPOPLPUSH"
"asgi:specific.eYdlWGyC!" "asgi:specific.eYdlWGyC!$inflight" "5"
1542149530.080909 [0 172.19.0.4:45506] "EVAL" "\n local
backed_up = redis.call('LRANGE', ARGV[2], 0, -1)\n for i =
#backed_up, 1, -1 do\n redis.call('LPUSH', ARGV[1],
backed_up[i])\n end\n redis.call('DEL',
ARGV[2])\n " "0" "asgi:specific.eYdlWGyC!"
"asgi:specific.eYdlWGyC!$inflight"
1542149530.080999 [0 lua] "LRANGE" "asgi:specific.eYdlWGyC!$inflight" "0"
"-1"
1542149530.081015 [0 lua] "DEL" "asgi:specific.eYdlWGyC!$inflight"
1542149530.081372 [0 172.19.0.4:45506] "BRPOPLPUSH"
"asgi:specific.eYdlWGyC!" "asgi:specific.eYdlWGyC!$inflight" "5"
I'll let you know where I get with this. Thanks for the help so far.
Yaro
I've never tried using the async-to-sync stuff with multiprocessing -
first, I'd try removing that and see if it still works (just run from a
management command or script directly).
If it still fails, I'd suggest running a redis MONITOR to see if any
traffic makes it to Redis at all.
Andrew
Post by Yaro Kifor
Hello,
I am a little new to channels, but I can't seem to figure out this
issue. I am trying to use group_send from an external process and I am
from channels.layers import get_channel_layer
from multiprocessing import Process
from asgiref.sync import AsyncToSync, SyncToAsync
self.channel_layer = get_channel_layer()
self.send_message = AsyncToSync(self.channel_layer.group_send)
super().__init__(*args, **kwargs)
print("Sending data")
self.send_message('test', {'test': 0})
time.sleep(1)
Anytime I run the process it will print out "Sending data" once and do
nothing else. Any channel that is part of the test group will not receive
anything as well. If anyone has a clue to what the issue might be please
let know your help is greatly appreciated.
aioredis==1.2.0
asgiref==2.3.2
async-timeout==3.0.1
attrs==18.2.0
autobahn==18.11.1
Automat==0.7.0
certifi==2018.4.16
channels==2.1.5
channels-redis==2.3.1
chardet==3.0.4
constantly==15.1.0
daphne==2.2.3
Django==2.0.6
easy-thumbnails==2.5
future==0.15.2
hiredis==0.2.0
hyperlink==18.0.0
idna==2.7
imageio==2.3.0
incremental==17.5.0
interop==0.0.0
LatLon23==1.0.7
monotonic==1.2
msgpack==0.5.6
numpy==1.14.4
Pillow==5.1.0
psycopg2-binary==2.7.5
PyHamcrest==1.9.0
pyproj==1.9.5.1
pytz==2018.4
rawpy==0.10.1
requests==2.19.1
six==1.11.0
sorl-thumbnail==12.4.1
Twisted==18.9.0
txaio==18.8.1
urllib3==1.23
zope.interface==4.6.0
Thanks in advance,
Yaro
--
You received this message because you are subscribed to the Google
Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/4916f184-9db9-4d34-a9cd-3d6ff5238aff%40googlegroups.com
<https://groups.google.com/d/msgid/django-users/4916f184-9db9-4d34-a9cd-3d6ff5238aff%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to a topic in the
Google Groups "Django users" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/django-users/Y2xNv_aOVqE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/CAFwN1upK-iXuxbZNTcPu78OzPnDAfS8-E%3DfOHnPUio0u1dppRw%40mail.gmail.com
<https://groups.google.com/d/msgid/django-users/CAFwN1upK-iXuxbZNTcPu78OzPnDAfS8-E%3DfOHnPUio0u1dppRw%40mail.gmail.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CANkeueVHi9xaPTNPEHAHkmzF7571QFLi_ChjtCSReAoG7fkQOA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Loading...