Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

[Cal Leeming [Simplicity Media Ltd]]

Hi all,

Some of you may have noticed, in the last few months I've done quite a few
posts/snippets about handling large data sets in Django. At the end of this
month (after what seems like a lifetime of trial and error), we're finally
going to be releasing a new site which holds around 40mil+ rows of data,
grows by about 300-500k rows each day, handles 5GB of uploads per day, and
can handle around 1024 requests per second on stress test on a moderately
spec'd server.

As the entire thing is written in Django (and a bunch of other open source
products), I'd really like to give something back to the community. (stack
incls Celery/RabbitMQ/Sphinx SE/PYQuery/Percona
MySQL/NGINX/supervisord/debian etc)

Therefore, I'd like to see if there would be any interest in webcast in
which I would explain how we handle such large amounts of data, the trial
and error processes we went through, some really neat tricks we've done to
avoid bottlenecks, our own approach to smart content filtering, and some of
the valuable lessons we have learned. The webcast would be completely free
of charge, last a couple of hours (with a short break) and anyone can
attend. I'd also offer up a Q&A session at the end.

If you're interested, please reply on-list so others can see.

Thanks

Cal

[Michał Sawicz]

Dnia 2011-06-22, śro o godzinie 14:15 +0100, Cal Leeming [Simplicity

Post by Cal Leeming [Simplicity Media Ltd]
If you're interested, please reply on-list so others can see.

Sure, I'd attend.

[Thomas Weholt]

Yes! I'm in.

Out of curiosity: When inserting lots of data, how do you do it? Using
the orm? Have you looked at http://pypi.python.org/pypi/dse/2.1.0 ? I
wrote DSE to solve inserting/updating huge sets of data, but if
there's a better way to do it that would be especially interesting to
hear more about ( and sorry for the self promotion ).

Regards,
Thomas

On Wed, Jun 22, 2011 at 3:15 PM, Cal Leeming [Simplicity Media Ltd]

Post by Cal Leeming [Simplicity Media Ltd]
Hi all,
Some of you may have noticed, in the last few months I've done quite a few
posts/snippets about handling large data sets in Django. At the end of this
month (after what seems like a lifetime of trial and error), we're finally
going to be releasing a new site which holds around 40mil+ rows of data,
grows by about 300-500k rows each day, handles 5GB of uploads per day, and
can handle around 1024 requests per second on stress test on a moderately
spec'd server.
As the entire thing is written in Django (and a bunch of other open source
products), I'd really like to give something back to the community. (stack
incls Celery/RabbitMQ/Sphinx SE/PYQuery/Percona
MySQL/NGINX/supervisord/debian etc)
Therefore, I'd like to see if there would be any interest in webcast in
which I would explain how we handle such large amounts of data, the trial
and error processes we went through, some really neat tricks we've done to
avoid bottlenecks, our own approach to smart content filtering, and some of
the valuable lessons we have learned. The webcast would be completely free
of charge, last a couple of hours (with a short break) and anyone can
attend. I'd also offer up a Q&A session at the end.
If you're interested, please reply on-list so others can see.
Thanks
Cal
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

--
Mvh/Best regards,
Thomas Weholt
http://www.weholt.org

[Cal Leeming [Simplicity Media Ltd]]

Hey Thomas,

Yeah we actually spoke a little while ago about DSE. In the end, we actually
used a custom approach which analyses data in blocks of 50k rows, builds a
list of rows which need changing to the same value, then applied them in
bulk using update() + F().

Here's our benchmark:

(42.11s) Found 49426 objs (match: 16107) (db writes: 50847) (range: 72300921
~ 72350921), (avg 13.8 mins/million) - [('is_checked', 49426),
('is_image_blocked', 0), ('has_link', 1420), ('is_spam', 1)]
(44.50s) Found 49481 objs (match: 16448) (db writes: 50764) (range: 72350921
~ 72400921), (avg 14.6 mins/million) - [('is_checked', 49481),
('is_image_blocked', 0), ('has_link', 1283), ('is_spam', 0)]
(55.78s) Found 49627 objs (match: 18516) (db writes: 50832) (range: 72400921
~ 72450921), (avg 18.3 mins/million) - [('is_checked', 49627),
('is_image_blocked', 0), ('has_link', 1205), ('is_spam', 0)]
(42.03s) Found 49674 objs (match: 17244) (db writes: 51655) (range: 72450921
~ 72500921), (avg 13.6 mins/million) - [('is_checked', 49674),
('is_image_blocked', 0), ('has_link', 1971), ('is_spam', 10)]
(51.98s) Found 49659 objs (match: 16563) (db writes: 51180) (range: 72500921
~ 72550921), (avg 16.9 mins/million) - [('is_checked', 49659),
('is_image_blocked', 0), ('has_link', 1517), ('is_spam', 4)]

Could you let me know if those benchmarks are better/worse than using DSE?
I'd be interested to see the comparison!

Cal

Post by Thomas Weholt
Yes! I'm in.
Out of curiosity: When inserting lots of data, how do you do it? Using
the orm? Have you looked at http://pypi.python.org/pypi/dse/2.1.0 ? I
wrote DSE to solve inserting/updating huge sets of data, but if
there's a better way to do it that would be especially interesting to
hear more about ( and sorry for the self promotion ).
Regards,
Thomas
On Wed, Jun 22, 2011 at 3:15 PM, Cal Leeming [Simplicity Media Ltd]

Post by Cal Leeming [Simplicity Media Ltd]
Hi all,
Some of you may have noticed, in the last few months I've done quite a

few

Post by Cal Leeming [Simplicity Media Ltd]
posts/snippets about handling large data sets in Django. At the end of

this

Post by Cal Leeming [Simplicity Media Ltd]
month (after what seems like a lifetime of trial and error), we're

finally

Post by Cal Leeming [Simplicity Media Ltd]
going to be releasing a new site which holds around 40mil+ rows of data,
grows by about 300-500k rows each day, handles 5GB of uploads per day,

and

Post by Cal Leeming [Simplicity Media Ltd]
can handle around 1024 requests per second on stress test on a moderately
spec'd server.
As the entire thing is written in Django (and a bunch of other open

source

Post by Cal Leeming [Simplicity Media Ltd]
products), I'd really like to give something back to the

community. (stack

Post by Cal Leeming [Simplicity Media Ltd]
incls Celery/RabbitMQ/Sphinx SE/PYQuery/Percona
MySQL/NGINX/supervisord/debian etc)
Therefore, I'd like to see if there would be any interest in webcast in
which I would explain how we handle such large amounts of data, the trial
and error processes we went through, some really neat tricks we've done

to

Post by Cal Leeming [Simplicity Media Ltd]
avoid bottlenecks, our own approach to smart content filtering, and some

of

Post by Cal Leeming [Simplicity Media Ltd]
the valuable lessons we have learned. The webcast would be completely

free

Post by Cal Leeming [Simplicity Media Ltd]
of charge, last a couple of hours (with a short break) and anyone can
attend. I'd also offer up a Q&A session at the end.
If you're interested, please reply on-list so others can see.
Thanks
Cal
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

--
Mvh/Best regards,
Thomas Weholt
http://www.weholt.org
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

[Thomas Weholt]

On Wed, Jun 22, 2011 at 3:36 PM, Cal Leeming [Simplicity Media Ltd]

Post by Cal Leeming [Simplicity Media Ltd]
Hey Thomas,
Yeah we actually spoke a little while ago about DSE. In the end, we actually
used a custom approach which analyses data in blocks of 50k rows, builds a
list of rows which need changing to the same value, then applied them in
bulk using update() + F().

Hmmm, what do you mean by "bulk using update() + F()? Something like
"update sometable set somefield1 = somevalue1, somefield2 = somevalue2
where id in (1,2,3 .....)" ? Does "avg 13.8 mins/million" mean you
processed 13.8 million rows pr minute? What kind of hardware did you
use?

Thomas

Post by Cal Leeming [Simplicity Media Ltd]
(42.11s) Found 49426 objs (match: 16107) (db writes: 50847) (range: 72300921
~ 72350921), (avg 13.8 mins/million) - [('is_checked', 49426),
('is_image_blocked', 0), ('has_link', 1420), ('is_spam', 1)]
(44.50s) Found 49481 objs (match: 16448) (db writes: 50764) (range: 72350921
~ 72400921), (avg 14.6 mins/million) - [('is_checked', 49481),
('is_image_blocked', 0), ('has_link', 1283), ('is_spam', 0)]
(55.78s) Found 49627 objs (match: 18516) (db writes: 50832) (range: 72400921
~ 72450921), (avg 18.3 mins/million) - [('is_checked', 49627),
('is_image_blocked', 0), ('has_link', 1205), ('is_spam', 0)]
(42.03s) Found 49674 objs (match: 17244) (db writes: 51655) (range: 72450921
~ 72500921), (avg 13.6 mins/million) - [('is_checked', 49674),
('is_image_blocked', 0), ('has_link', 1971), ('is_spam', 10)]
(51.98s) Found 49659 objs (match: 16563) (db writes: 51180) (range: 72500921
~ 72550921), (avg 16.9 mins/million) - [('is_checked', 49659),
('is_image_blocked', 0), ('has_link', 1517), ('is_spam', 4)]
Could you let me know if those benchmarks are better/worse than using DSE?
I'd be interested to see the comparison!
Cal

Post by Thomas Weholt
Yes! I'm in.
Out of curiosity: When inserting lots of data, how do you do it? Using
the orm? Have you looked at http://pypi.python.org/pypi/dse/2.1.0 ? I
wrote DSE to solve inserting/updating huge sets of data, but if
there's a better way to do it that would be especially interesting to
hear more about ( and sorry for the self promotion ).
Regards,
Thomas
On Wed, Jun 22, 2011 at 3:15 PM, Cal Leeming [Simplicity Media Ltd]

Post by Cal Leeming [Simplicity Media Ltd]
Hi all,
Some of you may have noticed, in the last few months I've done quite a few
posts/snippets about handling large data sets in Django. At the end of this
month (after what seems like a lifetime of trial and error), we're finally
going to be releasing a new site which holds around 40mil+ rows of data,
grows by about 300-500k rows each day, handles 5GB of uploads per day, and
can handle around 1024 requests per second on stress test on a moderately
spec'd server.
As the entire thing is written in Django (and a bunch of other open source
products), I'd really like to give something back to the
community. (stack
incls Celery/RabbitMQ/Sphinx SE/PYQuery/Percona
MySQL/NGINX/supervisord/debian etc)
Therefore, I'd like to see if there would be any interest in webcast in
which I would explain how we handle such large amounts of data, the trial
and error processes we went through, some really neat tricks we've done to
avoid bottlenecks, our own approach to smart content filtering, and some of
the valuable lessons we have learned. The webcast would be completely free
of charge, last a couple of hours (with a short break) and anyone can
attend. I'd also offer up a Q&A session at the end.
If you're interested, please reply on-list so others can see.
Thanks
Cal
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

--
Mvh/Best regards,
Thomas Weholt
http://www.weholt.org
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

--
Mvh/Best regards,
Thomas Weholt
http://www.weholt.org

[Cal Leeming [Simplicity Media Ltd]]

Sorry, let me explain a little better.

(51.98s) Found 49659 objs (match: 16563) (db writes: 51180) (range:
72500921 ~ 72550921), (avg 16.9 mins/million) - [('is_checked',
49659), ('is_image_blocked', 0), ('has_link', 1517), ('is_spam', 4)]

map(lambda x: (x[0], len(x[1])), _obj_incs.iteritems()) = [('is_checked',
49659), ('is_image_blocked', 0), ('has_link', 1517), ('is_spam', 4)]

In the above example, it has found 49659 rows which need 'is_checked'
changing to the value '1' (same principle applied to the other 3), giving a
total of 51,130 database writes, split into 4 queries.

Those 4 fields have the IDs assigned to them:

if _f == 'block_images':

_obj_incs.get('is_image_blocked').append(_hit_id)
if _parent_id:

_obj_incs.get('is_image_blocked').append(_parent_id)

Then I loop through those fields, and do an update() using the necessary
IDs:

# now apply the obj changes in bulk (massive speed
improvements)
for _key, _value in _obj_incs.iteritems():
# update the child object
Post.objects.filter(
id__in = _value
).update(
**{
_key : 1
}
)

So in simple terms, we're not doing 51 thousand update queries, instead
we're grouping them into bulk queries based on the row to be updated. It
doesn't yet to grouping based on key AND value, simply because we didn't
need it at the time, but if we release the code for public use,
we'd definitely add this in.

Hope this makes sense, let me know if I didn't explain it very well lol.

Cal

Post by Thomas Weholt
On Wed, Jun 22, 2011 at 3:36 PM, Cal Leeming [Simplicity Media Ltd]

Post by Cal Leeming [Simplicity Media Ltd]
Hey Thomas,
Yeah we actually spoke a little while ago about DSE. In the end, we

actually

Post by Cal Leeming [Simplicity Media Ltd]
used a custom approach which analyses data in blocks of 50k rows, builds

a

Post by Cal Leeming [Simplicity Media Ltd]
list of rows which need changing to the same value, then applied them in
bulk using update() + F().

Hmmm, what do you mean by "bulk using update() + F()? Something like
"update sometable set somefield1 = somevalue1, somefield2 = somevalue2
where id in (1,2,3 .....)" ? Does "avg 13.8 mins/million" mean you
processed 13.8 million rows pr minute? What kind of hardware did you
use?
Thomas
72300921

Post by Cal Leeming [Simplicity Media Ltd]
~ 72350921), (avg 13.8 mins/million) - [('is_checked', 49426),
('is_image_blocked', 0), ('has_link', 1420), ('is_spam', 1)]

72350921

Post by Cal Leeming [Simplicity Media Ltd]
~ 72400921), (avg 14.6 mins/million) - [('is_checked', 49481),
('is_image_blocked', 0), ('has_link', 1283), ('is_spam', 0)]

72400921

Post by Cal Leeming [Simplicity Media Ltd]
~ 72450921), (avg 18.3 mins/million) - [('is_checked', 49627),
('is_image_blocked', 0), ('has_link', 1205), ('is_spam', 0)]

72450921

Post by Cal Leeming [Simplicity Media Ltd]
~ 72500921), (avg 13.6 mins/million) - [('is_checked', 49674),
('is_image_blocked', 0), ('has_link', 1971), ('is_spam', 10)]

72500921

Post by Cal Leeming [Simplicity Media Ltd]
~ 72550921), (avg 16.9 mins/million) - [('is_checked', 49659),
('is_image_blocked', 0), ('has_link', 1517), ('is_spam', 4)]
Could you let me know if those benchmarks are better/worse than using

DSE?

Post by Cal Leeming [Simplicity Media Ltd]
I'd be interested to see the comparison!
Cal

Post by Thomas Weholt
Yes! I'm in.
Out of curiosity: When inserting lots of data, how do you do it? Using
the orm? Have you looked at http://pypi.python.org/pypi/dse/2.1.0 ? I
wrote DSE to solve inserting/updating huge sets of data, but if
there's a better way to do it that would be especially interesting to
hear more about ( and sorry for the self promotion ).
Regards,
Thomas
On Wed, Jun 22, 2011 at 3:15 PM, Cal Leeming [Simplicity Media Ltd]

Post by Cal Leeming [Simplicity Media Ltd]
Hi all,
Some of you may have noticed, in the last few months I've done quite a few
posts/snippets about handling large data sets in Django. At the end of this
month (after what seems like a lifetime of trial and error), we're finally
going to be releasing a new site which holds around 40mil+ rows of

data,

Post by Cal Leeming [Simplicity Media Ltd]

Post by Thomas Weholt

Post by Cal Leeming [Simplicity Media Ltd]
grows by about 300-500k rows each day, handles 5GB of uploads per day, and
can handle around 1024 requests per second on stress test on a moderately
spec'd server.
As the entire thing is written in Django (and a bunch of other open source
products), I'd really like to give something back to the
community. (stack
incls Celery/RabbitMQ/Sphinx SE/PYQuery/Percona
MySQL/NGINX/supervisord/debian etc)
Therefore, I'd like to see if there would be any interest in webcast

in

Post by Cal Leeming [Simplicity Media Ltd]

Post by Thomas Weholt

Post by Cal Leeming [Simplicity Media Ltd]
which I would explain how we handle such large amounts of data, the trial
and error processes we went through, some really neat tricks we've

done

Post by Cal Leeming [Simplicity Media Ltd]

Post by Thomas Weholt

Post by Cal Leeming [Simplicity Media Ltd]
to
avoid bottlenecks, our own approach to smart content filtering, and

some

Post by Cal Leeming [Simplicity Media Ltd]

Post by Thomas Weholt

Post by Cal Leeming [Simplicity Media Ltd]
of
the valuable lessons we have learned. The webcast would be completely free
of charge, last a couple of hours (with a short break) and anyone can
attend. I'd also offer up a Q&A session at the end.
If you're interested, please reply on-list so others can see.
Thanks
Cal
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

--
Mvh/Best regards,
Thomas Weholt
http://www.weholt.org
--
You received this message because you are subscribed to the Google

Groups

Post by Cal Leeming [Simplicity Media Ltd]

Post by Thomas Weholt
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

--
Mvh/Best regards,
Thomas Weholt
http://www.weholt.org
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

[Cal Leeming [Simplicity Media Ltd]]

Also, the 13.8 minutes per million, is basically a benchmark based on the
amount of db writes, and the total amount of time it took to execute (which
was 51s).

Please also note, this code is doing a *heavy* amount of content analysis,
but if you were to strip that out, the only overheads would be the
map/filter/lambda, the time it takes to transmit to MySQL, and the time it
takes for MySQL to perform the writes.

The database hardware spec is:

1x X3440 quad core (2 cores assigned to MySQL).
12GB memory (4 GB assigned to MySQL).
/var/lib/mysql mapped to 2x Intel M3 SSD drives in RAID 1.

Cal


On Wed, Jun 22, 2011 at 2:52 PM, Cal Leeming [Simplicity Media Ltd] <

Post by Cal Leeming [Simplicity Media Ltd]
Sorry, let me explain a little better.
72500921 ~ 72550921), (avg 16.9 mins/million) - [('is_checked',
49659), ('is_image_blocked', 0), ('has_link', 1517), ('is_spam', 4)]
map(lambda x: (x[0], len(x[1])), _obj_incs.iteritems()) = [('is_checked',
49659), ('is_image_blocked', 0), ('has_link', 1517), ('is_spam', 4)]
In the above example, it has found 49659 rows which need 'is_checked'
changing to the value '1' (same principle applied to the other 3), giving a
total of 51,130 database writes, split into 4 queries.
_obj_incs.get('is_image_blocked').append(_hit_id)
_obj_incs.get('is_image_blocked').append(_parent_id)
Then I loop through those fields, and do an update() using the necessary
# now apply the obj changes in bulk (massive speed
improvements)
# update the child object
Post.objects.filter(
id__in = _value
).update(
**{
_key : 1
}
)
So in simple terms, we're not doing 51 thousand update queries, instead
we're grouping them into bulk queries based on the row to be updated. It
doesn't yet to grouping based on key AND value, simply because we didn't
need it at the time, but if we release the code for public use,
we'd definitely add this in.
Hope this makes sense, let me know if I didn't explain it very well lol.
Cal

Post by Thomas Weholt
On Wed, Jun 22, 2011 at 3:36 PM, Cal Leeming [Simplicity Media Ltd]

Post by Cal Leeming [Simplicity Media Ltd]
Hey Thomas,
Yeah we actually spoke a little while ago about DSE. In the end, we

actually

Post by Cal Leeming [Simplicity Media Ltd]
used a custom approach which analyses data in blocks of 50k rows, builds

a

Post by Cal Leeming [Simplicity Media Ltd]
list of rows which need changing to the same value, then applied them in
bulk using update() + F().

Hmmm, what do you mean by "bulk using update() + F()? Something like
"update sometable set somefield1 = somevalue1, somefield2 = somevalue2
where id in (1,2,3 .....)" ? Does "avg 13.8 mins/million" mean you
processed 13.8 million rows pr minute? What kind of hardware did you
use?
Thomas
72300921

Post by Cal Leeming [Simplicity Media Ltd]
~ 72350921), (avg 13.8 mins/million) - [('is_checked', 49426),
('is_image_blocked', 0), ('has_link', 1420), ('is_spam', 1)]

72350921

Post by Cal Leeming [Simplicity Media Ltd]
~ 72400921), (avg 14.6 mins/million) - [('is_checked', 49481),
('is_image_blocked', 0), ('has_link', 1283), ('is_spam', 0)]

72400921

Post by Cal Leeming [Simplicity Media Ltd]
~ 72450921), (avg 18.3 mins/million) - [('is_checked', 49627),
('is_image_blocked', 0), ('has_link', 1205), ('is_spam', 0)]

72450921

Post by Cal Leeming [Simplicity Media Ltd]
~ 72500921), (avg 13.6 mins/million) - [('is_checked', 49674),
('is_image_blocked', 0), ('has_link', 1971), ('is_spam', 10)]

72500921

Post by Cal Leeming [Simplicity Media Ltd]
~ 72550921), (avg 16.9 mins/million) - [('is_checked', 49659),
('is_image_blocked', 0), ('has_link', 1517), ('is_spam', 4)]
Could you let me know if those benchmarks are better/worse than using

DSE?

Post by Cal Leeming [Simplicity Media Ltd]
I'd be interested to see the comparison!
Cal

Post by Thomas Weholt
Yes! I'm in.
Out of curiosity: When inserting lots of data, how do you do it? Using
the orm? Have you looked at http://pypi.python.org/pypi/dse/2.1.0 ? I
wrote DSE to solve inserting/updating huge sets of data, but if
there's a better way to do it that would be especially interesting to
hear more about ( and sorry for the self promotion ).
Regards,
Thomas
On Wed, Jun 22, 2011 at 3:15 PM, Cal Leeming [Simplicity Media Ltd]

Post by Cal Leeming [Simplicity Media Ltd]
Hi all,
Some of you may have noticed, in the last few months I've done quite

a

Post by Cal Leeming [Simplicity Media Ltd]

Post by Thomas Weholt

Post by Cal Leeming [Simplicity Media Ltd]
few
posts/snippets about handling large data sets in Django. At the end

of

Post by Cal Leeming [Simplicity Media Ltd]

Post by Thomas Weholt

Post by Cal Leeming [Simplicity Media Ltd]
this
month (after what seems like a lifetime of trial and error), we're finally
going to be releasing a new site which holds around 40mil+ rows of

data,

Post by Cal Leeming [Simplicity Media Ltd]

Post by Thomas Weholt

Post by Cal Leeming [Simplicity Media Ltd]
grows by about 300-500k rows each day, handles 5GB of uploads per

day,

Post by Cal Leeming [Simplicity Media Ltd]

Post by Thomas Weholt

Post by Cal Leeming [Simplicity Media Ltd]
and
can handle around 1024 requests per second on stress test on a moderately
spec'd server.
As the entire thing is written in Django (and a bunch of other open source
products), I'd really like to give something back to the
community. (stack
incls Celery/RabbitMQ/Sphinx SE/PYQuery/Percona
MySQL/NGINX/supervisord/debian etc)
Therefore, I'd like to see if there would be any interest in webcast

in

Post by Cal Leeming [Simplicity Media Ltd]

Post by Thomas Weholt

Post by Cal Leeming [Simplicity Media Ltd]
which I would explain how we handle such large amounts of data, the trial
and error processes we went through, some really neat tricks we've

done

Post by Cal Leeming [Simplicity Media Ltd]

Post by Thomas Weholt

Post by Cal Leeming [Simplicity Media Ltd]
to
avoid bottlenecks, our own approach to smart content filtering, and

some

Post by Cal Leeming [Simplicity Media Ltd]

Post by Thomas Weholt

Post by Cal Leeming [Simplicity Media Ltd]
of
the valuable lessons we have learned. The webcast would be completely free
of charge, last a couple of hours (with a short break) and anyone can
attend. I'd also offer up a Q&A session at the end.
If you're interested, please reply on-list so others can see.
Thanks
Cal
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

--
Mvh/Best regards,
Thomas Weholt
http://www.weholt.org
--
You received this message because you are subscribed to the Google

Groups

Post by Cal Leeming [Simplicity Media Ltd]

Post by Thomas Weholt
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

--
You received this message because you are subscribed to the Google

Groups

Post by Cal Leeming [Simplicity Media Ltd]
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

--
Mvh/Best regards,
Thomas Weholt
http://www.weholt.org
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

[Andre Terra]

Hello, Cal

First of all, congrats on the newborn! The Django community will surely
benefit from having yet another success story, especially considering how
big this project sounds. Is there any chance you could open-source some of
your custom made improvements so that they could eventually be merged to
trunk?

I definitely noticed how you mentioned large dbs in the past few months. I,
along with many others I assume, would surely like to attend the webcast,
with the only impediment being my schedule/timezone.

I recently asked about working with temporary tables for filtering/grouping
data from uploads and inserting queries from that temporary table onto a
permanent database. To make matters worse, I wanted to make this as flexible
as possible (i.e. dynamic models) so that everything could be managed from a
web app. Do you have any experience you could share about any of these use
cases? As far as I know, there's nothing in the ORM that replicates
PostgreSQL's CREATE TEMPORARY TABLE. My experience with SQL is rather
limited, but from asking around, it seems like my project could indeed
benefit from such a feature. If I had to guess, I would assume other DBMSs
would offer something similar, but being limited to Postgres is okay for me,
for now, anyway.



Cheers,
André


On Wed, Jun 22, 2011 at 10:56 AM, Cal Leeming [Simplicity Media Ltd] <

Post by Cal Leeming [Simplicity Media Ltd]
Also, the 13.8 minutes per million, is basically a benchmark based on the
amount of db writes, and the total amount of time it took to execute (which
was 51s).
Please also note, this code is doing a *heavy* amount of content analysis,
but if you were to strip that out, the only overheads would be the
map/filter/lambda, the time it takes to transmit to MySQL, and the time it
takes for MySQL to perform the writes.
1x X3440 quad core (2 cores assigned to MySQL).
12GB memory (4 GB assigned to MySQL).
/var/lib/mysql mapped to 2x Intel M3 SSD drives in RAID 1.
Cal
On Wed, Jun 22, 2011 at 2:52 PM, Cal Leeming [Simplicity Media Ltd] <

Post by Cal Leeming [Simplicity Media Ltd]
Sorry, let me explain a little better.
72500921 ~ 72550921), (avg 16.9 mins/million) - [('is_checked',
49659), ('is_image_blocked', 0), ('has_link', 1517), ('is_spam', 4)]
map(lambda x: (x[0], len(x[1])), _obj_incs.iteritems()) = [('is_checked',
49659), ('is_image_blocked', 0), ('has_link', 1517), ('is_spam', 4)]
In the above example, it has found 49659 rows which need 'is_checked'
changing to the value '1' (same principle applied to the other 3), giving a
total of 51,130 database writes, split into 4 queries.
_obj_incs.get('is_image_blocked').append(_hit_id)
_obj_incs.get('is_image_blocked').append(_parent_id)
Then I loop through those fields, and do an update() using the necessary
# now apply the obj changes in bulk (massive speed
improvements)
# update the child object
Post.objects.filter(
id__in = _value
).update(
**{
_key : 1
}
)
So in simple terms, we're not doing 51 thousand update queries, instead
we're grouping them into bulk queries based on the row to be updated. It
doesn't yet to grouping based on key AND value, simply because we didn't
need it at the time, but if we release the code for public use,
we'd definitely add this in.
Hope this makes sense, let me know if I didn't explain it very well lol.
Cal

Post by Thomas Weholt
On Wed, Jun 22, 2011 at 3:36 PM, Cal Leeming [Simplicity Media Ltd]

Post by Cal Leeming [Simplicity Media Ltd]
Hey Thomas,
Yeah we actually spoke a little while ago about DSE. In the end, we

actually

Post by Cal Leeming [Simplicity Media Ltd]
used a custom approach which analyses data in blocks of 50k rows,

builds a

Post by Cal Leeming [Simplicity Media Ltd]
list of rows which need changing to the same value, then applied them

in

Post by Cal Leeming [Simplicity Media Ltd]
bulk using update() + F().

Hmmm, what do you mean by "bulk using update() + F()? Something like
"update sometable set somefield1 = somevalue1, somefield2 = somevalue2
where id in (1,2,3 .....)" ? Does "avg 13.8 mins/million" mean you
processed 13.8 million rows pr minute? What kind of hardware did you
use?
Thomas
72300921

Post by Cal Leeming [Simplicity Media Ltd]
~ 72350921), (avg 13.8 mins/million) - [('is_checked', 49426),
('is_image_blocked', 0), ('has_link', 1420), ('is_spam', 1)]

72350921

Post by Cal Leeming [Simplicity Media Ltd]
~ 72400921), (avg 14.6 mins/million) - [('is_checked', 49481),
('is_image_blocked', 0), ('has_link', 1283), ('is_spam', 0)]

72400921

Post by Cal Leeming [Simplicity Media Ltd]
~ 72450921), (avg 18.3 mins/million) - [('is_checked', 49627),
('is_image_blocked', 0), ('has_link', 1205), ('is_spam', 0)]

72450921

Post by Cal Leeming [Simplicity Media Ltd]
~ 72500921), (avg 13.6 mins/million) - [('is_checked', 49674),
('is_image_blocked', 0), ('has_link', 1971), ('is_spam', 10)]

72500921

Post by Cal Leeming [Simplicity Media Ltd]
~ 72550921), (avg 16.9 mins/million) - [('is_checked', 49659),
('is_image_blocked', 0), ('has_link', 1517), ('is_spam', 4)]
Could you let me know if those benchmarks are better/worse than using

DSE?

Post by Cal Leeming [Simplicity Media Ltd]
I'd be interested to see the comparison!
Cal
On Wed, Jun 22, 2011 at 2:31 PM, Thomas Weholt <

Post by Thomas Weholt
Yes! I'm in.
Out of curiosity: When inserting lots of data, how do you do it? Using
the orm? Have you looked at http://pypi.python.org/pypi/dse/2.1.0 ? I
wrote DSE to solve inserting/updating huge sets of data, but if
there's a better way to do it that would be especially interesting to
hear more about ( and sorry for the self promotion ).
Regards,
Thomas
On Wed, Jun 22, 2011 at 3:15 PM, Cal Leeming [Simplicity Media Ltd]

Post by Cal Leeming [Simplicity Media Ltd]
Hi all,
Some of you may have noticed, in the last few months I've done quite

a

Post by Cal Leeming [Simplicity Media Ltd]

Post by Thomas Weholt

Post by Cal Leeming [Simplicity Media Ltd]
few
posts/snippets about handling large data sets in Django. At the end

of

Post by Cal Leeming [Simplicity Media Ltd]

Post by Thomas Weholt

Post by Cal Leeming [Simplicity Media Ltd]
this
month (after what seems like a lifetime of trial and error), we're finally
going to be releasing a new site which holds around 40mil+ rows of

data,

Post by Cal Leeming [Simplicity Media Ltd]

Post by Thomas Weholt

Post by Cal Leeming [Simplicity Media Ltd]
grows by about 300-500k rows each day, handles 5GB of uploads per

day,

Post by Cal Leeming [Simplicity Media Ltd]

Post by Thomas Weholt

Post by Cal Leeming [Simplicity Media Ltd]
and
can handle around 1024 requests per second on stress test on a moderately
spec'd server.
As the entire thing is written in Django (and a bunch of other open source
products), I'd really like to give something back to the community. (stack
incls Celery/RabbitMQ/Sphinx SE/PYQuery/Percona
MySQL/NGINX/supervisord/debian etc)
Therefore, I'd like to see if there would be any interest in webcast

in

Post by Cal Leeming [Simplicity Media Ltd]

Post by Thomas Weholt

Post by Cal Leeming [Simplicity Media Ltd]
which I would explain how we handle such large amounts of data, the trial
and error processes we went through, some really neat tricks we've

done

Post by Cal Leeming [Simplicity Media Ltd]

Post by Thomas Weholt

Post by Cal Leeming [Simplicity Media Ltd]
to
avoid bottlenecks, our own approach to smart content filtering, and

some

Post by Cal Leeming [Simplicity Media Ltd]

Post by Thomas Weholt

Post by Cal Leeming [Simplicity Media Ltd]
of
the valuable lessons we have learned. The webcast would be

completely

Post by Cal Leeming [Simplicity Media Ltd]

Post by Thomas Weholt

Post by Cal Leeming [Simplicity Media Ltd]
free
of charge, last a couple of hours (with a short break) and anyone

can

Post by Cal Leeming [Simplicity Media Ltd]

Post by Thomas Weholt

Post by Cal Leeming [Simplicity Media Ltd]
attend. I'd also offer up a Q&A session at the end.
If you're interested, please reply on-list so others can see.
Thanks
Cal
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

--
Mvh/Best regards,
Thomas Weholt
http://www.weholt.org
--
You received this message because you are subscribed to the Google

Groups

Post by Cal Leeming [Simplicity Media Ltd]

Post by Thomas Weholt
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

--
You received this message because you are subscribed to the Google

Groups

Post by Cal Leeming [Simplicity Media Ltd]
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

--
Mvh/Best regards,
Thomas Weholt
http://www.weholt.org
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

[Cal Leeming [Simplicity Media Ltd]]

Post by Andre Terra
Hello, Cal
First of all, congrats on the newborn! The Django community will surely
benefit from having yet another success story, especially considering how
big this project sounds. Is there any chance you could open-source some of
your custom made improvements so that they could eventually be merged to
trunk?

Thank you! Yeah, the plan is to release as much of the improvements as open
source as possible. Although I'd rely heavily on the community to make them
'patch worthy' for the core, as the amount of spare time I have is somewhat
limited.

The improvements list is growing by the day, and I usually try and post as
many snippets as I can, and/or tickets etc.

It sounds like Thomas's DSE might be the perfect place for the bulk update
code too.

Post by Andre Terra
I definitely noticed how you mentioned large dbs in the past few months. I,
along with many others I assume, would surely like to attend the webcast,
with the only impediment being my schedule/timezone.

Once we've got a list of all the people who want to attend, I'll send out a
mail asking for everyones timezone and availability, so we can figure out
what is best for everyone.

Post by Andre Terra
I recently asked about working with temporary tables for filtering/grouping
data from uploads and inserting queries from that temporary table onto a
permanent database. To make matters worse, I wanted to make this as flexible
as possible (i.e. dynamic models) so that everything could be managed from a
web app. Do you have any experience you could share about any of these use
cases? As far as I know, there's nothing in the ORM that replicates
PostgreSQL's CREATE TEMPORARY TABLE. My experience with SQL is rather
limited, but from asking around, it seems like my project could indeed
benefit from such a feature. If I had to guess, I would assume other DBMSs
would offer something similar, but being limited to Postgres is okay for me,
for now, anyway.

I haven't had any exposure to Postgres, but my experience with temporary
tables hasn't been a nice one (in regards to MySQL at least). MySQL has many
gotchas when it comes to temporary tables and indexing, and on more than one
occasion, I found it was actually quicker to analyse/mangle/re-insert the
data via Python code, than it was to attempt the modifications within MySQL
using a temporary table.

It really does depend on what your data is, and what you want to do with it,
which can make planning ahead somewhat tedious lol.

For our stuff, when we need to do bulk modifications, we have a filtering
rules list which is ran every hour against new rows (with is_checked=1 set
on rows which have been checked). We then use bulk queries of 50k (id >= 0
AND id < 50000), rather than using LIMIT/OFFSET (because LIMIT/OFFSET gets
slower and slower the larger the result set). Those queries are
analysed/mangled within a transaction, and bulk updated using the method
mentioned in the reply to Thomas.

Sadly though, I can't say if the methods we use would be suitable for you,
as we haven't tried it against Postgres, and we've only tested it against
our own data set + requirements. This is what I mean by trial and error,
it's a pain in the ass :)

Post by Andre Terra
Cheers,
André
On Wed, Jun 22, 2011 at 10:56 AM, Cal Leeming [Simplicity Media Ltd] <

Post by Cal Leeming [Simplicity Media Ltd]
Also, the 13.8 minutes per million, is basically a benchmark based on the
amount of db writes, and the total amount of time it took to execute (which
was 51s).
Please also note, this code is doing a *heavy* amount of content analysis,
but if you were to strip that out, the only overheads would be the
map/filter/lambda, the time it takes to transmit to MySQL, and the time it
takes for MySQL to perform the writes.
1x X3440 quad core (2 cores assigned to MySQL).
12GB memory (4 GB assigned to MySQL).
/var/lib/mysql mapped to 2x Intel M3 SSD drives in RAID 1.
Cal
On Wed, Jun 22, 2011 at 2:52 PM, Cal Leeming [Simplicity Media Ltd] <

Post by Cal Leeming [Simplicity Media Ltd]
Sorry, let me explain a little better.
72500921 ~ 72550921), (avg 16.9 mins/million) - [('is_checked',
49659), ('is_image_blocked', 0), ('has_link', 1517), ('is_spam', 4)]
map(lambda x: (x[0], len(x[1])), _obj_incs.iteritems()) = [('is_checked',
49659), ('is_image_blocked', 0), ('has_link', 1517), ('is_spam', 4)]
In the above example, it has found 49659 rows which need 'is_checked'
changing to the value '1' (same principle applied to the other 3), giving a
total of 51,130 database writes, split into 4 queries.
_obj_incs.get('is_image_blocked').append(_hit_id)
_obj_incs.get('is_image_blocked').append(_parent_id)
Then I loop through those fields, and do an update() using the necessary
# now apply the obj changes in bulk (massive speed
improvements)
# update the child object
Post.objects.filter(
id__in = _value
).update(
**{
_key : 1
}
)
So in simple terms, we're not doing 51 thousand update queries, instead
we're grouping them into bulk queries based on the row to be updated. It
doesn't yet to grouping based on key AND value, simply because we didn't
need it at the time, but if we release the code for public use,
we'd definitely add this in.
Hope this makes sense, let me know if I didn't explain it very well lol.
Cal

Post by Thomas Weholt
On Wed, Jun 22, 2011 at 3:36 PM, Cal Leeming [Simplicity Media Ltd]

Post by Cal Leeming [Simplicity Media Ltd]
Hey Thomas,
Yeah we actually spoke a little while ago about DSE. In the end, we

actually

Post by Cal Leeming [Simplicity Media Ltd]
used a custom approach which analyses data in blocks of 50k rows,

builds a

Post by Cal Leeming [Simplicity Media Ltd]
list of rows which need changing to the same value, then applied them

in

Post by Cal Leeming [Simplicity Media Ltd]
bulk using update() + F().

Hmmm, what do you mean by "bulk using update() + F()? Something like
"update sometable set somefield1 = somevalue1, somefield2 = somevalue2
where id in (1,2,3 .....)" ? Does "avg 13.8 mins/million" mean you
processed 13.8 million rows pr minute? What kind of hardware did you
use?
Thomas
72300921

Post by Cal Leeming [Simplicity Media Ltd]
~ 72350921), (avg 13.8 mins/million) - [('is_checked', 49426),
('is_image_blocked', 0), ('has_link', 1420), ('is_spam', 1)]

72350921

Post by Cal Leeming [Simplicity Media Ltd]
~ 72400921), (avg 14.6 mins/million) - [('is_checked', 49481),
('is_image_blocked', 0), ('has_link', 1283), ('is_spam', 0)]

72400921

Post by Cal Leeming [Simplicity Media Ltd]
~ 72450921), (avg 18.3 mins/million) - [('is_checked', 49627),
('is_image_blocked', 0), ('has_link', 1205), ('is_spam', 0)]

72450921

Post by Cal Leeming [Simplicity Media Ltd]
~ 72500921), (avg 13.6 mins/million) - [('is_checked', 49674),
('is_image_blocked', 0), ('has_link', 1971), ('is_spam', 10)]

72500921

Post by Cal Leeming [Simplicity Media Ltd]
~ 72550921), (avg 16.9 mins/million) - [('is_checked', 49659),
('is_image_blocked', 0), ('has_link', 1517), ('is_spam', 4)]
Could you let me know if those benchmarks are better/worse than using

DSE?

Post by Cal Leeming [Simplicity Media Ltd]
I'd be interested to see the comparison!
Cal
On Wed, Jun 22, 2011 at 2:31 PM, Thomas Weholt <

Post by Thomas Weholt
Yes! I'm in.
Out of curiosity: When inserting lots of data, how do you do it?

Using

Post by Cal Leeming [Simplicity Media Ltd]

Post by Thomas Weholt
the orm? Have you looked at http://pypi.python.org/pypi/dse/2.1.0 ?

I

Post by Cal Leeming [Simplicity Media Ltd]

Post by Thomas Weholt
wrote DSE to solve inserting/updating huge sets of data, but if
there's a better way to do it that would be especially interesting to
hear more about ( and sorry for the self promotion ).
Regards,
Thomas
On Wed, Jun 22, 2011 at 3:15 PM, Cal Leeming [Simplicity Media Ltd]

Post by Cal Leeming [Simplicity Media Ltd]
Hi all,
Some of you may have noticed, in the last few months I've done

quite a

Post by Cal Leeming [Simplicity Media Ltd]

Post by Thomas Weholt

Post by Cal Leeming [Simplicity Media Ltd]
few
posts/snippets about handling large data sets in Django. At the end

of

Post by Cal Leeming [Simplicity Media Ltd]

Post by Thomas Weholt

Post by Cal Leeming [Simplicity Media Ltd]
this
month (after what seems like a lifetime of trial and error), we're finally
going to be releasing a new site which holds around 40mil+ rows of

data,

Post by Cal Leeming [Simplicity Media Ltd]

Post by Thomas Weholt

Post by Cal Leeming [Simplicity Media Ltd]
grows by about 300-500k rows each day, handles 5GB of uploads per

day,

Post by Cal Leeming [Simplicity Media Ltd]

Post by Thomas Weholt

Post by Cal Leeming [Simplicity Media Ltd]
and
can handle around 1024 requests per second on stress test on a moderately
spec'd server.
As the entire thing is written in Django (and a bunch of other open source
products), I'd really like to give something back to the community. (stack
incls Celery/RabbitMQ/Sphinx SE/PYQuery/Percona
MySQL/NGINX/supervisord/debian etc)
Therefore, I'd like to see if there would be any interest in

webcast in

Post by Cal Leeming [Simplicity Media Ltd]

Post by Thomas Weholt

Post by Cal Leeming [Simplicity Media Ltd]
which I would explain how we handle such large amounts of data, the trial
and error processes we went through, some really neat tricks we've

done

Post by Cal Leeming [Simplicity Media Ltd]

Post by Thomas Weholt

Post by Cal Leeming [Simplicity Media Ltd]
to
avoid bottlenecks, our own approach to smart content filtering, and

some

Post by Cal Leeming [Simplicity Media Ltd]

Post by Thomas Weholt

Post by Cal Leeming [Simplicity Media Ltd]
of
the valuable lessons we have learned. The webcast would be

completely

Post by Cal Leeming [Simplicity Media Ltd]

Post by Thomas Weholt

Post by Cal Leeming [Simplicity Media Ltd]
free
of charge, last a couple of hours (with a short break) and anyone

can

Post by Cal Leeming [Simplicity Media Ltd]

Post by Thomas Weholt

Post by Cal Leeming [Simplicity Media Ltd]
attend. I'd also offer up a Q&A session at the end.
If you're interested, please reply on-list so others can see.
Thanks
Cal
--
You received this message because you are subscribed to the Google Groups
"Django users" group.

.

Post by Cal Leeming [Simplicity Media Ltd]

Post by Thomas Weholt

Post by Cal Leeming [Simplicity Media Ltd]
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

--
Mvh/Best regards,
Thomas Weholt
http://www.weholt.org
--
You received this message because you are subscribed to the Google

Groups

Post by Cal Leeming [Simplicity Media Ltd]

Post by Thomas Weholt
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

--
You received this message because you are subscribed to the Google

Groups

Post by Cal Leeming [Simplicity Media Ltd]
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

--
Mvh/Best regards,
Thomas Weholt
http://www.weholt.org
--
You received this message because you are subscribed to the Google
Groups "Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

[Andre Terra]

On Wed, Jun 22, 2011 at 11:47 AM, Cal Leeming [Simplicity Media Ltd] <

Post by Cal Leeming [Simplicity Media Ltd]

Post by Andre Terra
Hello, Cal
First of all, congrats on the newborn! The Django community will surely
benefit from having yet another success story, especially considering how
big this project sounds. Is there any chance you could open-source some of
your custom made improvements so that they could eventually be merged to
trunk?

Thank you! Yeah, the plan is to release as much of the improvements as open
source as possible. Although I'd rely heavily on the community to make them
'patch worthy' for the core, as the amount of spare time I have is somewhat
limited.
The improvements list is growing by the day, and I usually try and post as
many snippets as I can, and/or tickets etc.
It sounds like Thomas's DSE might be the perfect place for the bulk update
code too.

Thanks a lot for the quick reply. I'll keep my eyes open for the code, and
if unable to contribute with relevant modifications to the patches, I'll at
least try to doc and test them!

Post by Cal Leeming [Simplicity Media Ltd]
I definitely noticed how you mentioned large dbs in the past few months. I,

Post by Andre Terra
along with many others I assume, would surely like to attend the webcast,
with the only impediment being my schedule/timezone.

Once we've got a list of all the people who want to attend, I'll send out a
mail asking for everyones timezone and availability, so we can figure out
what is best for everyone.

Definitely write me up for the list of attendees, then!

Post by Cal Leeming [Simplicity Media Ltd]
I recently asked about working with temporary tables for

Post by Andre Terra
filtering/grouping data from uploads and inserting queries from that
temporary table onto a permanent database. To make matters worse, I wanted
to make this as flexible as possible (i.e. dynamic models) so that
everything could be managed from a web app. Do you have any experience you
could share about any of these use cases? As far as I know, there's nothing
in the ORM that replicates PostgreSQL's CREATE TEMPORARY TABLE. My
experience with SQL is rather limited, but from asking around, it seems like
my project could indeed benefit from such a feature. If I had to guess, I
would assume other DBMSs would offer something similar, but being limited to
Postgres is okay for me, for now, anyway.

I haven't had any exposure to Postgres, but my experience with temporary
tables hasn't been a nice one (in regards to MySQL at least). MySQL has many
gotchas when it comes to temporary tables and indexing, and on more than one
occasion, I found it was actually quicker to analyse/mangle/re-insert the
data via Python code, than it was to attempt the modifications within MySQL
using a temporary table.
It really does depend on what your data is, and what you want to do with
it, which can make planning ahead somewhat tedious lol.
For our stuff, when we need to do bulk modifications, we have a filtering
rules list which is ran every hour against new rows (with is_checked=1 set
on rows which have been checked). We then use bulk queries of 50k (id >= 0
AND id < 50000), rather than using LIMIT/OFFSET (because LIMIT/OFFSET gets
slower and slower the larger the result set). Those queries are
analysed/mangled within a transaction, and bulk updated using the method
mentioned in the reply to Thomas.
Sadly though, I can't say if the methods we use would be suitable for you,
as we haven't tried it against Postgres, and we've only tested it against
our own data set + requirements. This is what I mean by trial and error,
it's a pain in the ass :)

Thanks again for your enlightening input. Even with our different
requirements, this was actually quite relevant as far as solving several
doubts I had on how to go about this project.


Cheers,

André


On Wed, Jun 22, 2011 at 10:56 AM, Cal Leeming [Simplicity Media Ltd] <

Post by Cal Leeming [Simplicity Media Ltd]
Also, the 13.8 minutes per million, is basically a benchmark based on the
amount of db writes, and the total amount of time it took to execute (which
was 51s).
Please also note, this code is doing a *heavy* amount of content analysis,
but if you were to strip that out, the only overheads would be the
map/filter/lambda, the time it takes to transmit to MySQL, and the time it
takes for MySQL to perform the writes.
1x X3440 quad core (2 cores assigned to MySQL).
12GB memory (4 GB assigned to MySQL).
/var/lib/mysql mapped to 2x Intel M3 SSD drives in RAID 1.
Cal
On Wed, Jun 22, 2011 at 2:52 PM, Cal Leeming [Simplicity Media Ltd] <

Post by Andre Terra
Sorry, let me explain a little better.

(...)

[Thomas Weholt]

On Wed, Jun 22, 2011 at 4:47 PM, Cal Leeming [Simplicity Media Ltd]

Post by Cal Leeming [Simplicity Media Ltd]

Post by Andre Terra
Hello, Cal
First of all, congrats on the newborn! The Django community will surely
benefit from having yet another success story, especially considering how
big this project sounds. Is there any chance you could open-source some of
your custom made improvements so that they could eventually be merged to
trunk?

Thank you! Yeah, the plan is to release as much of the improvements as open
source as possible. Although I'd rely heavily on the community to make them
'patch worthy' for the core, as the amount of spare time I have is somewhat
limited.
The improvements list is growing by the day, and I usually try and post as
many snippets as I can, and/or tickets etc.
It sounds like Thomas's DSE might be the perfect place for the bulk update
code too.

FYI: Inspired by this discussion I've allready started on a similar
feature ( allthough somewhat simplified ) for DSE v2.2.0 and you're
right; the speed increase is huge using the method described here,
even compared to my current solution ( using cursor.executemany ),
which is considerably faster than the django orm allready. My testing
so far have been using postgresql, not sure how mysql will perform. I
expect to release DSE v.2.2.0 with this feature in the next few days.
--
Mvh/Best regards,
Thomas Weholt
http://www.weholt.org

[Cal Leeming [Simplicity Media Ltd]]

Nice!

Post by Thomas Weholt
On Wed, Jun 22, 2011 at 4:47 PM, Cal Leeming [Simplicity Media Ltd]

Post by Cal Leeming [Simplicity Media Ltd]

Post by Andre Terra
Hello, Cal
First of all, congrats on the newborn! The Django community will surely
benefit from having yet another success story, especially considering

how

Post by Cal Leeming [Simplicity Media Ltd]

Post by Andre Terra
big this project sounds. Is there any chance you could open-source some

of

Post by Cal Leeming [Simplicity Media Ltd]

Post by Andre Terra
your custom made improvements so that they could eventually be merged to
trunk?

Thank you! Yeah, the plan is to release as much of the improvements as

open

Post by Cal Leeming [Simplicity Media Ltd]
source as possible. Although I'd rely heavily on the community to make

them

Post by Cal Leeming [Simplicity Media Ltd]
'patch worthy' for the core, as the amount of spare time I have is

somewhat

Post by Cal Leeming [Simplicity Media Ltd]
limited.
The improvements list is growing by the day, and I usually try and post

as

Post by Cal Leeming [Simplicity Media Ltd]
many snippets as I can, and/or tickets etc.
It sounds like Thomas's DSE might be the perfect place for the bulk

update

Post by Cal Leeming [Simplicity Media Ltd]
code too.

FYI: Inspired by this discussion I've allready started on a similar
feature ( allthough somewhat simplified ) for DSE v2.2.0 and you're
right; the speed increase is huge using the method described here,
even compared to my current solution ( using cursor.executemany ),
which is considerably faster than the django orm allready. My testing
so far have been using postgresql, not sure how mysql will perform. I
expect to release DSE v.2.2.0 with this feature in the next few days.

Nice! Looking forward to seeing this :)

Post by Thomas Weholt
--
Mvh/Best regards,
Thomas Weholt
http://www.weholt.org
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

[william ratcliff]

Definitely looking forward to it!

On Wed, Jun 22, 2011 at 6:50 PM, Cal Leeming [Simplicity Media Ltd] <

Post by Cal Leeming [Simplicity Media Ltd]
Nice!

Post by Thomas Weholt
On Wed, Jun 22, 2011 at 4:47 PM, Cal Leeming [Simplicity Media Ltd]

Post by Cal Leeming [Simplicity Media Ltd]

Post by Andre Terra
Hello, Cal
First of all, congrats on the newborn! The Django community will surely
benefit from having yet another success story, especially considering

how

Post by Cal Leeming [Simplicity Media Ltd]

Post by Andre Terra
big this project sounds. Is there any chance you could open-source some

of

Post by Cal Leeming [Simplicity Media Ltd]

Post by Andre Terra
your custom made improvements so that they could eventually be merged

to

Post by Cal Leeming [Simplicity Media Ltd]

Post by Andre Terra
trunk?

Thank you! Yeah, the plan is to release as much of the improvements as

open

Post by Cal Leeming [Simplicity Media Ltd]
source as possible. Although I'd rely heavily on the community to make

them

Post by Cal Leeming [Simplicity Media Ltd]
'patch worthy' for the core, as the amount of spare time I have is

somewhat

Post by Cal Leeming [Simplicity Media Ltd]
limited.
The improvements list is growing by the day, and I usually try and post

as

Post by Cal Leeming [Simplicity Media Ltd]
many snippets as I can, and/or tickets etc.
It sounds like Thomas's DSE might be the perfect place for the bulk

update

Post by Cal Leeming [Simplicity Media Ltd]
code too.

FYI: Inspired by this discussion I've allready started on a similar
feature ( allthough somewhat simplified ) for DSE v2.2.0 and you're
right; the speed increase is huge using the method described here,
even compared to my current solution ( using cursor.executemany ),
which is considerably faster than the django orm allready. My testing
so far have been using postgresql, not sure how mysql will perform. I
expect to release DSE v.2.2.0 with this feature in the next few days.

Nice! Looking forward to seeing this :)

Post by Thomas Weholt
--
Mvh/Best regards,
Thomas Weholt
http://www.weholt.org
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

[Fatrix]

I would like to attend, too.

Thank you in advance!

CU
Fatrix

[Alex Grigorievskiy]

Hi, Cal and everybody,
I'd like to see the webcast too,
thanks very much for that :)

On Jun 23, 2:50 am, "Cal Leeming [Simplicity Media Ltd]"

Post by Cal Leeming [Simplicity Media Ltd]
Nice!

Post by Thomas Weholt
On Wed, Jun 22, 2011 at 4:47 PM, Cal Leeming [Simplicity Media Ltd]

Post by Cal Leeming [Simplicity Media Ltd]

Post by Andre Terra
Hello, Cal
First of all, congrats on the newborn! The Django community will surely
benefit from having yet another success story, especially considering

how

Post by Cal Leeming [Simplicity Media Ltd]

Post by Andre Terra
big this project sounds. Is there any chance you could open-source some

of

Post by Cal Leeming [Simplicity Media Ltd]

Post by Andre Terra
your custom made improvements so that they could eventually be merged to
trunk?

Thank you! Yeah, the plan is to release as much of the improvements as

open

Post by Cal Leeming [Simplicity Media Ltd]
source as possible. Although I'd rely heavily on the community to make

them

Post by Cal Leeming [Simplicity Media Ltd]
'patch worthy' for the core, as the amount of spare time I have is

somewhat

Post by Cal Leeming [Simplicity Media Ltd]
limited.
The improvements list is growing by the day, and I usually try and post

as

Post by Cal Leeming [Simplicity Media Ltd]
many snippets as I can, and/or tickets etc.
It sounds like Thomas's DSE might be the perfect place for the bulk

update

Post by Cal Leeming [Simplicity Media Ltd]
code too.

FYI: Inspired by this discussion I've allready started on a similar
feature ( allthough somewhat simplified ) for DSE v2.2.0 and you're
right; the speed increase is huge using the method described here,
even compared to my current solution ( using cursor.executemany ),
which is considerably faster than the django orm allready. My testing
so far have been using postgresql, not sure how mysql will perform. I
expect to release DSE v.2.2.0 with this feature in the next few days.

Nice! Looking forward to seeing this :)

Post by Thomas Weholt
--
Mvh/Best regards,
Thomas Weholt
http://www.weholt.org
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

[Matteius]

If I am unable to attend at the scheduled time I absolutely must watch
the resulting recording. Thanks for your efforts.

On Jun 22, 5:50 pm, "Cal Leeming [Simplicity Media Ltd]"

Post by Cal Leeming [Simplicity Media Ltd]
Nice!

Post by Thomas Weholt
On Wed, Jun 22, 2011 at 4:47 PM, Cal Leeming [Simplicity Media Ltd]

Post by Cal Leeming [Simplicity Media Ltd]

Post by Andre Terra
Hello, Cal
First of all, congrats on the newborn! The Django community will surely
benefit from having yet another success story, especially considering

how

Post by Cal Leeming [Simplicity Media Ltd]

Post by Andre Terra
big this project sounds. Is there any chance you could open-source some

of

Post by Cal Leeming [Simplicity Media Ltd]

Post by Andre Terra
your custom made improvements so that they could eventually be merged to
trunk?

Thank you! Yeah, the plan is to release as much of the improvements as

open

Post by Cal Leeming [Simplicity Media Ltd]
source as possible. Although I'd rely heavily on the community to make

them

Post by Cal Leeming [Simplicity Media Ltd]
'patch worthy' for the core, as the amount of spare time I have is

somewhat

Post by Cal Leeming [Simplicity Media Ltd]
limited.
The improvements list is growing by the day, and I usually try and post

as

Post by Cal Leeming [Simplicity Media Ltd]
many snippets as I can, and/or tickets etc.
It sounds like Thomas's DSE might be the perfect place for the bulk

update

Post by Cal Leeming [Simplicity Media Ltd]
code too.

FYI: Inspired by this discussion I've allready started on a similar
feature ( allthough somewhat simplified ) for DSE v2.2.0 and you're
right; the speed increase is huge using the method described here,
even compared to my current solution ( using cursor.executemany ),
which is considerably faster than the django orm allready. My testing
so far have been using postgresql, not sure how mysql will perform. I
expect to release DSE v.2.2.0 with this feature in the next few days.

Nice! Looking forward to seeing this :)

Post by Thomas Weholt
--
Mvh/Best regards,
Thomas Weholt
http://www.weholt.org
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

[Cal Leeming [Simplicity Media Ltd]]

I was thinking.. what would be really nice would be a monkey patch that
could catch any inserts/updates/saves from the ORM, and defer them until the
user triggered the bulk update some how.. That way, DSE would be completely
interchangeable with existing code, making it much more likely for people to
adopt it.

Post by Thomas Weholt
On Wed, Jun 22, 2011 at 4:47 PM, Cal Leeming [Simplicity Media Ltd]

Post by Cal Leeming [Simplicity Media Ltd]

Post by Andre Terra
Hello, Cal
First of all, congrats on the newborn! The Django community will surely
benefit from having yet another success story, especially considering

how

Post by Cal Leeming [Simplicity Media Ltd]

Post by Andre Terra
big this project sounds. Is there any chance you could open-source some

of

Post by Cal Leeming [Simplicity Media Ltd]

Post by Andre Terra
your custom made improvements so that they could eventually be merged to
trunk?

Thank you! Yeah, the plan is to release as much of the improvements as

open

Post by Cal Leeming [Simplicity Media Ltd]
source as possible. Although I'd rely heavily on the community to make

them

Post by Cal Leeming [Simplicity Media Ltd]
'patch worthy' for the core, as the amount of spare time I have is

somewhat

Post by Cal Leeming [Simplicity Media Ltd]
limited.
The improvements list is growing by the day, and I usually try and post

as

Post by Cal Leeming [Simplicity Media Ltd]
many snippets as I can, and/or tickets etc.
It sounds like Thomas's DSE might be the perfect place for the bulk

update

Post by Cal Leeming [Simplicity Media Ltd]
code too.

FYI: Inspired by this discussion I've allready started on a similar
feature ( allthough somewhat simplified ) for DSE v2.2.0 and you're
right; the speed increase is huge using the method described here,
even compared to my current solution ( using cursor.executemany ),
which is considerably faster than the django orm allready. My testing
so far have been using postgresql, not sure how mysql will perform. I
expect to release DSE v.2.2.0 with this feature in the next few days.
--
Mvh/Best regards,
Thomas Weholt
http://www.weholt.org
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

[Thomas Weholt]

On Wed, Jun 22, 2011 at 3:52 PM, Cal Leeming [Simplicity Media Ltd]

Post by Cal Leeming [Simplicity Media Ltd]
Sorry, let me explain a little better.
72500921 ~ 72550921), (avg 16.9 mins/million) - [('is_checked',
49659), ('is_image_blocked', 0), ('has_link', 1517), ('is_spam', 4)]
map(lambda x: (x[0], len(x[1])), _obj_incs.iteritems()) = [('is_checked',
49659), ('is_image_blocked', 0), ('has_link', 1517), ('is_spam', 4)]
In the above example, it has found 49659 rows which need 'is_checked'
changing to the value '1' (same principle applied to the other 3), giving a
total of 51,130 database writes, split into 4 queries.
_obj_incs.get('is_image_blocked').append(_hit_id)
_obj_incs.get('is_image_blocked').append(_parent_id)
Then I loop through those fields, and do an update() using the necessary
                    # now apply the obj changes in bulk (massive speed
improvements)
                        # update the child object
                        Post.objects.filter(
                            id__in = _value
                        ).update(
                            **{
                                _key : 1
                            }
                        )
So in simple terms, we're not doing 51 thousand update queries, instead
we're grouping them into bulk queries based on the row to be updated. It
doesn't yet to grouping based on key AND value, simply because we didn't
need it at the time, but if we release the code for public use,
we'd definitely add this in.
Hope this makes sense, let me know if I didn't explain it very well lol.
Cal

Actually, I started working on something similar, but tried to find
sets of fields, instead of just updating one field pr update, but
didn't finish it because the actual grouping of the fields seem to
take alot of time/cpu/memory. Perhaps if I focused on updating one
field at the time it would be simpler. Might look at it again for DSE
3.0 ;-)

Thomas

[Cal Leeming [Simplicity Media Ltd]]

Hmm, that's odd, the grouping (map/reduce/filter/lambda) is extremely quick
for me (even on a heavy data set).

My guess is that grouping would need to be done on a combination of field
name+value, and would need to allow the user to specify what bulk to use (to
prevent MemoryError exception - or find some way to reduce the bulk when
MemoryError is encountered).

If you end up introducing it into 3.0, I'll definitely be interested in
taking a look at the code :)

Cal

Post by Thomas Weholt
On Wed, Jun 22, 2011 at 3:52 PM, Cal Leeming [Simplicity Media Ltd]

Post by Cal Leeming [Simplicity Media Ltd]
Sorry, let me explain a little better.
72500921 ~ 72550921), (avg 16.9 mins/million) - [('is_checked',
49659), ('is_image_blocked', 0), ('has_link', 1517), ('is_spam', 4)]
map(lambda x: (x[0], len(x[1])), _obj_incs.iteritems()) = [('is_checked',
49659), ('is_image_blocked', 0), ('has_link', 1517), ('is_spam', 4)]
In the above example, it has found 49659 rows which need 'is_checked'
changing to the value '1' (same principle applied to the other 3), giving

a

Post by Cal Leeming [Simplicity Media Ltd]
total of 51,130 database writes, split into 4 queries.
_obj_incs.get('is_image_blocked').append(_hit_id)
_obj_incs.get('is_image_blocked').append(_parent_id)
Then I loop through those fields, and do an update() using the necessary
# now apply the obj changes in bulk (massive speed
improvements)
# update the child object
Post.objects.filter(
id__in = _value
).update(
**{
_key : 1
}
)
So in simple terms, we're not doing 51 thousand update queries, instead
we're grouping them into bulk queries based on the row to be updated. It
doesn't yet to grouping based on key AND value, simply because we didn't
need it at the time, but if we release the code for public use,
we'd definitely add this in.
Hope this makes sense, let me know if I didn't explain it very well lol.
Cal

Actually, I started working on something similar, but tried to find
sets of fields, instead of just updating one field pr update, but
didn't finish it because the actual grouping of the fields seem to
take alot of time/cpu/memory. Perhaps if I focused on updating one
field at the time it would be simpler. Might look at it again for DSE
3.0 ;-)
Thomas
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

[Shawn Milochik]

Cal,

That sounds awesome. I wish you could present it at DjangoCon US too. :o/

Shawn

[Steve Holden]

And, as luck would have it, the US Call For Papers was just published:

http://djangocon.us/blog/2011/06/22/call-papers/

regards
Steve

Post by Shawn Milochik
Cal,
That sounds awesome. I wish you could present it at DjangoCon US too. :o/
Shawn
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
For more options, visit this group at http://groups.google.com/**
group/django-users?hl=en<http://groups.google.com/group/django-users?hl=en>
.

[Cal Leeming [Simplicity Media Ltd]]

(sorry, should have sent this reply on-list)

You know, I was actually thinking about that, but the flights would cost way
too much (like $1400+) ;/ If the project has interest next year, I'll
probably ask about presenting at DjangoCon EU though.

Post by Steve Holden
http://djangocon.us/blog/2011/06/22/call-papers/
regards
Steve

Post by Shawn Milochik
Cal,
That sounds awesome. I wish you could present it at DjangoCon US too. :o/
Shawn
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
For more options, visit this group at http://groups.google.com/**
group/django-users?hl=en<http://groups.google.com/group/django-users?hl=en>
.

--
Steve Holden [phone] [phone]
Holden Web LLC http://www.holdenweb.com/
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

[Ivan Aleman]

On 22 June 2011 08:15, Cal Leeming [Simplicity Media Ltd] <

Post by Cal Leeming [Simplicity Media Ltd]
If you're interested, please reply on-list so others can see.

Sweet! Count me in :)

[Anurag Chourasia]

I am in :-)

Post by Ivan Aleman
On 22 June 2011 08:15, Cal Leeming [Simplicity Media Ltd] <

Post by Cal Leeming [Simplicity Media Ltd]
If you're interested, please reply on-list so others can see.

Sweet! Count me in :)
--
Iván
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

[creecode]

Hello SleepyCal,

On Wednesday, June 22, 2011 6:15:48 AM UTC-7, SleepyCal wrote:

If you're interested, please reply on-list so others can see.


+1

Also if the webcast could be stored for later viewing that would be grand.

Toodle-loooooooooooooo..........
creecode

[Cal Leeming [Simplicity Media Ltd]]

Post by creecode
Hello SleepyCal,
If you're interested, please reply on-list so others can see.
+1
Also if the webcast could be stored for later viewing that would be grand.

Yup, I'm planning on recording in 1080p and posting on Youtube shortly
afterwards.

Post by creecode
Toodle-loooooooooooooo..........
creecode
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/django-users/-/SZabiWnq_S0J.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

[creecode]

Hello SleepyCal,

On Wednesday, June 22, 2011 9:00:15 AM UTC-7, SleepyCal wrote:

Yup, I'm planning on recording in 1080p and posting on Youtube shortly

Post by Cal Leeming [Simplicity Media Ltd]
afterwards.

Fantastic!

Thank you,

Toodle-loooo...........
creecode

[serek]

Great idea.
I also would like to see this webcast

On Jun 22, 6:00 pm, "Cal Leeming [Simplicity Media Ltd]"

Post by Cal Leeming [Simplicity Media Ltd]

Post by creecode
Hello SleepyCal,
If you're interested, please reply on-list so others can see.
+1
Also if the webcast could be stored for later viewing that would be grand.

Yup, I'm planning on recording in 1080p and posting on Youtube shortly
afterwards.

Post by creecode
Toodle-loooooooooooooo..........
creecode
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/django-users/-/SZabiWnq_S0J.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

[Brian Bouterse]

+1 for making it viewable after the fact

Post by creecode
Hello SleepyCal,
If you're interested, please reply on-list so others can see.
+1
Also if the webcast could be stored for later viewing that would be grand.
Toodle-loooooooooooooo..........
creecode
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/django-users/-/SZabiWnq_S0J.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

[Malcolm Box]

On 22 June 2011 14:15, Cal Leeming [Simplicity Media Ltd] <

Post by Cal Leeming [Simplicity Media Ltd]
Hi all,
Therefore, I'd like to see if there would be any interest in webcast in
which I would explain how we handle such large amounts of data, the trial
and error processes we went through, some really neat tricks we've done to
avoid bottlenecks, our own approach to smart content filtering, and some of
the valuable lessons we have learned. The webcast would be completely free
of charge, last a couple of hours (with a short break) and anyone can
attend. I'd also offer up a Q&A session at the end.
If you're interested, please reply on-list so others can see.

Count me in.

Malcolm

[Mishen'ka]

I really like the idea, hope i can see it soon.

Interessting thanks

On 22 jun, 15:15, "Cal Leeming [Simplicity Media Ltd]"

Post by Cal Leeming [Simplicity Media Ltd]
Hi all,
Some of you may have noticed, in the last few months I've done quite a few
posts/snippets about handling large data sets in Django. At the end of this
month (after what seems like a lifetime of trial and error), we're finally
going to be releasing a new site which holds around 40mil+ rows of data,
grows by about 300-500k rows each day, handles 5GB of uploads per day, and
can handle around 1024 requests per second on stress test on a moderately
spec'd server.
As the entire thing is written in Django (and a bunch of other open source
products), I'd really like to give something back to the community. (stack
incls Celery/RabbitMQ/Sphinx SE/PYQuery/Percona
MySQL/NGINX/supervisord/debian etc)
Therefore, I'd like to see if there would be any interest in webcast in
which I would explain how we handle such large amounts of data, the trial
and error processes we went through, some really neat tricks we've done to
avoid bottlenecks, our own approach to smart content filtering, and some of
the valuable lessons we have learned. The webcast would be completely free
of charge, last a couple of hours (with a short break) and anyone can
attend. I'd also offer up a Q&A session at the end.
If you're interested, please reply on-list so others can see.
Thanks
Cal

[Cal Leeming [Simplicity Media Ltd]]

Nice, seems like there is a lot of positive feedback.

Here's some further info about the webcast/webinar:

*Expected Date:* July/August
*Expected Time:* Somewhere between 9AM - 9PM GMT+0 (UK Time), I'll probably
run a webpage vote so you guys can decide.
*Via:* GoToWebinar - screen share with voip.
*Length:* 1 hour max for actual presentation, 5 minute break, then another 1
hour max for Q&As.
*Presentation Size: *1920x1080 (1080p)

There will be some basic slides (to keep things structured), but the
majority of the time will be spent inside the browser + IDE + shell etc.

Cal

Post by Mishen'ka
I really like the idea, hope i can see it soon.
Interessting thanks
On 22 jun, 15:15, "Cal Leeming [Simplicity Media Ltd]"

Post by Cal Leeming [Simplicity Media Ltd]
Hi all,
Some of you may have noticed, in the last few months I've done quite a

few

Post by Cal Leeming [Simplicity Media Ltd]
posts/snippets about handling large data sets in Django. At the end of

this

Post by Cal Leeming [Simplicity Media Ltd]
month (after what seems like a lifetime of trial and error), we're

finally

Post by Cal Leeming [Simplicity Media Ltd]
going to be releasing a new site which holds around 40mil+ rows of data,
grows by about 300-500k rows each day, handles 5GB of uploads per day,

and

Post by Cal Leeming [Simplicity Media Ltd]
can handle around 1024 requests per second on stress test on a moderately
spec'd server.
As the entire thing is written in Django (and a bunch of other open

source

Post by Cal Leeming [Simplicity Media Ltd]
products), I'd really like to give something back to the community.

(stack

Post by Cal Leeming [Simplicity Media Ltd]
incls Celery/RabbitMQ/Sphinx SE/PYQuery/Percona
MySQL/NGINX/supervisord/debian etc)
Therefore, I'd like to see if there would be any interest in webcast in
which I would explain how we handle such large amounts of data, the trial
and error processes we went through, some really neat tricks we've done

to

Post by Cal Leeming [Simplicity Media Ltd]
avoid bottlenecks, our own approach to smart content filtering, and some

of

Post by Cal Leeming [Simplicity Media Ltd]
the valuable lessons we have learned. The webcast would be completely

free

Post by Cal Leeming [Simplicity Media Ltd]
of charge, last a couple of hours (with a short break) and anyone can
attend. I'd also offer up a Q&A session at the end.
If you're interested, please reply on-list so others can see.
Thanks
Cal

--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

[Jason]

+1 Very interested.

On Jun 22, 3:47 pm, "Cal Leeming [Simplicity Media Ltd]"

Post by Cal Leeming [Simplicity Media Ltd]
Nice, seems like there is a lot of positive feedback.
*Expected Date:* July/August
*Expected Time:* Somewhere between 9AM - 9PM GMT+0 (UK Time), I'll probably
run a webpage vote so you guys can decide.
*Via:* GoToWebinar - screen share with voip.
*Length:* 1 hour max for actual presentation, 5 minute break, then another 1
hour max for Q&As.
*Presentation Size: *1920x1080 (1080p)
There will be some basic slides (to keep things structured), but the
majority of the time will be spent inside the browser + IDE + shell etc.
Cal

Post by Mishen'ka
I really like the idea, hope i can see it soon.
Interessting thanks
On 22 jun, 15:15, "Cal Leeming [Simplicity Media Ltd]"

Post by Cal Leeming [Simplicity Media Ltd]
Hi all,
Some of you may have noticed, in the last few months I've done quite a

few

Post by Cal Leeming [Simplicity Media Ltd]
posts/snippets about handling large data sets in Django. At the end of

this

Post by Cal Leeming [Simplicity Media Ltd]
month (after what seems like a lifetime of trial and error), we're

finally

Post by Cal Leeming [Simplicity Media Ltd]
going to be releasing a new site which holds around 40mil+ rows of data,
grows by about 300-500k rows each day, handles 5GB of uploads per day,

and

Post by Cal Leeming [Simplicity Media Ltd]
can handle around 1024 requests per second on stress test on a moderately
spec'd server.
As the entire thing is written in Django (and a bunch of other open

source

Post by Cal Leeming [Simplicity Media Ltd]
products), I'd really like to give something back to the community.

(stack

Post by Cal Leeming [Simplicity Media Ltd]
incls Celery/RabbitMQ/Sphinx SE/PYQuery/Percona
MySQL/NGINX/supervisord/debian etc)
Therefore, I'd like to see if there would be any interest in webcast in
which I would explain how we handle such large amounts of data, the trial
and error processes we went through, some really neat tricks we've done

to

Post by Cal Leeming [Simplicity Media Ltd]
avoid bottlenecks, our own approach to smart content filtering, and some

of

Post by Cal Leeming [Simplicity Media Ltd]
the valuable lessons we have learned. The webcast would be completely

free

Post by Cal Leeming [Simplicity Media Ltd]
of charge, last a couple of hours (with a short break) and anyone can
attend. I'd also offer up a Q&A session at the end.
If you're interested, please reply on-list so others can see.
Thanks
Cal

--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

[Władysław Mettler]

Definitely interested. Struggling with scaling a
Django/Celery/Postgres/CouchDB/Memcach project now. Would love to see your
approach.

Cheers
Vlad

[higs]

Post by Cal Leeming [Simplicity Media Ltd]
If you're interested, please reply on-list so others can see.

would love to see this. thanks for posting.

[Derek]

Carl

Please post here when the recording becomes available... some of us do
not have access to HD-capable bandwidth yet!

Thanks
Derek

[Cal Leeming [Simplicity Media Ltd]]

Hi Derek,

I've had a look and apparently 4-5mbit should be sufficient enough for a HD
webinar. If quite a few of you don't have this capability (or screen size),
I could look into reducing it down to 1440x900 perhaps?

Let me know your thoughts guys. (maybe it's something I need to do a web
vote for, along with the timezone etc)

Cal

Post by Derek
Carl
Please post here when the recording becomes available... some of us do
not have access to HD-capable bandwidth yet!
Thanks
Derek
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

[Chris Calitz]

Sounds really cool. I'm definitely in.

Post by Cal Leeming [Simplicity Media Ltd]
Hi all,
Some of you may have noticed, in the last few months I've done quite a few posts/snippets about handling large data sets in Django. At the end of this month (after what seems like a lifetime of trial and error), we're finally going to be releasing a new site which holds around 40mil+ rows of data, grows by about 300-500k rows each day, handles 5GB of uploads per day, and can handle around 1024 requests per second on stress test on a moderately spec'd server.
As the entire thing is written in Django (and a bunch of other open source products), I'd really like to give something back to the community. (stack incls Celery/RabbitMQ/Sphinx SE/PYQuery/Percona MySQL/NGINX/supervisord/debian etc)
Therefore, I'd like to see if there would be any interest in webcast in which I would explain how we handle such large amounts of data, the trial and error processes we went through, some really neat tricks we've done to avoid bottlenecks, our own approach to smart content filtering, and some of the valuable lessons we have learned. The webcast would be completely free of charge, last a couple of hours (with a short break) and anyone can attend. I'd also offer up a Q&A session at the end.
If you're interested, please reply on-list so others can see.
Thanks
Cal
--
You received this message because you are subscribed to the Google Groups "Django users" group.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

[John DeRosa]

Me

Post by Chris Calitz
Sounds really cool. I'm definitely in.

Post by Cal Leeming [Simplicity Media Ltd]
Hi all,
Some of you may have noticed, in the last few months I've done quite a few posts/snippets about handling large data sets in Django. At the end of this month (after what seems like a lifetime of trial and error), we're finally going to be releasing a new site which holds around 40mil+ rows of data, grows by about 300-500k rows each day, handles 5GB of uploads per day, and can handle around 1024 requests per second on stress test on a moderately spec'd server.
As the entire thing is written in Django (and a bunch of other open source products), I'd really like to give something back to the community. (stack incls Celery/RabbitMQ/Sphinx SE/PYQuery/Percona MySQL/NGINX/supervisord/debian etc)
Therefore, I'd like to see if there would be any interest in webcast in which I would explain how we handle such large amounts of data, the trial and error processes we went through, some really neat tricks we've done to avoid bottlenecks, our own approach to smart content filtering, and some of the valuable lessons we have learned. The webcast would be completely free of charge, last a couple of hours (with a short break) and anyone can attend. I'd also offer up a Q&A session at the end.
If you're interested, please reply on-list so others can see.
Thanks
Cal
--
You received this message because you are subscribed to the Google Groups "Django users" group.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

[bruno desthuilliers]

+1

[Nolan Brubaker]

Yes, very interested. +1

On Fri, Jun 24, 2011 at 9:04 AM, bruno desthuilliers <

Post by Cal Leeming [Simplicity Media Ltd]
+1
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

[Cal Leeming [Simplicity Media Ltd]]

Really glad to see there has been so much interest in this!

If possible, can everyone please use the following form to indicate what
week day / timeslot they are available, along with their screen resolution.
This will be open until 1st July.

https://spreadsheets0.google.com/spreadsheet/viewform?formkey=dENyOVFSSkhSYnhBLVZGTktiN1Z3Y2c6MQ

Whichever week day / timeslot / screen resolution has 100% of the votes,
will be the chosen one :) If it's split, we'll look at doing the webcast on
two separate occasions. Hopefully this should give everyone a fair chance.

Thanks

Cal

Post by Nolan Brubaker
Yes, very interested. +1
On Fri, Jun 24, 2011 at 9:04 AM, bruno desthuilliers <

Post by Cal Leeming [Simplicity Media Ltd]
+1
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

[Cal Leeming [Simplicity Media Ltd]]

Second call for anyone who wants to attend this webcast, 3 days left to
place your vote.

Cal

On Fri, Jun 24, 2011 at 7:34 PM, Cal Leeming [Simplicity Media Ltd] <

Post by Cal Leeming [Simplicity Media Ltd]
Really glad to see there has been so much interest in this!
If possible, can everyone please use the following form to indicate what
week day / timeslot they are available, along with their screen resolution.
This will be open until 1st July.
https://spreadsheets0.google.com/spreadsheet/viewform?formkey=dENyOVFSSkhSYnhBLVZGTktiN1Z3Y2c6MQ
Whichever week day / timeslot / screen resolution has 100% of the votes,
will be the chosen one :) If it's split, we'll look at doing the webcast on
two separate occasions. Hopefully this should give everyone a fair chance.
Thanks
Cal

Post by Nolan Brubaker
Yes, very interested. +1
On Fri, Jun 24, 2011 at 9:04 AM, bruno desthuilliers <

Post by Cal Leeming [Simplicity Media Ltd]
+1
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

[Andre Terra]

Can't access google spreadsheets through my corporate proxy (go figure), but
I will vote later today.

Really interested in watching this!


Andre

On Tue, Jun 28, 2011 at 12:30 PM, Cal Leeming [Simplicity Media Ltd] <

Post by Cal Leeming [Simplicity Media Ltd]
Second call for anyone who wants to attend this webcast, 3 days left to
place your vote.
Cal
On Fri, Jun 24, 2011 at 7:34 PM, Cal Leeming [Simplicity Media Ltd] <

Post by Cal Leeming [Simplicity Media Ltd]
Really glad to see there has been so much interest in this!
If possible, can everyone please use the following form to indicate what
week day / timeslot they are available, along with their screen resolution.
This will be open until 1st July.
https://spreadsheets0.google.com/spreadsheet/viewform?formkey=dENyOVFSSkhSYnhBLVZGTktiN1Z3Y2c6MQ
Whichever week day / timeslot / screen resolution has 100% of the votes,
will be the chosen one :) If it's split, we'll look at doing the webcast on
two separate occasions. Hopefully this should give everyone a fair chance.
Thanks
Cal

Post by Nolan Brubaker
Yes, very interested. +1
On Fri, Jun 24, 2011 at 9:04 AM, bruno desthuilliers <

Post by Cal Leeming [Simplicity Media Ltd]
+1
--
You received this message because you are subscribed to the Google
Groups "Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

[Cal Leeming [Simplicity Media Ltd]]

Wow, that sucks! :X Never known a corp to block google docs

Post by Andre Terra
Can't access google spreadsheets through my corporate proxy (go figure),
but I will vote later today.
Really interested in watching this!
Andre
On Tue, Jun 28, 2011 at 12:30 PM, Cal Leeming [Simplicity Media Ltd] <

Post by Cal Leeming [Simplicity Media Ltd]
Second call for anyone who wants to attend this webcast, 3 days left to
place your vote.
Cal
On Fri, Jun 24, 2011 at 7:34 PM, Cal Leeming [Simplicity Media Ltd] <

Post by Cal Leeming [Simplicity Media Ltd]
Really glad to see there has been so much interest in this!
If possible, can everyone please use the following form to indicate what
week day / timeslot they are available, along with their screen resolution.
This will be open until 1st July.
https://spreadsheets0.google.com/spreadsheet/viewform?formkey=dENyOVFSSkhSYnhBLVZGTktiN1Z3Y2c6MQ
Whichever week day / timeslot / screen resolution has 100% of the votes,
will be the chosen one :) If it's split, we'll look at doing the webcast on
two separate occasions. Hopefully this should give everyone a fair chance.
Thanks
Cal
On Fri, Jun 24, 2011 at 4:14 PM, Nolan Brubaker <

Post by Nolan Brubaker
Yes, very interested. +1
On Fri, Jun 24, 2011 at 9:04 AM, bruno desthuilliers <

Post by Cal Leeming [Simplicity Media Ltd]
+1
--
You received this message because you are subscribed to the Google
Groups "Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

--
You received this message because you are subscribed to the Google
Groups "Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

[Wagner Vaz]

Please, count me in.
On Jun 22, 2011 10:15 AM, "Cal Leeming [Simplicity Media Ltd]" <

Post by Cal Leeming [Simplicity Media Ltd]
Hi all,
Some of you may have noticed, in the last few months I've done quite a few
posts/snippets about handling large data sets in Django. At the end of this
month (after what seems like a lifetime of trial and error), we're finally
going to be releasing a new site which holds around 40mil+ rows of data,
grows by about 300-500k rows each day, handles 5GB of uploads per day, and
can handle around 1024 requests per second on stress test on a moderately
spec'd server.
As the entire thing is written in Django (and a bunch of other open source
products), I'd really like to give something back to the community. (stack
incls Celery/RabbitMQ/Sphinx SE/PYQuery/Percona
MySQL/NGINX/supervisord/debian etc)
Therefore, I'd like to see if there would be any interest in webcast in
which I would explain how we handle such large amounts of data, the trial
and error processes we went through, some really neat tricks we've done to
avoid bottlenecks, our own approach to smart content filtering, and some of
the valuable lessons we have learned. The webcast would be completely free
of charge, last a couple of hours (with a short break) and anyone can
attend. I'd also offer up a Q&A session at the end.
If you're interested, please reply on-list so others can see.
Thanks
Cal
--
You received this message because you are subscribed to the Google Groups

"Django users" group.

Post by Cal Leeming [Simplicity Media Ltd]
To unsubscribe from this group, send email to
For more options, visit this group at

http://groups.google.com/group/django-users?hl=en.

[Cal Leeming [Simplicity Media Ltd]]

Hey all,

Last call for registering your interest in this webcast (28 votes so far).

If you haven't done so already, please visit the following URL:

https://spreadsheets.google.com/spreadsheet/viewform?hl=en_GB&formkey=dENyOVFSSkhSYnhBLVZGTktiN1Z3Y2c6MQ#gid=0

Poll ends tomorrow, at which point we will decide upon a fixed time/date(s).

Cal

Post by Wagner Vaz
Please, count me in.
On Jun 22, 2011 10:15 AM, "Cal Leeming [Simplicity Media Ltd]" <

Post by Cal Leeming [Simplicity Media Ltd]
Hi all,
Some of you may have noticed, in the last few months I've done quite a

few

Post by Cal Leeming [Simplicity Media Ltd]
posts/snippets about handling large data sets in Django. At the end of

this

Post by Cal Leeming [Simplicity Media Ltd]
month (after what seems like a lifetime of trial and error), we're

finally

Post by Cal Leeming [Simplicity Media Ltd]
going to be releasing a new site which holds around 40mil+ rows of data,
grows by about 300-500k rows each day, handles 5GB of uploads per day,

and

Post by Cal Leeming [Simplicity Media Ltd]
can handle around 1024 requests per second on stress test on a moderately
spec'd server.
As the entire thing is written in Django (and a bunch of other open

source

Post by Cal Leeming [Simplicity Media Ltd]
products), I'd really like to give something back to the community.

(stack

Post by Cal Leeming [Simplicity Media Ltd]
incls Celery/RabbitMQ/Sphinx SE/PYQuery/Percona
MySQL/NGINX/supervisord/debian etc)
Therefore, I'd like to see if there would be any interest in webcast in
which I would explain how we handle such large amounts of data, the trial
and error processes we went through, some really neat tricks we've done

to

Post by Cal Leeming [Simplicity Media Ltd]
avoid bottlenecks, our own approach to smart content filtering, and some

of

Post by Cal Leeming [Simplicity Media Ltd]
the valuable lessons we have learned. The webcast would be completely

free

Post by Cal Leeming [Simplicity Media Ltd]
of charge, last a couple of hours (with a short break) and anyone can
attend. I'd also offer up a Q&A session at the end.
If you're interested, please reply on-list so others can see.
Thanks
Cal
--
You received this message because you are subscribed to the Google Groups

"Django users" group.

Post by Cal Leeming [Simplicity Media Ltd]
To unsubscribe from this group, send email to
For more options, visit this group at

http://groups.google.com/group/django-users?hl=en.
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

[Benedict Verheyen]

Post by Cal Leeming [Simplicity Media Ltd]
Hey all,
Last call for registering your interest in this webcast (28 votes so far).
https://spreadsheets.google.com/spreadsheet/viewform?hl=en_GB&formkey=dENyOVFSSkhSYnhBLVZGTktiN1Z3Y2c6MQ#gid=0
<https://spreadsheets.google.com/spreadsheet/viewform?hl=en_GB&formkey=dENyOVFSSkhSYnhBLVZGTktiN1Z3Y2c6MQ#gid=0>
Poll ends tomorrow, at which point we will decide upon a fixed time/date(s).

I'm interested, visited the URL above already.
Time isn't going to be that big of an issue for me as I'm only 1 timezone away :)

Cheers,
Benedict

[Cal Leeming [Simplicity Media Ltd]]

Hi guys,

Alright, vote is closed and here are the results:

https://spreadsheets.google.com/spreadsheet/gform?key=0AiciLOVZqDWqdENyOVFSSkhSYnhBLVZGTktiN1Z3Y2c&hl=en_GB&gridId=0#chart

It looks like I might have to do two separate webcasts to cater for
all timezones, but we'll see.

I'll post up a fixed time/date (or two) shortly.

Cal

On Thu, Jun 30, 2011 at 2:02 PM, Benedict Verheyen <

Post by Cal Leeming [Simplicity Media Ltd]

Post by Cal Leeming [Simplicity Media Ltd]
Hey all,
Last call for registering your interest in this webcast (28 votes so

far).
https://spreadsheets.google.com/spreadsheet/viewform?hl=en_GB&formkey=dENyOVFSSkhSYnhBLVZGTktiN1Z3Y2c6MQ#gid=0

Post by Cal Leeming [Simplicity Media Ltd]
<

https://spreadsheets.google.com/spreadsheet/viewform?hl=en_GB&formkey=dENyOVFSSkhSYnhBLVZGTktiN1Z3Y2c6MQ#gid=0

Post by Cal Leeming [Simplicity Media Ltd]
Poll ends tomorrow, at which point we will decide upon a fixed

time/date(s).
I'm interested, visited the URL above already.
Time isn't going to be that big of an issue for me as I'm only 1 timezone away :)
Cheers,
Benedict
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

[Cal Leeming [Simplicity Media Ltd]]

Oops, sorry guys, I pasted the wrong link.

Here you go:

https://spreadsheets0.google.com/spreadsheet/viewanalytics?formkey=dENyOVFSSkhSYnhBLVZGTktiN1Z3Y2c6MQ

On Sat, Jul 2, 2011 at 11:35 AM, Cal Leeming [Simplicity Media Ltd]

Post by Cal Leeming [Simplicity Media Ltd]
Hi guys,
https://spreadsheets.google.com/spreadsheet/gform?key=0AiciLOVZqDWqdENyOVFSSkhSYnhBLVZGTktiN1Z3Y2c&hl=en_GB&gridId=0#chart
It looks like I might have to do two separate webcasts to cater for
all timezones, but we'll see.
I'll post up a fixed time/date (or two) shortly.
Cal
On Thu, Jun 30, 2011 at 2:02 PM, Benedict Verheyen

Post by Benedict Verheyen

Post by Cal Leeming [Simplicity Media Ltd]
Hey all,
Last call for registering your interest in this webcast (28 votes so far).
https://spreadsheets.google.com/spreadsheet/viewform?hl=en_GB&formkey=dENyOVFSSkhSYnhBLVZGTktiN1Z3Y2c6MQ#gid=0
<https://spreadsheets.google.com/spreadsheet/viewform?hl=en_GB&formkey=dENyOVFSSkhSYnhBLVZGTktiN1Z3Y2c6MQ#gid=0>
Poll ends tomorrow, at which point we will decide upon a fixed time/date(s).

I'm interested, visited the URL above already.
Time isn't going to be that big of an issue for me as I'm only 1 timezone away :)
Cheers,
Benedict
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

[gradja]

Post by Cal Leeming [Simplicity Media Ltd]
If you're interested, please reply on-list so others can see.

Count me in, and thanks to share your experience with others.
Graziella

[Andre Santos]

Maybe someone can record it and upload somewhere for the ones who cant be
online when you do the presentation...

Post by gradja

Post by Cal Leeming [Simplicity Media Ltd]
If you're interested, please reply on-list so others can see.

Count me in, and thanks to share your experience with others.
Graziella
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

[Cal Leeming [Simplicity Media Ltd]]

This has already been discussed earlier in the thread :)

Post by Andre Santos
Maybe someone can record it and upload somewhere for the ones who cant be
online when you do the presentation...

Post by gradja

Post by Cal Leeming [Simplicity Media Ltd]
If you're interested, please reply on-list so others can see.

Count me in, and thanks to share your experience with others.
Graziella
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

[Andre Santos]

Oh, sorry :).

2011/7/3 Cal Leeming [Simplicity Media Ltd] <

Post by Cal Leeming [Simplicity Media Ltd]
This has already been discussed earlier in the thread :)

Post by Andre Santos
Maybe someone can record it and upload somewhere for the ones who cant be
online when you do the presentation...

Post by gradja

Post by Cal Leeming [Simplicity Media Ltd]
If you're interested, please reply on-list so others can see.

Count me in, and thanks to share your experience with others.
Graziella
--
You received this message because you are subscribed to the Google

Groups

Post by Andre Santos

Post by gradja
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

[Cal Leeming [Simplicity Media Ltd]]

Hi all,

Great response to this, 45 registered votes in total.

The webcast will take place on Monday 29th August 2011 - (Minimum resolution
1920x1080)

However, because the time zone results are almost split 50/50, I'm going to
allow users to select from two time slots, if the results are still split
50/50, then I'll do both an afternoon and evening session.

Please use this form to register your place.
https://spreadsheets.google.com/a/foxwhisper.co.uk/spreadsheet/viewform?formkey=dENPX3BMUUFMbi1CbElwV3BvOEdkNmc6MQ

Cal

[Venkatraman S]

Can you please ..please...please..please record this session!?

-V

[Cal Leeming [Simplicity Media Ltd]]

Of course :) I'll make sure to take a high quality recording and stick it on
YT.

Cal

Post by Venkatraman S
Can you please ..please...please..please record this session!?
-V
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

[Andre Terra]

Unfortunately, I'm on GMT-0300 and none of those timezones work for me.
Looking forward to the recorded version, though!

Cheers,
André

On Tue, Jul 12, 2011 at 10:48 AM, Cal Leeming [Simplicity Media Ltd] <

Post by Cal Leeming [Simplicity Media Ltd]
Hi all,
Great response to this, 45 registered votes in total.
The webcast will take place on Monday 29th August 2011 - (Minimum
resolution 1920x1080)
However, because the time zone results are almost split 50/50, I'm going to
allow users to select from two time slots, if the results are still split
50/50, then I'll do both an afternoon and evening session.
Please use this form to register your place.
https://spreadsheets.google.com/a/foxwhisper.co.uk/spreadsheet/viewform?formkey=dENPX3BMUUFMbi1CbElwV3BvOEdkNmc6MQ
Cal
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

[Cal Leeming [Simplicity Media Ltd]]

Second call for register your attendance on this webcast, if you haven't
already done so.

Thanks

Cal

On Tue, Jul 12, 2011 at 2:48 PM, Cal Leeming [Simplicity Media Ltd] <

Post by Cal Leeming [Simplicity Media Ltd]
Hi all,
Great response to this, 45 registered votes in total.
The webcast will take place on Monday 29th August 2011 - (Minimum
resolution 1920x1080)
However, because the time zone results are almost split 50/50, I'm going to
allow users to select from two time slots, if the results are still split
50/50, then I'll do both an afternoon and evening session.
Please use this form to register your place.
https://spreadsheets.google.com/a/foxwhisper.co.uk/spreadsheet/viewform?formkey=dENPX3BMUUFMbi1CbElwV3BvOEdkNmc6MQ
Cal

[nicolas HERSOG]

I won t be able to follow the live session but I can't wait to watch your
record on YT.

Thx!

On Mon, Jul 25, 2011 at 2:42 PM, Cal Leeming [Simplicity Media Ltd] <

Post by Cal Leeming [Simplicity Media Ltd]
Second call for register your attendance on this webcast, if you haven't
already done so.
Thanks
Cal
On Tue, Jul 12, 2011 at 2:48 PM, Cal Leeming [Simplicity Media Ltd] <

Post by Cal Leeming [Simplicity Media Ltd]
Hi all,
Great response to this, 45 registered votes in total.
The webcast will take place on Monday 29th August 2011 - (Minimum
resolution 1920x1080)
However, because the time zone results are almost split 50/50, I'm going
to allow users to select from two time slots, if the results are still split
50/50, then I'll do both an afternoon and evening session.
Please use this form to register your place.
https://spreadsheets.google.com/a/foxwhisper.co.uk/spreadsheet/viewform?formkey=dENPX3BMUUFMbi1CbElwV3BvOEdkNmc6MQ
Cal

--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

[Cal Leeming [Simplicity Media Ltd]]

Hi all,

Okay, good news and bad news.

The bad news, is that the site/project which sparked this webcast to be
made, has had some legal complications and is being shut down - less than 1
month after being released :L

The good news, is that the webcast will still be going ahead.

For those of you that want to see the site it was related to, it's
http://www.iliketochan.com .

The site itself was a complete 4chan archive, dating back to 2010, peaking
at 50 million posts and 17 million images.

Particularly sad about this because so much time and effort went into the
site, but at the same time, it has been a great learning experience (both in
code and business). So being able to share that on the webcast will be of
some small comfort :)

Cal

On Thu, Aug 11, 2011 at 6:31 PM, Cal Leeming [Simplicity Media Ltd] <

https://spreadsheets.google.com/a/foxwhisper.co.uk/spreadsheet/viewform?hl=en_US&formkey=dENPX3BMUUFMbi1CbElwV3BvOEdkNmc6MQ#gid=0
If you want to register your attendance, please do so now.
Thanks

Post by nicolas HERSOG
I won t be able to follow the live session but I can't wait to watch your
record on YT.
Thx!
On Mon, Jul 25, 2011 at 2:42 PM, Cal Leeming [Simplicity Media Ltd] <

Post by Cal Leeming [Simplicity Media Ltd]
Second call for register your attendance on this webcast, if you haven't
already done so.
Thanks
Cal
On Tue, Jul 12, 2011 at 2:48 PM, Cal Leeming [Simplicity Media Ltd] <

Post by Cal Leeming [Simplicity Media Ltd]
Hi all,
Great response to this, 45 registered votes in total.
The webcast will take place on Monday 29th August 2011 - (Minimum
resolution 1920x1080)
However, because the time zone results are almost split 50/50, I'm going
to allow users to select from two time slots, if the results are still split
50/50, then I'll do both an afternoon and evening session.
Please use this form to register your place.
https://spreadsheets.google.com/a/foxwhisper.co.uk/spreadsheet/viewform?formkey=dENPX3BMUUFMbi1CbElwV3BvOEdkNmc6MQ
Cal

--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

[Cal Leeming [Simplicity Media Ltd]]

Hey guys,

I'm sending out the webcast invitations now (should receive them in about 10
minutes from GoToMeeting). Last chance to jump on if you haven't already.

Cal

On Sat, Aug 20, 2011 at 12:32 AM, Cal Leeming [Simplicity Media Ltd] <

Post by Cal Leeming [Simplicity Media Ltd]
Hi all,
Okay, good news and bad news.
The bad news, is that the site/project which sparked this webcast to be
made, has had some legal complications and is being shut down - less than 1
month after being released :L
The good news, is that the webcast will still be going ahead.
For those of you that want to see the site it was related to, it's
http://www.iliketochan.com .
The site itself was a complete 4chan archive, dating back to 2010, peaking
at 50 million posts and 17 million images.
Particularly sad about this because so much time and effort went into the
site, but at the same time, it has been a great learning experience (both in
code and business). So being able to share that on the webcast will be of
some small comfort :)
Cal
On Thu, Aug 11, 2011 at 6:31 PM, Cal Leeming [Simplicity Media Ltd] <

https://spreadsheets.google.com/a/foxwhisper.co.uk/spreadsheet/viewform?hl=en_US&formkey=dENPX3BMUUFMbi1CbElwV3BvOEdkNmc6MQ#gid=0
If you want to register your attendance, please do so now.
Thanks

Post by nicolas HERSOG
I won t be able to follow the live session but I can't wait to watch your
record on YT.
Thx!
On Mon, Jul 25, 2011 at 2:42 PM, Cal Leeming [Simplicity Media Ltd] <

Post by Cal Leeming [Simplicity Media Ltd]
Second call for register your attendance on this webcast, if you haven't
already done so.
Thanks
Cal
On Tue, Jul 12, 2011 at 2:48 PM, Cal Leeming [Simplicity Media Ltd] <

Post by Cal Leeming [Simplicity Media Ltd]
Hi all,
Great response to this, 45 registered votes in total.
The webcast will take place on Monday 29th August 2011 - (Minimum
resolution 1920x1080)
However, because the time zone results are almost split 50/50, I'm
going to allow users to select from two time slots, if the results are still
split 50/50, then I'll do both an afternoon and evening session.
Please use this form to register your place.
https://spreadsheets.google.com/a/foxwhisper.co.uk/spreadsheet/viewform?formkey=dENPX3BMUUFMbi1CbElwV3BvOEdkNmc6MQ
Cal

--
You received this message because you are subscribed to the Google
Groups "Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

[william ratcliff]

Any problem if we register for both?

On Sat, Aug 27, 2011 at 4:33 PM, Cal Leeming [Simplicity Media Ltd] <

Post by Cal Leeming [Simplicity Media Ltd]
Hey guys,
I'm sending out the webcast invitations now (should receive them in about
10 minutes from GoToMeeting). Last chance to jump on if you haven't already.
Cal
On Sat, Aug 20, 2011 at 12:32 AM, Cal Leeming [Simplicity Media Ltd] <

Post by Cal Leeming [Simplicity Media Ltd]
Hi all,
Okay, good news and bad news.
The bad news, is that the site/project which sparked this webcast to be
made, has had some legal complications and is being shut down - less than 1
month after being released :L
The good news, is that the webcast will still be going ahead.
For those of you that want to see the site it was related to, it's
http://www.iliketochan.com .
The site itself was a complete 4chan archive, dating back to 2010, peaking
at 50 million posts and 17 million images.
Particularly sad about this because so much time and effort went into the
site, but at the same time, it has been a great learning experience (both in
code and business). So being able to share that on the webcast will be of
some small comfort :)
Cal
On Thu, Aug 11, 2011 at 6:31 PM, Cal Leeming [Simplicity Media Ltd] <

https://spreadsheets.google.com/a/foxwhisper.co.uk/spreadsheet/viewform?hl=en_US&formkey=dENPX3BMUUFMbi1CbElwV3BvOEdkNmc6MQ#gid=0
If you want to register your attendance, please do so now.
Thanks

Post by nicolas HERSOG
I won t be able to follow the live session but I can't wait to watch
your record on YT.
Thx!
On Mon, Jul 25, 2011 at 2:42 PM, Cal Leeming [Simplicity Media Ltd] <

Post by Cal Leeming [Simplicity Media Ltd]
Second call for register your attendance on this webcast, if you
haven't already done so.
Thanks
Cal
On Tue, Jul 12, 2011 at 2:48 PM, Cal Leeming [Simplicity Media Ltd] <

Post by Cal Leeming [Simplicity Media Ltd]
Hi all,
Great response to this, 45 registered votes in total.
The webcast will take place on Monday 29th August 2011 - (Minimum
resolution 1920x1080)
However, because the time zone results are almost split 50/50, I'm
going to allow users to select from two time slots, if the results are still
split 50/50, then I'll do both an afternoon and evening session.
Please use this form to register your place.
https://spreadsheets.google.com/a/foxwhisper.co.uk/spreadsheet/viewform?formkey=dENPX3BMUUFMbi1CbElwV3BvOEdkNmc6MQ
Cal

--
You received this message because you are subscribed to the Google
Groups "Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

--
You received this message because you are subscribed to the Google
Groups "Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

[Cal Leeming [Simplicity Media Ltd]]

Nope, we're allowed up to 100 attendees per time slot, and we are at 55 at
the moment, so feel free to register for both.

On Sat, Aug 27, 2011 at 9:46 PM, william ratcliff <

Post by william ratcliff
Any problem if we register for both?
On Sat, Aug 27, 2011 at 4:33 PM, Cal Leeming [Simplicity Media Ltd] <

Post by Cal Leeming [Simplicity Media Ltd]
Hey guys,
I'm sending out the webcast invitations now (should receive them in about
10 minutes from GoToMeeting). Last chance to jump on if you haven't already.
Cal
On Sat, Aug 20, 2011 at 12:32 AM, Cal Leeming [Simplicity Media Ltd] <

Post by Cal Leeming [Simplicity Media Ltd]
Hi all,
Okay, good news and bad news.
The bad news, is that the site/project which sparked this webcast to be
made, has had some legal complications and is being shut down - less than 1
month after being released :L
The good news, is that the webcast will still be going ahead.
For those of you that want to see the site it was related to, it's
http://www.iliketochan.com .
The site itself was a complete 4chan archive, dating back to 2010,
peaking at 50 million posts and 17 million images.
Particularly sad about this because so much time and effort went into the
site, but at the same time, it has been a great learning experience (both in
code and business). So being able to share that on the webcast will be of
some small comfort :)
Cal
On Thu, Aug 11, 2011 at 6:31 PM, Cal Leeming [Simplicity Media Ltd] <

https://spreadsheets.google.com/a/foxwhisper.co.uk/spreadsheet/viewform?hl=en_US&formkey=dENPX3BMUUFMbi1CbElwV3BvOEdkNmc6MQ#gid=0
If you want to register your attendance, please do so now.
Thanks

Post by nicolas HERSOG
I won t be able to follow the live session but I can't wait to watch
your record on YT.
Thx!
On Mon, Jul 25, 2011 at 2:42 PM, Cal Leeming [Simplicity Media Ltd] <

Post by Cal Leeming [Simplicity Media Ltd]
Second call for register your attendance on this webcast, if you
haven't already done so.
Thanks
Cal
On Tue, Jul 12, 2011 at 2:48 PM, Cal Leeming [Simplicity Media Ltd] <

Post by Cal Leeming [Simplicity Media Ltd]
Hi all,
Great response to this, 45 registered votes in total.
The webcast will take place on Monday 29th August 2011 - (Minimum
resolution 1920x1080)
However, because the time zone results are almost split 50/50, I'm
going to allow users to select from two time slots, if the results are still
split 50/50, then I'll do both an afternoon and evening session.
Please use this form to register your place.
https://spreadsheets.google.com/a/foxwhisper.co.uk/spreadsheet/viewform?formkey=dENPX3BMUUFMbi1CbElwV3BvOEdkNmc6MQ
Cal

--
You received this message because you are subscribed to the Google
Groups "Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

--
You received this message because you are subscribed to the Google
Groups "Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

[Cal Leeming [Simplicity Media Ltd]]

Bloody thing, had to send it directly to the list cos it doesn't have the
option of bulk inviting people. How annoying :X

Anyone who didn't receive the invite email, please let me know.

Cal

On Sat, Aug 27, 2011 at 9:50 PM, Cal Leeming [Simplicity Media Ltd] <

Post by Cal Leeming [Simplicity Media Ltd]
Nope, we're allowed up to 100 attendees per time slot, and we are at 55 at
the moment, so feel free to register for both.
On Sat, Aug 27, 2011 at 9:46 PM, william ratcliff <

Post by william ratcliff
Any problem if we register for both?
On Sat, Aug 27, 2011 at 4:33 PM, Cal Leeming [Simplicity Media Ltd] <

Post by Cal Leeming [Simplicity Media Ltd]
Hey guys,
I'm sending out the webcast invitations now (should receive them in about
10 minutes from GoToMeeting). Last chance to jump on if you haven't already.
Cal
On Sat, Aug 20, 2011 at 12:32 AM, Cal Leeming [Simplicity Media Ltd] <

Post by Cal Leeming [Simplicity Media Ltd]
Hi all,
Okay, good news and bad news.
The bad news, is that the site/project which sparked this webcast to be
made, has had some legal complications and is being shut down - less than 1
month after being released :L
The good news, is that the webcast will still be going ahead.
For those of you that want to see the site it was related to, it's
http://www.iliketochan.com .
The site itself was a complete 4chan archive, dating back to 2010,
peaking at 50 million posts and 17 million images.
Particularly sad about this because so much time and effort went into
the site, but at the same time, it has been a great learning experience
(both in code and business). So being able to share that on the webcast will
be of some small comfort :)
Cal
On Thu, Aug 11, 2011 at 6:31 PM, Cal Leeming [Simplicity Media Ltd] <

https://spreadsheets.google.com/a/foxwhisper.co.uk/spreadsheet/viewform?hl=en_US&formkey=dENPX3BMUUFMbi1CbElwV3BvOEdkNmc6MQ#gid=0
If you want to register your attendance, please do so now.
Thanks

Post by nicolas HERSOG
I won t be able to follow the live session but I can't wait to watch
your record on YT.
Thx!
On Mon, Jul 25, 2011 at 2:42 PM, Cal Leeming [Simplicity Media Ltd] <

Post by Cal Leeming [Simplicity Media Ltd]
Second call for register your attendance on this webcast, if you
haven't already done so.
Thanks
Cal
On Tue, Jul 12, 2011 at 2:48 PM, Cal Leeming [Simplicity Media Ltd]

Post by Cal Leeming [Simplicity Media Ltd]
Hi all,
Great response to this, 45 registered votes in total.
The webcast will take place on Monday 29th August 2011 - (Minimum
resolution 1920x1080)
However, because the time zone results are almost split 50/50, I'm
going to allow users to select from two time slots, if the results are still
split 50/50, then I'll do both an afternoon and evening session.
Please use this form to register your place.
https://spreadsheets.google.com/a/foxwhisper.co.uk/spreadsheet/viewform?formkey=dENPX3BMUUFMbi1CbElwV3BvOEdkNmc6MQ
Cal

--
You received this message because you are subscribed to the Google
Groups "Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

--
You received this message because you are subscribed to the Google
Groups "Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.

[AlexH]

Post by Cal Leeming [Simplicity Media Ltd]
If you're interested, please reply on-list so others can see.

Would be very interested in attending. Thanks!