Amund Tveit's Blog - Latest Comments in Tools for Accelerating Python

Re: Tools for Accelerating Python

Jeremy — Wed, 17 Dec 2008 01:03:31 -0000

Great list, thankx

Re: Tools for Accelerating Python

Ashish — Fri, 05 Dec 2008 09:31:00 -0000

Excellent list. Can you post similar tools for Google App Engine too.

Re: Tools for Accelerating Python

Ben Cummings — Mon, 01 Dec 2008 08:53:03 -0000

Putting Stackless Python and threads in the same breath seems pretty misleading to me, and I would think long and hard before using either approach. Rewrite your code with threads, and you need to make sure at least one of your threads will be spending enough time in C code that releases the GIL to actually give the other thread time to run concurrently, or else you've just wasted a lot of time and introduced a whole lot of concurrency bugs to write a program that still only runs on one core most of the time, thanks to the aforementioned GIL.

Stackless Python gives you logical parallelism, but it doesn't really give you distribution. You may have 100,000 tasklets running, but you're still only using one core of your one processor until you use one of the other approaches to actually parallelize your program. Stackless will allow you to then pickle your tasklets and swap them between processes or machines, but you've just tossed a lot of the nice, deterministic nature of Stackless tasklets out the window.

It also bears mentioning that RPython isn't really specified and isn't intended for use outside of PyPy, meaning it might change out from underneath you if you try to build anything on top of it, and it apparently has some pretty awful error handling (are you alright with your Python program segfaulting?) I'd love for them to standardize it and be willing to make RPython a language for general use, but they aren't willing to do that yet.

Re: Tools for Accelerating Python

lorg — Sun, 30 Nov 2008 12:15:26 -0000

You should add the multiprocessing module as well. (PyProcessing before it was included in the standard library.)
When seeking to take advantage of multiple cores, it performs well enough.

Re: Tools for Accelerating Python

amund — Sun, 30 Nov 2008 09:00:44 -0000

Thanks Ian, added a benchmark link next to ShedSkin

Re: Tools for Accelerating Python

amund — Sun, 30 Nov 2008 09:00:10 -0000

Thanks Jacinto, added now.

Re: Tools for Accelerating Python

Ian Ozsvald — Sun, 30 Nov 2008 07:58:28 -0000

I've recently written a long article on using ShedSkin to give my Python neural-network code a 196* speed increase. Several contributors took the code and made Cython, Numpy and raw-C variants alongside my Vanilla Python and Psyco timings to offer their own comparisons.
The numbers are very useful if you're looking into these frameworks:
http://ianozsvald.com/2008/...
Ian.

Re: Tools for Accelerating Python

Jacinto — Sun, 30 Nov 2008 07:49:04 -0000

There is also Boost.Python which provides interoperability between Python and C++.

Re: Tools for Accelerating Python

amund — Sun, 30 Nov 2008 04:32:24 -0000

thanks George, have added those now.

Re: Tools for Accelerating Python

George — Sat, 29 Nov 2008 22:40:42 -0000

Hi. Maybe, you should also add Twisted and rpyc...

Re: Tools for Accelerating Python

amund — Sat, 29 Nov 2008 18:07:02 -0000

Thanks, wasn't aware of Cython, will have a look. (The one I most recently tested was cinpy)

Re: Tools for Accelerating Python

ryszard — Sat, 29 Nov 2008 17:29:51 -0000

You forgot about Cython [http://cython.org/]. It somehow connected to PyRex and seems to be mice to use.