64 bit and multi-threading

Ok, I've started to make things in Nodebox that are rather big: running into the millions of lines and as you can guess, you just have to switch the monitor off and go make dinner while the computer does its thing.
Now I've noticed that nodebox only makes use of one cpu core at a time, I have a mac pro quad with 8gig ram and it doesn't really speed things up.
If nodebox was to make use of all four (or 8, in some people's cases) cores then that would speed things up nicely.
Also, I'm assuming that its all 32 bit... It would be really nice to see a 64bit, multi-core update one day!

Hi,

This is a problem inherent with Python, called the Global Interpreter Lock. You might want to read this mailing list post that explains it better than I can:

http://mail.python.org/pipermail/python-list/2007-August/453662.html

Having said that, we are currently exploring ways to get rid of these limitations. Obviously, this is not an easy problem, and the solution will not be finished soon. But the future is distributed, and we would be foolish to ignore that.

Kind regards,
Frederik

Thanks Frederik, that explains a few things. I'm sure that there's a work-around though, there always is.
I'll look into this in more depth.
Thanks again.

Dimitri, perhaps it could be interesting to have a stab at some of the recent parallel modules for python. I've had good luck with Parallel Python, but Processing (ironic?) seems to be quite a bit more advanced, and is the most likely candidate for addition to the standard modules. I think the trick it does is forking the original python process, which is far more advanced than what Parallel Python does. If you have millions of lines to draw, implement a generator, and have this generator consumed by your processor pool.
Since I just installed my 2*quad core machine, I'm _very_ curious to hear whether you're making progress on this.

overview page:
http://wiki.python.org/moin/ParallelProcessing
Processing:
http://pypi.python.org/pypi/processing
( its on macports, otherwise try: easy_install processing )

best of luck,

-jelle

Those links make for an interesting read Jelle, it looks like pooling is what we're looking at here.
For some reason I was under the impression that the OS dealt with that side of things but that just isn't true.
It's a catch 22 situation here, stick by a programming language that non-programmers (like myself) can understand or have a go at something VERY complex as a work-around...
I'll do some digging around and nag some geek friends to see if this can actually happen, relatively easily.

I know its a long shot but imagine if it used the GPU as well..... We can always dream.

I'll report back with any news.

Hi Dimitri,

Well if you could tell us what your program approx. looks like, then its easier to see whether Pool is a relevant way of speeding up things. Since your using millions of lines, you could easily re-write your program, such that you would have a generator that would be consumed by all the processes in the Pool:

lines = (line for l in GenerateLines(1e6))
p = Pool(8)
p.get()

Something along those lines.
Actually, it would make an interesting experiment...

Best,

-jelle