Generators for Xtreams

While at Smalltalk Solutions 2011 in Las Vegas, a question was raised about Generators. Martin and I had no idea what Generators were, so we filed that one away for “later pontification”. Well, that later came this week when Nicolas Cellier posted about using generators and Xtreams to compute primes.

One criticism Nicolas raised was the lack of Generators in Xtreams. We wanted to understand what Generators were and why they needed to manipulate the stack to work. In the end, we decided it’s actually faster and easier to implement them using a second process and a couple of semaphores. Also, more portable.

So what do generators do? Well, possibly the simplest example is this:

1 to: 10 do: [:i | …do something…]

In this case, you cannot easily transform it in to a stream using [] reading, because you get stuck inside the iteration block, controlled by the #to:do: method. A generator allows you to push out elements piece-meal regardless of the nature of the code you’re running, eg:

readStream := [:out | 1 to: 10 do: [:i | out put: i ] ] reading

The block will begin running immediately, but as soon as it tries to write out a value on to the @out variable, it will halt execution until you attempt to read from @readStream. This makes it possible for Nicolas to re-write his example as:

primes := [:out | Integer primesUpTo: 100 do: [:prime | out put: prime ] ] reading.

Note that @out is actually a write stream, so you can use all the regular streaming API on it too, such as #write: instead of a single element #put:.

The last part of this story isn’t to do with generators at all, but about using buffers to make non-positionable information positionable. In the case of Nicolas’s primes, he wanted the last element on the stream. You can’t “skip to end, rewind one, read last element” on a non-positionable stream. But you can on a positionable stream.

The default buffer of a positionable stream is endless though, so it will record everything, allowing you to rewind right back to the start. In this case, that’s a complete waste. We also only produce one element at a time, because it’s coming from a generator. So, we can use a RingBuffer of size 1 as the buffer for positioning.

(primes positioning buffer: (RingBuffer new: 1 class: Array))
    -= 1;

This is where the positioning API of Xtreams really becomes handy. We use #-= which means “From the end of the stream, rewind ‘n’ elements”. Now that we have a positioning buffer that always remembers the last element, we can skip to the end and rewind by one element and then call #get to fetch it.

The [:out | ] reading generator terminal is available in XtreamsDevelopment 508.

MLP Xtream Beach Volleyball by CrispyChris

The idea for this popped into my head on my way back from Bronycon. I’ve been thinking about what kind of print I should make for Grand Brony Gala and wanted it to be Florida and summer themed (even though summer’s almost over. Better late then never I guess) and this is what I came up with.

I also got help/advice/critique from Tsitra360. Thank you~

This will be a 11x17 print for sale and premiering at Grand Brony Gala

Xtreams 492 Update: Marshaling and Multiplexing

It’s not often I get to write a change log for an “add-on” library. Usually things move incrementally toward the original goals. But with Xtreams, sometimes we add new things, or improve old things.

Xtreams changes from v472 to v492:

  • #write:from:at: and #insert:from:at: now consistently return an Integer. This mirrors #read:into:at: which already returned an Integer consistently.
  • Added #buffering: to WriteStream, which buffers up all writes in to it, until it hits the buffer size, at which point it flushes the contents down to its destination. You can send #flush to force the contents out immediately, if you’re working with a particular protocol. This is useful when dealing with protocols that don’t internally buffer (which is most of Xtreams).
  • Included Xtreams-IRC in XtreamsDevelopment. The XtreamsDevelopment bundle is not something you should ever prerequisite. But it is a convenient way to pull everything in. We’ve decided to include applications of Xtreams in to the XtreamsDevelopment bundle. Yes, I have my eye on you StoMP.
  • Hub is removed from Xtreams-Xperiments.
  • #multiplexing is removed from Xtreams-Xperiments
  • Stitch(Read/Write)Stream can be instantiated from the class side with a semi-private API of #on:contentsSpecies: which avoids the default behavior of stitch streams, which always pre-fetch the very first substream to stitch.
  • Added #monitoring:every: to (Read/Write)Stream, which allows you to measure the through-put on a stream. This is particularly useful when transferring files and other large data across a network.
  • ObjectMarshaler will now marshal/unmarshal/analyse very large integers. Previously this was a #notYetImplemented method. The format is inefficient, but integers that large are by their very nature inefficient.
  • ObjectMarshaler will now create light-weight shadow classes for classes that do not exist when unmarshaling objects. The classes are full subclasses of Object with getters/setters created on them for their instance variable names.
  • Xtreams-Multiplexer package added, which contains the previously implemented #multiplexing and Hub behavior. It now has a sliding ChannelWindow to avoid stream saturation for unread data. As well as being promoted from Xperiments, it now has tests. 

* Items in bold represent new behavior / API.

Watch on


Polycephaly finds new home in Xtreams-Grid

When I first proposed Polycephaly to Arden Thomas, the product manager, and wrangled Martin Kobetic in to help me; I was unsure if we were really going to help our customers solve real world problems with it or not. Polycephaly, meaning “Many Heads” is a framework like Squeak’s Hydra. It lets you run multiple Smalltalk Images simultaneously, with one image controlling the “drones”. Ironically, the drones run headless, so despite it being called Many Heads it really should have been called Many Headlesses.

To our delight, customers started emailing in with reports of its usefulness in their development and production environments. We also got a lot of feedback too. Polycephaly used stdin/stdout to communicate over pipes between the master and drone. Any customer code that just so happened to write out to Stdout or read from Stdin would break Polycephaly.

Customers found it very useful to spawn 3-5 extra images and give them work to better utilize their multi-core machines. But they also wanted to do the same over the network. And so, Polycephaly-NetworkVirtualMachine was published by Holger Kleinsorgen.

Another criticism was how VirtualMachines worked; specifically it was not useful as a load balancer. So Runar Jordahl blogged about his solution, which round-robin’d “jobs” to different VirtualMachine instances.

Meanwhile, I was somewhat unimpressed with the inability for a drone to respond in a streaming fashion back to the master; or for the master to supply job objects in a stream either. This meant that as soon as a drone had completed its job, it would typically lock up in communication with the master through BOSS.

So, the technical debt on Polycephaly finally reached a turning point when a review was written on the VWNC mailing list and I was resolved to fix the problems. And what luck, I had an engineering trip planned to travel to Ottawa where two of my colleagues live, and then to Victoria (Vancouver Isle.) where another one of my colleagues live. It just so happens that these colleagues could help me and face to face time is invaluable.

I present to you Xtreams-Grid, a VisualWorks and ObjectStudio solution for working with multiple images on your local machine and across a network. Here is the features list when compared with Polycephaly:

  • Remove drones and remote masters. Drones can be activated via inetd style, or be pre-configured. Drones can connect to a pre-determined master, or be listening for a master(s) to connect.
  • Sockets instead of Pipes, no more Stdin/Stdout breaking the system.
  • Streaming results and arguments to-and-fro. The stream is now multiplexed too, with high through-put using the Xtreams-Multiplexer protocol.
  • VirtualMachines now acts as a load balancer with a “job” style API and can easily be used with #promise.
  • When starting a VirtualMachine locally, you can specify a different image to use and/or a different virtual machine executable.
  • Xtreams-Grid has actual documentation, where Polycephaly was still a work-in-progress and had none.
  • If you have the Xtreams parcels available (they will be updated in VisualWorks 7.9 at some point in the nearish future); an image no longer has to be saved, the Xtreams drone script will auto-load Xtreams-Grid and its prerequisites on start up.

It also has a much more sensible name. Every one knows what grid computing is and typically looks up ‘grid’ when trying to find it. Polycephaly, while having a very unique name that is highly recognizable, was always a bit of a tongue in cheek job. You can consider it as a past-tense code name at this point.

The API of Xtreams-Grid is very similar to Polycephaly, though not 100% the same. There’s no #timeout:do: method any more, since you can use [] onTimeout: aDuration do: aBlock. VirtualMachines is a complete rewrite and the #do:environment: API is gone completely.

There are new tricks up its sleeve though. Consider, for a moment, this peculiar little example:

evens := machine do: [ :odds | odds collecting: [ :e | e + 1 ] ] with: (1 to: 9) reading.

In this case, we are streaming in odd numbers to the drone and the drone will increment each element in the stream by one, then stream the results back out to the master. The master can leisurely read results, only taxing the drone as required.

As well as this, the drone can consume multiple streams as arguments from the master. It does not need to consume each argument stream in full; since underneath the hood is the Xtreams-Multiplexer protocol, which works similarly to the SSH2 multiplexing protocol.

There is lots to explore here. There are tests which demonstrate some of these capabilities, as well as updated documentation at Or, more specifically:

The latest version of XtreamsDevelopment as of this writing is 526.

Super Mario Bros in ตรัง
บ้านเพื่อนของ@mimpanita สายxtream
กำลังจะสร้างRamp ในบ้าน! (ช่วงโปรโมท)
ย้ง!ทำให้ได้น่ะ ไม่งั้นเลิกคบ!!!!

Watch on


Code coverage for Xtreams using StasisField to detect what has and hasn’t been called in the image. StasisField is an experimental code-coverage technique that unloads code from the image and dynamically reloads it back in to the image when a method is called. It requires no changes to the VM and once a method has been resumed, there is no further cost to the code-coverage mechanism, making it one of the most efficient techniques for reducing runtime memory and for measure code-coverage.

Watch on

#KatyPerry “This Is How We Do” - Disfrútala por HTTP:// - Somos buena música las 24 horas, sin comprometer tu consumo de megas, y en HD. Somos XTREAM FM VENEZUELA.