Python says hello to Ruby

Today we’re going to finish our REPL that we started a couple of posts ago. We’ll add support for switching among available DLR languages and for execution of multi-line snippets of code. We’ll then use the REPL to show how IronPython and IronRuby can interact with each other.

Let’s get right to the final REPL code:

# github: repl.rb
load_assembly 'Microsoft.Scripting'
Hosting = Microsoft::Scripting::Hosting
Scripting = Microsoft::Scripting

class REPL
  def initialize
    @engine = IronRuby.create_engine
    @scope = @engine.create_scope
    @exception_service = @engine.method(:get_service).of(Hosting::ExceptionOperations).call
    @language = "rb"
  end

  def run
    while true  
      print "#@language> "
      line = gets
      break if line.nil?

      if line[0] == ?#
        execute_command line[1..-1].rstrip 
      else
        execute_code read_code(line)
      end
    end
  end
  
  # Reads lines from standard input until a complete or invalid code snippet is entered.
  # Returns ScriptSource that represents an interactive code.
  def read_code first_line
    code = first_line
    while true
      interactive_code = @engine.create_script_source_from_string(code, Scripting::SourceCodeKind.InteractiveCode)
      case interactive_code.get_code_properties
        when Scripting::ScriptCodeParseResult.Complete, Scripting::ScriptCodeParseResult.Invalid:
          return interactive_code
                    
        else
          print "#@language| "
          next_line = gets
          return interactive_code if next_line.nil? or next_line.strip.size == 0 
          code += next_line
      end      
    end
  end

  # Executes given ScriptSource and prints any exceptions that it might raise.
  def execute_code source
    source.execute(@scope)
  rescue Exception => e
    message, name = @exception_service.get_exception_message(e)
    puts "#{name}: #{message}"
  end

  def execute_command command
    case command
      when 'exit': exit
      when 'ls?': display_languages
      else puts "Unknown command '#{command}'" unless switch_language command
    end
  end

  def display_languages
    @engine.runtime.setup.language_setups.each { |ls| puts "#{ls.display_name}: #{ls.names.inspect}" }
  end

  def switch_language name
    has_engine, engine = @engine.runtime.try_get_engine(name)
    @language, @engine = name, engine if has_engine
    has_engine
  end
end

REPL.new.run

As you can see, we’ve encapsulated the code from previous posts into a class called “REPL”. We’ve added a support for meta commands, i. e. commands processed by the REPL itself. Any string starting with “#” chracter is treated as a meta command. The commands recognized are “#exit” and “#ls?”. The former terminates the REPL and the latter displays the available languages. If the command you enter is not one of these two we try to get an engine of that name and change the current language of the REPL. For example, “#py” makes Python the current language, which is indicated by the prompt “py>”.

More of DLR Hosting API

We’ve used a couple of Hosting API concepts that we’ve not explained yet. They are, in order of appearance:

  • Engine Services

    ScriptEngine represents a language in Hosting API. We already used it for code execution (ScriptEngine.Execute) and performing dynamic object operations (ScriptEngine.Operations). Apart from these basic functionality an engine can also provide various services that might or might not be language specific. The idea here is that if languages and hosts agree on a new dynamic language service API the languages can implement it and the hosts can consume it without modifications to DLR hosting API. It’s up to each language whether it provides a particular service, so the host should be prepared for the case that some of the languages it hosts might not respond to a service request.

    Services are identified by the CLR type that provides the service API. The service we use in our REPL is ExceptionOperations from Microsoft.Scripting.Hosting namespace. It provides API for handling exceptions, formatting stack traces in a language specific way, etc. We use it in execute_code method to retrieve message and name from an exception raised by the executed script. A default exception operations are provided by DLR if the language doesn’t provide implementation.

  • Script Source

    So far we’ve used ScriptEngine.Execute for execution of code stored in a string. ScriptEngine also provides a set of methods prefixed CreateScriptFrom- that create ScriptSource objects representing source code stored in a string, file, stream, an editor buffer, etc. ScriptSource is mainly designed for hosts that need to provide a custom storage for the code, handle encoded code, analyze code, or pre-compile the code so that multiple executions are faster.

    Each ScriptSource instance is also tagged by a value of SourceCodeKind enumeration. This value tells the language parser how to parse the code: whether it should be parsed as an expression, a single statement, multiple statements, like it was a file, or an interactive code snippet. The default value is AutoDetect meaning that the parser should parse any piece of code and decide from its structure what kind of code it is.

    Internally, ScriptEngine.Execute creates a string based ScriptSource with code kind auto-detection and executes it.

    In our REPL, read_code method creates an interactive ScriptSource from a string that user entered. It then queries the ScriptSource for its code properties. ScriptSource asks the parser of the language is it bound to to parse the code and determine whether it is correct and complete, incorrect (has a syntax error), correct but its last token is incomplete (for example, an unterminated string literal), correct but the last statement is incomplete (for example, “1+”). These properties are captured by ScriptCodeParseResult enumeration in Microsoft.Scripting namespace. We use this value in read_code to determine if we should continue reading lines from the input. We continue reading until we get a complete or invalid code.

.NET interop

We have just explained what Hosting APIs we use in the REPL, yet there is still some magic going on in how we call these APIs from Ruby. Let’s look at .NET interop involved here.

  • Generic method call

    The first piece of .NET interop magic is used in REPL#initialize method. We have an instance of ScriptEngine in @engine variable and we want to get its ExceptionOperations service. ScriptEngine.GetService<TService> is a generic method. To invoke a generic method you need to provide the generic parameters somehow. The way it works in IronRuby right now is not ideal but one gets the job done nevertheless. Calling Kernel#method on an object gives us a Ruby Method object that represents the (generic) method. IronRuby monkey-patches this class adding Method#of method that takes a list of classes/modules that represent .NET types and returns a new instance of Method class with the method’s generic parameters bound to the given types. Once the generic parameters are bound the method is callable like any other Ruby method (i.e. via Method#call).

  • Out parameters

    The call to try_get_engine in switch_language doesn’t seem to be any special at the first glance. However, if you look at the C# implementation of ScriptRuntime.TryGetEngine method, you’ll see that it has two parameters, second of which is an out-parameter:

    bool TryGetEngine(string languageName, out ScriptEngine engine)
    

    When IronRuby encounters a call to such a method it basically takes all the out parameters in the order they appear in the signature and returns them in a CLR array along with the return value. Like if the method’s implementation was as follows:

    object[] TryGetEngine(string languageName) {
      ...
      return new object[] { return_value, engine }
    }
    

    IronRuby is able to splat CLR vector arrays. In fact, any class implementing IList interface can be splatted. Hence the parallel assignment we use in switch_language assigns the return value (a Boolean) to has_engine and the value of the out parameter to engine local variable.

Python interop preview

Finally, we can enter Ruby multi-line snippets and also some Python code!

rb> class C
rb|   def say_hello caller
rb|     puts "#{caller} says hello to Ruby"
rb|   end
rb| end
=> nil
rb> #py
py> import C
py> c = C()
py> c.say_hello("Python")
Python says hello to Ruby

This is a nice little example of Ruby-Python interop that gives you a taste of DLR potential. What’s going on here? First we declared a class C in Ruby with a method say_hello. Then we switched over to Python (using “#py” meta command) and executed some Python code. We imported “C” global variable to the local Python scope. IronRuby maps all constants defined on Object class to DLR global variables. IronPython maps its global variables to the DLR globals as well. Hence IronRuby and IronPython see each other’s variables and can exchange them.

Now that we successfully imported Ruby class object “C” into Python we can call it, right? Wait a second! You can’t call a Ruby class! Well, that’s right, but we are in Python – a call to a class object in Python makes an instance of the class. Python has no new method like Ruby has or new keyword like C# has. When IronPython sees an invocation of a dynamic object that is not its own it sends it an DLR interop message “Invoke”. IronRuby’s class objects don’t respond to this message. After all you can’t call a Ruby class. IronPython is smart and knows that if the object doesn’t respond to “Invoke” message it might respond to “CreateInstance” message. And so it sends the “CreateInstance” message to which the Ruby class responds by sending over a new instance of itself. And after a short chat between IronPython and IronRuby mediated by DLR, we store an instance of class C in Python’s local variable c.

We call say_hello method on the next line. Since we are in Python there are actually two operations involved here: “GetMember” and “Invoke”. Python first asks the object for its member say_hello (by sending “GetMember” message to the object). The Ruby object knows how to get a member – it returns an instance of Method class that represents say_hello method. Python now asks to invoke the returned object and indeed Ruby’s Method knows how to invoke itself.

 

 



Follow

Get every new post delivered to your Inbox.