Puppet Internals - Modeling and Custom Logic

In this post about Modeling with Ecore and RGen I will show how do achieve various common implementation tasks. Don't worry if you glanced over the the very technical previous post about the ECore model, If needed, I will try to repeat some of the information, but you may want to go back to it for details.

Derived Attributes

Sometimes we need to use derived (computed) attributes. Say we want to model a Person and record the birth-date, but we also like to be able to ask how old a Person is right now. To do this we would have two attributes birth_date, and a derived attribute age.

class Person < MyModelElement
  has_attr 'birth_date', Integer
  has_attr 'age', Integer, :derived => true
end

(Here I completely skip all aspects of handling date/time formats, time zones etc., and simply use a date/time converted to an Integer).

Since a derived attribute needs to be computed, and thus requires us to implement a method, we must define this method somewhere. All logic for a modeled class should be defined in a module called ClassModule nested inside the class. The definition in this module will be mixed into the runtime class.

A derived attribute is implemented by defining a method with the same name as the attribute plus the suffix '_derived'.

The full definition of the Person class then looks like this:

class Person < MyModelElement
  has_attr 'birth_date', Integer
  has_attr 'age', Integer, :derived => true

  module ClassModule
    def age_derived
      Time.now.year - birth_date.year
    end
  end
end

Derived attributes are good for a handful of intrinsic things like this (information that is very closely related / an integral part of the class), but it should not be overused as we in general want our models to be as anemic as possible; operations on models are best implemented outside of the model as functions, the model should really just contain an implementation that maintains its integrity and provides intrinsic information about the objects.

Here is an another example from the new Puppet Type System:

class PRegexpType < PScalarType
  has_attr 'pattern', String, :lowerBound => 1
  has_attr 'regexp', Object, :derived => true

  module ClassModule
    def regexp_derived
      @_regexp = Regexp.new(pattern) unless @_regexp && @_regexp.source == pattern
      @_regexp
    end
  end
end

Here, we want to be able to get the real Ruby Regexp instance (the regexp attribute) from the PRegexpType based on the pattern that is stored in string form (pattern). Derived attributes are by default also virtual (not serialized), volatile (they have no storage in memory), and not changeable (there is no setter).

Here is an example of using the PRegexpType.

rt = Puppet::Pops::Types::PRegexpType.new(:pattern => '[a-z]+')
the_regexp = rt.regexp
the_regexp.is_a?(Regexp)      # => true

Going back to the implementation. Remember, that all features (attributes and references) that are marked as being derived, must have a defined method named after the feature and with the suffix _derived. Thus, in this example, since the attribute is called 'regexp', we implement the method 'regexp_derived'. Since we do not have any storage and no generated supporting methods to read/write the Regexp we need to create this storage ourself. (Note that we do not want to recompile the Regexp on each request unless the pattern has changed). Thus, we assign the result to the instance variable @_regexp. The leading _ has no special technical semantics, but it is there to say 'hands off, this is private stuff'.

Adding Arbitrary Methods

You can naturally add arbitrary methods to the ClassModule, they do not have to be derived features. This does however go against the anemic principle. It also means that the method is not reflected in the model. Such methods are sometimes useful as private implementation method that are called from methods that represent derived features, or that are for purely technical Ruby runtime reasons (as you will see in the next example).

Using Modeled Objects as Hash Keys

In order for something to be useful as a hash key, it needs to have a hash value that reflects the significant parts of the object "as a key". Regular Ruby objects use a default that is typically not what we want.

Again, here is the PRegexpType, now also with support for being a hash key.

class PRegexpType < PScalarType
  has_attr 'pattern', String, :lowerBound => 1
  has_attr 'regexp', Object, :derived => true

  module ClassModule
    def regexp_derived
      @_regexp = Regexp.new(pattern) unless @_regexp && @_regexp.source == pattern
      @_regexp
    end

    def hash
      [self.class, pattern].hash
    end

    def ==(o)
      self.class == o.class && pattern == o.pattern
    end
  end
end

This implementation allows us to match PRegexpType instances if they are a) of the same class, and b) have the same source pattern. To support this, we simply create a hash based on the class and pattern in an Array. We also need to implement == since it is required that two objects that have the same hash also compute true on ==.

Can you think of improvements to this implementation?

(We do compute the hash value on every request, we could cache it in an instance variable. We must then however ensure that if pattern is changed, that we do not use a stale hash. In order to to know we must measure if it is faster to recompute the hash, than compute if the pattern has changed - this is an exercise I have yet to do).

Overriding Setters

Another use case is to handle setting of multiple values from a single given value - and worst case setting them cross-wise. (Eg. in the example with the Person, imagine wanting to set either the birth_date or computing from a given age in years - yeah it would be a dumb thing to do, but I had to come up with a simple example).

Here is an example from the AST model - again dealing with regular expressions, but now in the form of an instruction to create one.

# A Regular Expression Literal.
#
class LiteralRegularExpression < LiteralValue
  has_attr 'value', Object, :lowerBound => 1, :transient => true
  has_attr 'pattern', String, :lowerBound => 1

  module ClassModule
    # Go through the gymnastics of making either value or pattern settable
    # with synchronization to the other form. A derived value cannot be serialized
    # and we want to serialize the pattern. When recreating the object we need to
    # recreate it from the pattern string.
    # The below sets both values if one is changed.
    #
    def value= regexp
      setValue regexp
      setPattern regexp.to_s
    end

    def pattern= regexp_string
      setPattern regexp_string
      setValue Regexp.new(regexp_string)
    end
  end
end

Here you can see that we override the regular setters value=, and pattern=, and that these methods in turn use the internal methods setValue, and setPattern. This implementation is however not ideal, since the setValue and setPattern methods are also exposed, and if they are called the attributes value and pattern will get out of sync!

We can improve this by doing a renaming trick. We want the original setters to be callable, but only from methods inside the class since we want the automatic type checking performed by the generated setters.

module ClassModule
  alias :setPattern :_setPattern_private
  private :_setPattern_private

  alias :setValue :_setValue_private
  private :_setValue_private

  def setPattern(regexp_string)
    _setPattern_private(regexp_string)
    _setValue_private(Regexp.new(regexp_string))
  end

  def setValue(regexp)
    _setValue_private(regexp)
    _setPattern_private(regexp.source)
  end
end      

Here we squirrel away the original implementations by renaming them, and making them private. Since we did this, we do not have to implement the value= and pattern= methods since they default to calling the set methods we just introduced.

Now we have a safe version of the LiteralRegularExpression.

Defining Relationships Out of Band

Bi-directional references are sometimes tricky to define when there are multiple relations. The classes we are referencing must be known by Ruby and sometimes the model is not a a hierarchy. And even if it is, it is more natural to define it top down than bottom up order.

To handle this, we need to specify the relationships out of band. This is very easy in Ruby since classes can be reopened, and it especially easy with RGen since the builder methods are available for modifying the structure that is built while we are building it.

Here is an example (from RGen documentation):

class Person < RGen::MetamodelBuilder::MMBase
  has_attr 'name', String
  has_attr 'age', Integer
end

class House < RGen::MetamodelBuilder::MMBase
  has_attr 'address', String
end

Person.many_to_many 'homes', House, 'inhabitants'

What RGen does is to simply build the runtime model, for some constructs with intermediate meta-data recording our desire what our model should look like. The runtime classes and intermediate meta-data is then mutated until we have completed the definition of the model. The runtime classes are ready to use as soon as they are defined, but caution should be taken to use the classes for anything while the module they are in is being defined (classes may be unfinished until the very end of the module's body). Then, the first request to get the meta-model (e.g. calling Person.class.ecore) will trigger the building of the actual meta-model as an ECore model). It is computed on demand, since if it is not needed by the logic (only the concrete implementation of it), there is little point taking cycles to construct it, or having it occupy memory.

As you may have guessed, it is a terribly bad idea to modify the meta-model after it has been defined and there are live objects around. (There is nothing stopping you though if you know what you are doing). If you really need to jump through hoops like these, you need to come up with a scheme that safely creates new modules and classes in different "contexts".

In this Post

In this post I have shown some common tasks when using RGen. You should now have a grip on how derived attributes are handled and how to provide implementation logic for the declaratively modeled classes.

In a future post I will cover additional topics, such as dealing with custom data types, serialization of models, and how to work with fragmented models. It may take a while before I post on those topics as I have a bit of exploratory work to do regarding how these features work in RGen. meanwhile, if you are curious, you can read about these topics in the EMF book mentioned in the ECore blog post.

Puppet Internals - The Ecore Model

In my three previous posts about Modeling you can read about what a model is and the technologies used to implement such models, as well as learn about basic modeling with RGen - the implementation of Ecore for Ruby.

In this post I am going to take you on a tour of the Ecore model itself. In order for you to be able to have an idea of where we are when we stop at the interesting sites I am going to give you the complete map of Ecore up front. However, just like any guided city tour, I am not going to stop and show you every back alley.

A slight feeling of dizziness is excepted since we are at high meta-meta altitude, but it should be alright as long as you look where you step and go back to concrete ground when you are lost.

This post is the most theoretical in the series, and you probably want to come back to it for reference. I am sorry there are not that many examples, I promise to come back in more posts with concrete examples where all the meta-magic gets put to good use.

If you want a model to try thing out on, you can use this Car RGen model.

ECore Model

So here is the ECore model in full glory, thanks to Ed Merks (creator of Ecore) that produced this very compact while still readable diagram.

I post this map here at the beginning so you can jump back to it for reference - if you immediately want to find the sites of interest on the map, look for EClass, EReference, and EAttribute since they should be somewhat familiar from previous posts.

I am going to explain this first from a structural point of view (top-down), and then dive into the interesting details of the various elements.

Ecore Structure

All definitions in ECore live inside an EPackage which corresponds to a Ruby Module, or a Java Package (or similar concept in other technologies). If you have located the EPackage box in the diagram, you see two attributes; nsURI, and nsPrefix which are used to uniquely identify the model (the "schema identifier" if you like), and a name prefix that is typically used when generating code. You do not have to set these when you just want to model classes and use them at runtime, but they are valuable in other scenarios.

EPackage can be a sub package inside another package (this is rarely used).

An EPackage contains EClassifier elements and those come in concrete forms of; EClass, and EDataType. You have already seen a glimpse of EClass and EDataType in the previous posts; EClass is used for the classes in our model, and EDataType is used for data types that we can use in the attributes of classes (e.g. String, Integer).

An EClass contains EStructuralFeature elements and those come in concrete form of; EAttribute, and EReference. Again, in the previous posts you saw examples of both attributes and references.

ECore can also be used to describe operations (i.e. methods), but since Ecore is not an implementation language, it cannot contain the actual implementation of such operations, only a declaration of them. When using EMF for Java, the code generator will generate method stubs for the operations, and it is easy to fill in the implementation logic. When we use RGen and implement the model in Ruby, we typically do not generate code (although it is possible), and we must instead define the implementation of operations a different way if we get a model where operations are specified (and we want to provide the declared operations - there is no requirement to do so).

And finally, there is support for annotations. Any EModelElement can have annotations. RGen (and EMF) does not know what to do with these annotation other than knowing about the association. It is however an important mechanism for tools in a model toolchain, and they are typically used for things like generation of documentation, defining properties that are of importance for code generation, mark something as deprecated, etc. The use of an EAnnotation is often a light weight alternative to defining a completely separate model. There are some known annotation identities, say if we want to generate Java code from a model that we author with the RGen Metamodel Builder, and we would like JavaDoc comments to be generated in the Java source. (That is all that I planned to say about EAnnotations).

EPackage

We get the EPackage from our model's Ruby Module:

MyModel.ecore   # => ECore::EPackage

The interesting operations on EPackage are:

method	description	type
eClasses	The EClasses defined in this package	Enumeration<ECore::EClass>
eClassifiers	The EClassifiers defined in this package	Enumeration<ECore::EClassifiers>
eAllClasses	The EClasses defined in this and all super packages	Enumeration<ECore::EClass>
eAllClassifiers	The EClassifiers defined in this package and all super packages	Enumeration<ECore::EClassifiers>

Remember that we are using a model; the ECore model, and we can naturally manipulate this model like any other model. In fact, we can build the model using Ruby logic, and then generate the implementation on the fly if we want. Thus, in addition to the methods shown above, there are methods to add and remove classifiers, and to get/set the attributes of the EPackage.

EClass

An EClass has many useful features:

feature	description	type
name	The unqualified name, e.g. 'Car'	String
qualifiedName	The qualified name, e.g. 'MyModel::Car'	String
eAttributes	All attributes defined in this class.	Enumeration<ECore::EAttribute>
eReferences	All references (containment and regular) defined in this class	Enumeration<ECore::EReference>
eAllAttributes	eAttributes from this and all super classes	Enumeration<ECore::EAttribute>
eAllContainments	eReferences from this and all super classes that are containments	Enumeration<ECore::EReference>
eAllReferences	eReferences from this and all super classes	Enumeration<ECore::EReference>
eAllStructuralFeatures	all eAttributes and eReferences from this and all super classes	Enumeration<ECore::EStructuralFeature>
eAllSubTypes	all sub types of the EClass	Enumeration<ECore::EClass>
eAllSuperTypes	all super types of the EClass	Enumeration<ECore::EClass>

And again, since this is in an ECore model, it is possible to manipulate the model, we can add / remove structural features and get/set the attributes.

In essence, the ECore model API makes it possible to do the same kind of meta programming that can be done in Ruby; dynamically creating classes with attributes and references. The differences are that what we create with ECore is type safe, and that we also get the model (which we can serialize, and later deserialize to get exactly the same implementation, or we can send it to another system using another platform, and it can in turn dynamically generate the logic that is needed to operate on this model). This is what makes modeling a corner stone for polyglot programming.

ENamedElement

Simply something that has a formal name.

ETypedElement

An ETypedElement is an ENamedElement, and adds the following:

feature	description	type
ordered	(only for information stating that order is significant)	Boolean
unique	for multi valued attributes control if values are unique (references are always unique)	Boolean
lowerBound	the minimum number of occurances	Integer
upperBound	the maximum number of ocurrances	Integer
many	derived, if upperBound > 1	Integer
required	derived, if lowerBound < 1	Integer
eType	reference to the type	EClass or EDataType

Concretely, we use these attributes when modeling an attribute:

has_many_attr 'nick_names', String, :lowerBound => 0, :upperBound => -1, :unique => true

This means no nick name, or as many as you like, and they are unique.

EStrucuralFeature

An EStrucuralFeature is an ETypedElement, and it is the base class for EAttribute and EReference, and it has attributes and operations that are common to both.

The attributes of interest are:

attribute	description	type
changeable	can the value be externally set	Boolean
volatile	does the attribute have storage in the object (typically false for computed/derived values)	Boolean
transient	transient objects are omitted from serialization	Boolean
defaultValueLiteral	the default value in String form	String
defaultValue	the default value as instance of the attribute's data type	Object
unsettable	can the attribute be unset (see previous post)	Boolean
derived	is the attribute computed (requires implementing a method if true)	Boolean

Again, by looking at the ECore model, you can find additional methods.

Concretely, we use these when defining attributes (typically only for attributes, but we may defined derived references that represent filtered sets of other references, we may define transient containments etc.)

has_attr 'reg_nbr', String, :defaultValueLiteral => "UNREGISTERED"

I will come back to how to define derived/computed features in a future post.

EAttribute

An EAttribute only adds a reference to EDataType (the type of the attribute), and the ability to denotes that it is a primary key/identity attribute (which can be useful if we are transforming the model and need to be able to determine a primary key for a class, but it is not generally needed).

EReference

As you have seen in earlier posts, references are either containment references (the diamond shaped relationships in the diagrams), or regular references (no diamond in the diagram). This is expressed with a Boolean attribute containment on the containing side, and container on the contained side (if the containment reference is bi-directional). When a reference is bi-directional, this is represented by two instances that know about each other via the eOpposite attribute.

If you study the model you will find that it is also possible to define that the relationship is based on a set of key attributes. (This is rarely used, but of value for models that should be easily transformed to database schemas).

Gratefully we do not have to deal with all these details when defining references using the RGen Metamodel Builder. You have already seen in the previous posts how easy it is to create references - filling in the containment/container, and eOpposite is done for us, but now you know about the structure that is actually build inside the Ecore model.

EMF tutorial, Fun with Graphic Modeling in Eclipse

If you are interested in playing with Ecore models and want to use a graphical tool to do so, you can checkout this tutorial by Vogella. (If you want to do this, it is best to start with an Eclipse IDE and not install it into your Geppetto (the Puppet IDE), as it will bloat your Geppetto with a complete Java development environment).

I almost always use the graphical tools at the start of a project to organize my ideas even if I later do not use modeling technology in the implementation. I find that it forces me to have good definitions of what things are. If I can not express it in a model, or the model turns out to be horribly ugly, then I probably do not understand the domain well enough (or the domain is horribly messy to begin with...)

Further Reading

If you are interested in more in-depth information about Ecore, you can checkout this EMF book, although being Java and Eclipse centric the Ecore part itself is generic.

Closing Remarks

I am sorry about all the theory and documentation flavor of this post. I wanted to give you enough details about what happens under the hood while not dragging it out for too long. As you probably have understood, you really do not need to think about all these things when just defining and using models as an implementation tool, but it is valuable to know that it is there and what it can do for you should you have the need for polyglot programming, code generation, generation of data base or data schemas, transformations of data, etc.

In posts to follow I will show how to add operations to the modeled classes for things like being able to store modeled objects in hashes, how to define derived attributes etc.

(Ok, you can breathe normally again).

Puppet Internals - Using Model Meta Data

In this post about modeling with Ecore and RGen I will explain the operations that allows us to navigate the model, and how to generically get and set elements / values in the model, as well as how to get to the meta-model from model objects.

e-ness

As you will see, methods that operate on, or make use of Ecore meta data are typically named with an initial lower case 'e'. They also use Camel Cased names to make these operations as similar as possible to the corresponding operations in EMF / Java. Whenever you see such 'e' methods, they are about the Ecore aspects of the object (instance), or its class (access to the meta-model). In everyday speak we can talk about this as the object's e-ness (since it is much easier to say than "the meta aspects of..."

In this post I will show the various basic 'e' operations with examples. If you want to get the source of the model used in the examples, the final version is available in this gist.

Getting Content

One of the first things you typically want to do is to get the content of a model generically (that is, without any explicit knowledge about a particular model).

First, we need to make a couple of adjustments to the example model I showed in the previous post. RGen comes with some built in navigations, but not all that are available in the EMF (Java) implementation of Ecore. So, we have a module in Puppet that should be included to get full support.

We need to add:

require 'puppet'
require 'puppet/pops'

And then in our base class (that all our domain model classes inherit form) define it like this:

class MyModelElement < RGen::MetamodelBuilder::MMBase
  include Puppet::Pops::Containment
  abstract
end

Getting All Contents

A common operation is to iterate over all containments in an object. This is done with the method eAllContents which is typically called with a block receiving each contained element. It can also be called without a block to get a Ruby Enumerable.

The eAllContents method does not include the object it is invoked on, only its content is included in the enumeration. It visits all contained recursively in depth first order where parents appear before children.

We can try this out with the Car model.

engine = MyModel::CombustionEngine.new
car = MyModel::Car.new
car.engine = engine

car.eAllContents.map {|element| "#{element}" }

# => ["#<MyModel::CombustionEngine:0x007ff8a1b606d8>"]

In the Puppet implementation of the future parer, the ability to iterate over all contents is used when validating models. In particular the model produced by the parser.

Getting All Containers

If we have a model element, and would like to traverse all of its containers (until we reach the root) we use the eAllContainers. If we just want the immediate container we use eContainer.

# continuation of the previous example
engine.eContainer                       # MyModel::Car
engine.eAllContainers.to_a              # [MyModel::Car]

Not so exiting, but if we add a Garage that can contain cars - i.e. by adding this to the model:

class Garage < MyModelElement
  contains_many_uni 'cars', Car
end

And then add our car to the garage.

garage = MyModel::Garage.new
garage.addCars(car)
engine.eAllContainers.to_a              # [MyModel::Car, MyModel::Garage]

# and just to check what happens if we get all contents in the garage
garage.eAllContents.to_a                # [MyModel::Car, MyModel::CombustionEngine]

In the Puppet implementation of the future parser, the ability to search up the containment chain is used in validation (some object must be contained by top level constructs), and in order to find information such as a source text location index, and to find the loader that loaded the code (which is recorded at the root of the model).

Where am I? What is my role?

It is often useful to ask:

• Where is this object contained?
• What is this object's role in that container?

In the sample car model we have right now, this is not so valuable, since we do not have anything that can be contained in multiple places. So to make this a bit more interesting we can add the following to the model

class Car < MyModelElement
  # as before AND...
  has_attr 'reg_nbr', String, :defaultValueLiteral => 'UNREGISTERED'
  contains_one_uni 'left_front', Wheel
  contains_one_uni 'right_front', Wheel
  contains_one_uni 'left_rear', Wheel
  contains_one_uni, 'right_rear', Wheel
end

RimTypeEnum = RGen::MetamodelBuilder::DataTypes::Enum.new([:black, :silver])

class Wheel < MyModelElement
  has_attr 'rim', RimTypeEnum
  has_attr 'rim_serial', String
end 

The above example also shows how to set a default value for an attribute. Default values can only be used with single valued attributes. All other have an empty Array as their default.

Now we can create wheels and assign to the car. This also demonstrates that it is possible to give values to features in a Hash when creating the instance:

car = MyModel::Car.new(:reg_nbr => 'ABC123')
car.left_front  = w1 = MyModel::Wheel.new(:rim_serial => '1', :rim => :silver)
car.right_front = w2 = MyModel::Wheel.new(:rim_serial => '2', :rim => :silver)
car.left_rear   = w3 = MyModel::Wheel.new(:rim_serial => '3', :rim => :silver)
car.right_rear  = w4 = MyModel::Wheel.new(:rim_serial => '4', :rim => :silver)

And now we can start asking questions:

w1.eContainer.reg_nbr    # => 'ABC123'
w1.eContainingFeature    # => :left_front
w2.eContainer.reg_nbr    # => 'ABC123'
w2.eContainingFeature    # => :right_front

In the Puppet implementation these operations are typically used when generating error messages. An error is found in some deeply nested / contained element and a message should inform the user about where it is being used / the role it plays. Sometimes also used in validation when something is valid or not depending on the role it plays in its container.

Generic Get

Features can be read generically with the method getGeneric. Using the wheel example above:

car.getGeneric(:left_front).rim_serial  # => '1'
w1.getGeneric(:rim)                     # => :silver

There is also a getGenericAsArray which returns the value in an Array, if it is not already one (which it always is when the feature is multi-valued).

The method eIsSet can be used to test if a feature has been set. This returns true if the value has any other value than the feature's default value (and if default value is not defined, if the feature is nil).

car2 = MyModel::Car.new
car2.reg_nbr                 # => 'UNREGISTERED'
car2.eIsSet(:reg_nbr)        # => false
car2.reg_nbr = 'XYZ123'
car2.eIsSet(:reg_nbr)        # => true

This means we can define the value that represents "missing value" per single valued feature, and as you will see below we can reset the value to the default.

At present we do not make use of the generic operations anywhere in the Puppet implementation as the validation logic is aware of the individual classes. We may make use of it to add additional generic validation that is driven by meta data alone. We will probably also use this when we get to the catalog and resource type models.

Generic Mutating Operations

As you may have guessed, it is also possible to generically modify objects. The methods are:

• setGeneric(name, value)
• addGeneric(name, value)
• removeGeneric(name, value)
• eUnset(name) - sets feature to its default value (or nil/empty list if there is no default)

Here is an example using eUnset to return the car to the default for its reg_nbr:

carxyz123 = MyModel::Car.new(:reg_nbr => 'XYZ123')
carxyz123.eIsSet(:reg_nbr)                          # => true
carxyz.reg_nbr                                      # => 'XYZ123'
carxyz.eUnset(:reg_nbr)
carxyz123.eIsSet(:reg_nbr)                          # => false
carxyz.reg_nbr                                      # => 'UNREGISTERED'

So, what is missing from the picture?

You may have noticed it already. We did add an eAllContents, and eAllContainers to each model class by including the Puppet::Pops::Containment module, but these only operate on containment references.

• How can you get all features including attributes and regular references?
• Why are these not directly available on all model objects as individual methods?

The reasons for the design are that it is very common to navigate to the container, or to contained children e.g. for validation purposes, but these operations typically result ending up in logic that has specific knowledge about a particular class, and there we are in a context where we already know about all of the attributes and references and we can just use them directly.

While it would be possible to provide direct access to almost all e-ness methods directly, there is a limit to how much bloat we want in each class. Therefore, all other e-ness operations we may want to perform has to be written in a slightly more round-about fashion where we navigate to the meta model and get the information there ourselves.

All the information we need is available in the meta-model, and the following sections show how this is done.

Getting the Meta Model

We can get the entire meta-model from the Ruby module by calling the method ecore:

MyModel.ecore                  # => ECore::EPackage

From there we can navigate the contents of the entire package. As an example, one of the methods, eAllClasses, gives us all defined classes. Here is what happens if you try this in irb on MyModel (notice the map to each class' name to get something meaningful as output):

 MyModel.ecore.eAllClasses.map {|ec| ec.name }
 => ["MyModelElement", "Engine", "CombustionEngine", "ElectricalEngine", "Car",
     "ServiceOrder", "Garage", "Wheel"]

We can also get to the meta model element for each individual class in our model by calling the method ecore on that object's class. (We can not use this on the values of attributes since they are basic Ruby types, and thus do not have any e-ness).

car = MyModel::Car.new
car.class.ecore               # => RGen::ECore::EClass

Oh, look we got something back called an EClass which is the meta-model representation of a class. The EClass has many useful methods to get information about the class, its attributes, references, containments etc. As an example, the method eAllAttributes returns an Enumerable with ECore::EAttribute elements that describes all of the attributes for the class and all of its superclasses.

If you thought it just started to get interesting, don't worry, I will come back with more about about the ECore model in the next post.

In this Post

In this post you have seen how a model can be navigated, and how we can find the meta-data for elements in the model. In the next post I am going to dive deeper into the Ecore model itself.

Puppet Internals - Basic Modeling with RGen

In the previous post about modeling I covered the basic principles of models, i.e. - A model is an abstraction, a model that describes a model is a meta-model, and finally a model that defines the grammar for how meta-models are written is called a meta-meta model.

Since this may sound like hocus-pocus, I am going to show concrete examples of how this works. Before we can start doing tricks with meta-models, we must have something concrete to model, and we must learn the concepts to use to express these models, and that is the focus of this post.

I am going to use the RGen Metamodel Builder API, a Ruby DSL for creation of meta models to illustrate simply because it is the easiest way to create a model.

Modeling Process - an example

When we are modeling, we typically produce the model in several iterations - first just starting loosely with the nouns, verbs, and attributes of our problem domain. We recently did this exercise for a first version of Puppet's future Catalog Builder, and it went something like this:

• We need to be able to build a Catalog
• A Catalog contains Resources
• A Resource describes a wanted state of a particular resource type in terms of Properties
• A Resource also describes Parameters that are input to the resource provider performing the work
• It would be great if we could define a catalog consisting of multiple catalogs to be applied in sequence - maybe Section is a good name for this
• While the catalog is built we need to be able to make future references
• We need to track where resources in the catalog originates (they may be virtual and exported from elsewhere)
• ... etc.

Once we settled on a handful of statements, we could then whiteboard a tentative model, and reason about the implications. We continued with all the various sets of requirements, and we took pictures of the whiteboard, and made some written notes to remember what we did.

Next step was to document this more formally. I used the graphical modeling tool for Eclipse to make a diagram. Using this diagram, we then walked through it, continued discussing / testing use cases, and revising the diagram. The revised diagram after a couple of iterations looks like this:

At this point in the lifecycle of the model, it is fastest to make changes in the graphical tool, and it is much faster and easier to communicate what the model means than if everything is written in a programming language. After a while though, it becomes somewhat tedious to describe all the details in the diagram, and while we can use the output directly from the tool to get a version in Ruby, we really want to maintain the original model in Ruby source code form.

We then made an implementation of the model in Ruby that you can view on github.

The work of making the diagram and the implementation in Ruby took me something like 4 hours spread out over the breaks we took while discussing and white-boarding.

We expect to revise this model several times until it is done. If we want to generate a graph, we can now go in the other direction and create input to the graphical tool from the code we wrote in Ruby. This is basically a transformation from the Ruby code to Ecore in XMI since that is what the graphical tool understands.

The reason why I included this real life example is to show something that is relevant to the Puppet community. I am however going to switch to toy examples to demonstrate the various techniques when modeling since any real life model tends to overshadow the technique with domain specific issues - i.e. this post is not about the Catalog-model, it is about modeling in general.

Concepts

The basic concepts used when modeling are:

• We model classifiers / classes
• Classes have named features
• A feature is an attribute of a simple data type, or a relationship to another class
• Attributes can be multi valued
• A relationship can describe containment
• A relationship can be uni-, or bi-directional (a bi-directional reference allows us to conveniently navigate to an object that is "pointing to" the object in question).
• Relationships can be multi valued at one side (or both sides for non containment relationships).
• The term multiplicity is sometimes used to denote optional/single/multi-value in relationships, e.g. 0:1, 1:1, 0:M, 1:M, or M:M
• An abstract class can not be instantiated
• Ecore also contains modeling of interfaces, this is mostly useful for Java and I am going to skip explaining these.

Classes

When using RGen's Meta Model Builder, the classes in a model are simply placed in a module and made to inherit the base class RGen::MetamodelBuilder::MMBase. It is common to let each meta model define an abstract class that signals that it is part of that model.

require 'rgen/metamodel_builder'

module MyModel
  # Let RGen know this module is a model
  extend RGen::MetamodelBuilder::ModuleExtension

  # An abstract class that makes it easier to check if a given
  # object is an element of "MyModel"
  #
  class MyModelElement < RGen::MetamodelBuilder::MMBase
    # We make this class abstract to make it impossible to create instances of it
    abstract
  end

  class Car < MyModelElement
    # definition of Car
  end
end

At this point we cannot really do much except create an instance of a Car

require 'mymodel'
a_car = MyModel::Car.new

Attribute Features

Attribute features are used to define the attributes of the class that are represented by basic data types. Attributes have a name, a type, and multiplicity. A single value attribute is defined by calling has_attr, a multivalued attribute by calling has_many_attr.

The basic data types are:

• String
• Integer
• Float
• Numeric
• Boolean
• Enum

You can also use the completely generic Object type, but this also means that the model cannot be serialized, so this should be avoided. It is also possible to reference implementation classes, but this should also be avoided for serialization and cross platform reasons.

The Enum type can be specified separately, and given a name, or it may be defined inline.

EngineEnum = RGen::MetamodelBuilder::DataTypes::Enum.new([:combustion, :electric, :hybrid])

class Car < MyModelElement
  has_attr 'engine', EngineEnum
  has_attr 'doors', Integer
  has_attr 'steering_wheel', String
  has_attr 'wheels', Integer
end

If we have an attribute that should be capable of holding many values, we use has_many_attr. In the example below, an enum for extra equipment is used - it does not have to be an enum, a multi valued attribute can have any basic data type.

ExtraEquipmentEnum = RGen::MetamodelBuilder::DataTypes::Enum.new([
  :gps, :stereo, :sun_roof, :metallic
])
class Car < MyModelElement
  has_many_attr 'extras', ExtraEquipmentEnum
  # ...
end

The resulting implementation allows us to set and get values.

a_car = MyModel::Car.new()
a_car.engine = :combustion
a_car.extras = [:gps, :sun_roof]

For multi-valued attributes, we can add and remove entries

a_car.addExtras(:metallic)
a_car.removeExtras(:sun_roof)
a_car.extras   # => [:gps, :metallic]

If we attempt to assign something of the wrong type, we get an error:

a_car.addExtras(:catapult_chair)

=> In MyModel::Car : Can not use a Symbol(:catapult_chair) where a
   [:gps,:stereo,:sun_roof,:metallic] is expected

We can also specify additional metadata for each attribute, but I will return to that later.

Relationship Features

All non basic data types are handled via references. There are two types of references; containment, and regular. A containment reference is used when the referenced element is an integral part of the object - e.g. when it is an attribute of the object, when something cannot and should not be shared between objects (one particular wheel can only be mounted on once car at the time), and when they should be serialized as part of the object that holds the reference. All other references requires that the referenced object is contained somewhere else (although this is not quite true as we will see later when we talk about more advanced concepts, it is a reasonable conceptual approximation for now).

In order to have something meaningful to model, lets expand the notion of Engine.

class Engine < MyModelElement
  abstract
end

FuelTypeEnum = RGen::MetamodelBuilder::DataTypes::Enum.new([:diesel, :petrol, :etanol])

class CombustionEngine < Engine
  has_attr 'fuel', FuelTypeEnum
end

class ElectricalEngine < Engine
  has_attr 'charge_time_h', Integer
end

# skipping HybridEngine for now

Containment

We can now change the Car to contain an Engine. When we do this, we must also decide if the Engine should explicitly know about which car it is mounted in or not. That is, if the containment relationship is uni- or bi- directional. Let's start with a uni-directional containment:

class Car < ModelElement
  contains_one_uni 'engine', Engine
  # ...
end

We can now create and assign an Engine to a Car.

a_car = MyModel::Car.new
a_car.engine = MyModel::CombustionEngine.new

If we want to make the containment bi-directional:

class Car < MyModelElement
  contains_one 'engine', Engine, 'in_car'
  # ...
end

The assignment works as before, but we can now also navigate to the car the engine is mounted in. We achieved this, by defining the reverse role 'in_car' for the bi-directional containment. Now we can do this:

an_engine = MyModel::CombustionEngine.new
an_engine.in_car            # => nil
a_car = MyModel::Car.new
a_car.engine = an_engine
an_engine.in_car            # => Car

The semantics of containment means that if we assign the engine to another car we will move it!

# continued from previous example
another_car = MyModel::Car.new
a_car.engine                # => CombustionEngine
another_car.engine = an_engine
another_car.engine          # => CombustionEngine
a_car.engine                # => nil

This may seem scary, but it is actually quite natural. If we find that we want to contain something in more than one place at a given time our model (and thinking) is just wrong, and one of the references should not be a containment reference. Say if we have a ServiceOrder (imagine for the purpose of repairing an engine), the engine is not ever contained in the order (it is still mounted in the car). Model-wise we simply use a non-containment / regular reference from the ServiceOrder to an Engine.

We specify different kinds of containment with the methods:

• contains_one
• contains_many
• contains_one_uni
• contains_many_uni

Regular References

Regular references are defined with one of the methods:

• has_one, uni-directional
• has_many, uni-directional
• one_to_many, bi-directional
• many_to_one, bi-directional
• many_to_many, bi-directional

We can now define a ServiceOrder for service of engines:

class ServiceOrder < MyModelElement
  has_one 'serviced_engine', Engine
end

And we can use this:

an_engine = MyModel::CombustionEngine.new
a_car = MyModel::Car.new
a_car.engine = an_engine
a_car.engine                   # => CombustionEngine
so = MyModel.ServiceOrder.new
so.serviced_engine = an_engine
a_car.engine                   # => CombustionEngine

As you can see, since the relationship is non-containment, the engine does not move to become an integral part of the service order, it is still mounted in the car (which is exactly what we wanted).

For the bi-directional references, the reverse role is required. When using these, it is important to consider the cohesion of the system - we do not want every piece to know about everything else. Therefore, choose bi-directional references only where it really matters.

Testing the examples

You can easily try the examples in this gist. You need to have the rgen gem installed, and then you can run the examples in irb, and try things out.

In this Post

In this post you have seen examples of how to build a model with RGen, and how it can be used. As you probably have noticed, you get quite a lot of functionality with only a small amount of work. How much code would you have to write to correctly support fully typed many to many relationships? (And then discover that you want to model it differently).

I hope this post has showed the usefulness of modeling even when no fancy modeling tricks have been used simply because of the robust, type and referentially safe implementation that we get from a small an concise definition.

Puppet Internals - Introduction to Modeling

As you probably know, Puppet creates a Declarative Catalog that describes the desired system state of a machine. What you probably have not thought about is that this means that Puppet is actually Model Driven. The model in this case is the catalog and the descriptions of the desired state of a set of resources that is built by Puppet from the instructions given to it in the Puppet Programming Language.

While Puppet has always been based on this model, it has not been built using modeling technology. What is new in the future parser and evaluator features in Puppet is that we have started using modeling technology to also implement how Puppet itself works.

In this post, I am going to talk a bit about modeling in general, the Ecore modeling technology and the Ruby implementation of Ecore called RGen.

What is a "model" really?

If at first when you hear the word model, you think about Tyra Banks or Marcus Schenkenberg and then laugh a little because that is obviously not the kind of models we are talking about here. You were actually not far off the mark.

A model is just an abstraction, and we use them all the time - they are the essence of any spoken language; when we talk about something like a "Car" we are talking about an unspecific instance of something like "a powered means of transportation, probably a 4 door sedan". Since it is not very efficient to communicate in that way, we use the abstraction "Car". Fashion models such as Tyra or Marcus are also abstractions of "people wearing clothes" albeit very good looking ones (perhaps you think they represent "people that do not wear enough clothes", but they are still abstractions).

We can express a model concretely using natural language:

A Car has an engine, 2 to 5 doors, a steering wheel, breaks, and 3-4 wheels.
. . .

We can also express such a model in a programming language such as Ruby:

class Car
  attr_accessor :engine
  attr_accessor :doors
  attr_accessor :steering_wheel
  attr_accessor :breaks
  attr_accessor :wheels

. . .
end

As you can see in the above attempt to model a Car we lack the semantics in Ruby to declare more details about the Car's attributes - there is no way to declaratively state how many doors there could be, the number of wheels etc. There is also no way to declare that the engine attribute must be of Engine type, etc. All such details must be implemented as logic in the setter and getter methods that manipulate a Car instance. While this is fine from a runtime perspective (it protects us from mistakes when using the class), we can not (at least not easily) introspect the class and deduce the number of allowed doors, or the allowed type of the various attributes.

Introspection (or reflection) is the ability to programmatically obtain information about the expressed logic. Sometimes we talk about this as the ability to get meta-data (data about data) that describes what we are interested in.

While Ruby is a very fluid implementation language, in itself it is not very good at expressing a model.

Modeling Language

A modeling language (in contrast to an implementation language such as Ruby) lets us describe far more details about the abstraction without having to express them in imperative code in one particular implementation language. One such family of "languages" are those that are used to describe data formats - they are referred to a schemas - and you are probably familiar with some of them such as XmlSchema, JsonSchema, and Yamlschema. These schema technologies allows us to make declarations about what is allowed in data that conforms to the schema and we can use this to validate actual data.

A schema is a form of modeling language. What is interesting about them is that they enable transformation from one form to another! Given an XmlSchema, it is possible to transform it into a corresponding Yaml or Json schema, and likewise transform data conformant with one such schema into data conformant with the transformed schema. (The problems doing this in practice has to do with the difference in semantic power between the different technologies - we may be able to express rules/constraints in one such schema technology that does not exist the others).

Schemas and Meta Models

When we look at a schema - we are actually looking at a meta model; a model that describes a model. That is if we describe a Car in Json we have a Car model:

{ "engine": "combustion",
  "steering-wheel": "sport-leather",
  "wheels": ...
}

And if we describe a schema for it:

{ "title": "Car Schema",
  "type"; "object",
  "properties": {
    "engine": { "type": "string"},
    "steering-wheel": { "type": "string" },
    . . .
  }
  "required": ["engine", "steering-wheel", ...]
}

We have a Car meta-model.

In everyday speak, we typically refer to the schema as "schema" or "model" and simply ignore its "meta status". But since we are on the topic of meta models - what we can do now is to also express the meta model as a model - i.e. what is the schema for a jsonschema? Here is an excerpt from the Json "Core/Validation Meta-Schema"

{
  "id": "http://json-schema.org/draft-04/schema#",
  "$schema": "http://json-schema.org/draft-04/schema#",
  . . .
   "title": {
        "type": "string"
    },
  . . .
  "required": { "$ref": "#/definitions/stringArray" },
  . . .
}

If you are interested in what it looks like, do download it. Be warned that you will quickly become somewhat disoriented since it is a schema describing a schema that describes what Json data should look like...

A meta schema such as that for JsonSchema is very useful as it can be used to validate schemas that describe data.

Schemas such as XmlSchema, YamlSchema, and JsonSchema are good for describing data, but they becomes somewhat difficult to use in practice for the construction of software. There are other modeling languages that are more specifically targeting software system constructs. There are both graphical and textual languages as well as those that have both types of representations.

What we are specifically interested in is an Object modeling language, which I will explain in more details.

Levels of reality / meta-ness

We can organize what we just learned about a model and its relationship to a meta-model, and that all of these can be expressed as a model. Here is a table that illustrates this from most concrete to most abstract:

Meta-Level	Description
M0	Real Object - e.g. the movie "Casablanca on a DVD"
M1	User Model / Instance Level - e.g. a computer abstraction of the DVD - aVideo = Video.new('Casablanca')
M2	Meta Model - e.g. defines what Video is, its attributes and operations
M3	Meta meta model (the modeling language/grammar) - e.g. defines how the definition of "what a Video is", is expressed.

We very rarely talk about meta-meta models (this is the technology we use to implement a meta-model), and we also typically leave out the meta word when talking about a meta-model (i.e. just using the word model). We also typically talk about an "instance of a model" as being just a "model", without any distinction about its form; as live objects in memory that we can manipulate, or serialized to disk, stored in a database etc. It is only when we talk explicitly about modeling and modeling technology that we need to use the more precise terms, most of the time, it is perfectly clear what we are referring to when we use the word "model".

Object Modeling Language

An Object Modeling Language is a language that directly supports the kinds of elements we are interested in when constructing software. E.g. classes, methods, the properties of objects. You probably heard of one such technology called UML - Unified Modeling Language - this is a broad modeling technology and is associated with object oriented software development methodologies such as Booch, OMT, Objectory, IBM's RUP, and the Dynamic Systems Development Method. This was The Big Thing in the 90's, but UML has since then more or less slid into darkness as "the way to write better software" has shifted focus. An interesting debate from 2009, can be found here.

There is however a very useful part of the UML technology that is often overlooked - the so called Meta Object Facility (MOF) that sits at the very core of UML. It contains the (meta-meta) model that UML itself is defined in. MOF plays the same role for models as what Extended Backus Naur Form (EBNF) plays for programming languages - it defines the grammar. Thus MOF can be said to be a domain specific language used to define meta models. The technology used in MOF is called Ecore - and it is the reference implementation of MOF. (It is a model at level M3 in the table above).

Ecore

Ecore is part of Eclipse EMF, and is heavily used within Eclipse for a wide variety of IDE applications and application development domains. In the Puppet Domain EMF/Ecore technology is used in the Puppetlabs Geppetto IDE tool in combination with additional frameworks for language development such as Xtext.

Eclipse EMF is a Java centric implementation of Ecore. There are also implementations for Ruby (RGen, and C++ EMF4CPP).

Thus, there are many different ways to express an Ecore model. The UML MOF has defined one serialization format known as XMI, which is based on XML, but there are many other concrete formats such as Xcore (a DSL built with Xtext, annotated Java, JSon, binary serialization formats, the Rgen DSL in Ruby, etc.)

Car in RGen

Here is the Car expressed with RGen's MetamodelBuilder. In the next blog post about modeling I will talk a lot more about RGen - this is just a simple illustration:

class Car < RGen::MetamodelBuilder::MMBase
  has_attr 'engine', DataTypes::Enum.new([:combustion, :electric, :hybrid])
  has_attr 'doors', Integer
  has_attr 'steering_wheel', String
  has_attr 'wheels', Integer

. . .
end

A few points though:

• RGen::MetamodelBuilder::MMBase is the base class for all models implemented with RGen (irrespective of how they are defined; using the metamodel builder, using the API directly, loading an ecore XMI file, or any other serialization format).
• has_attr is similar to Ruby's attr_accessor, but it also specifies the type and type checking is automatic.
• You can probably guess what Enum does

If you are eager to see real RGen models as they are used in Puppet, you can take a look at the AST model, or the type model.

Benefits of Modeling Technology

So what is it that is appealing about using modeling technology?

• The model is declarative, and can often be free of implementation concerns
• Polyglot - a model can be used dynamically in different runtimes, or be used to generate code.
• Automatic type checking
• Support for containment and serialization
• Models can be navigated
• Objects in the model has information (meta-data) about where they are contained ("this is the left front wheel of the car XYZ 123").
• Direct support for relationships, including many to many, and bidirectional relationships

Which I guess boils down to "Modeling technology removes the need to write a lot of boilerplate code for non trivial structures" In the following posts I will talk about these various benefits and concepts of Ecore using examples in RGen.

In this Blog Post

In this blog post I explained that a model is an abstraction, and how such an abstraction can be implemented in software i.e. hand written, or using modeling technology such as Data Schemas or one specifically designed for software such as Ecore which available for Ruby in the form of the RGen gem. I also dived into the somewhat mysterious domain of meta-meta models - the grammar used to describe a meta-model, which in turn describes something that we want be manipulate/work with in our system.

Things will be more concrete in the next post, I promise.

Puppet Internals - Puppet Type System Ruby API

One question I got several times about the Puppet Type System is how it can be used from Ruby. So, in this post I am going to show just that - how the type system can be put to good use from within functions and in types and providers. This post is all about the Puppet Type System's Ruby API.

Background

The Puppet Type System was first introduced in Puppet 3.4. The API and functionality has been extended since then, and this blog post describes what is available in Puppet 3.5. The simple, straight forward examples should also work in Puppet 3.4, but many of the more advanced types where implemented in 3.5.

The Type System is implemented using the rgen gem - so if you want to play with code that uses the type system you should have that installed, and turn on --parser future to get all the things needed loaded and ready.

The Type System Implementation

At the core, the type system is built on a Type Model. This is what defines the classes that describe a Type. This model is anemic (using a term I introduced in an earlier post) and the model is accompanied by two implementations - the TypeFactory, which is used to construct types, and TypeCalculator which performs computations involving type (is this object an instance of this type, are these types compatible, etc.).

The Type Factory

While it is possible to interact directly with the Type Model, it far more practical to do so via the TypeFactory. One important thing to remember is that almost all of the types are parameterized, and thus quite unique - the Integer type can describe a range, the String type can express min and max length, etc. Thus, while there will be many instances describing a type flowing around in the system that describe the same kind of thing, the actual type instances are individual objects. In modeling terms, the parameters of a Type are contained in their parent type Just like a specific wheel is mounted on one specific car (or no car) at any given point in time is a specific Type instance associated with a parent type (or no type). This means that whenever we want to use a type we must have a fresh instance that is not already contained in some other type. (This may change in the future, but complicates the modeling).

With that bit of theory taken care of, I can move on to showing examples.

Creating an instance of Integer:

int_t = Puppet::Pops::Types::TypeFactory.integer()

(From this point forward, I am going to simplify the examples by assuming that FACTORY is a reference to Puppet::Pops::Types::TypeFactory.)

And, just to complete the example, to test if an object is an instance of that type we just created:

Puppet::Pops::Types::TypeCalculator.instance?(int_t, 3)        # true
Puppet::Pops::Types::TypeCalculator.instance?(int_t, 'hello')  # false

Factory methods

method	description
integer	creates an integer type range from -Infinity to +Infinity
range(from, to)	creates an integer type range with given from/to, where :default denotes Infinity
float	creates a float type range from -Infinity to + Infinity
float_range(from, to)	creates a float type range with given from/to, where :default denotes Infinity
string	creates a string type with size from 0 to Infinity
enum(*strings)	creates an enumeration type from the given set of strings
pattern(patterns)	creates an enumeration type based on regular expressions
regexp(pattern=nil)	creates a regexp type, optionally parameterized with a pattern
boolean	creates a boolean type
scalar	creates an abstract scalar type
object	creates an abstract object type
numeric	creates an abstract numeric type
array_of(o)	creates an array type parameterized by the given argument, its size is 0 to Infinity
array_of_data	creates an array parameterized by the abstract type Data, its size is 0 to Infinity
hash_of(value, key_scalar)	creates a hash parameterized by the given arguments for value and key, the default is a scalar key, its size is 0 to Infinity
hash_of_data	creates a hash parameterized with scalar key and Data value.
collection	creates a collection abstract type of size 0 to Infinity
data	creates the abstract Data type
resource(type_name=nil, title=nil)	creates a resource type, optionally parameterized with resource type name, and title
host_class(class_name=nil)	creates a host class resource type, optionally parameterized with a class name
catalog_entry	creates the abstract CatalogEntry type
optional(t)	creates a type that represents the given type t or undef
variant(*types)	creates a type that represents 'one of' the given types
struct(type_hash)	creates a Struct type that fully qualifies a hash
tuple(*types)	creates a Type that fully qualifies an array
ruby(o)	creates a ruby type from the given object or class
ruby_type(class_name)	creates a ruby type representing the given Ruby class name
undef	creates a type representing undef values
type_type(t=nil)	creates a meta type, optionally parameterized with the type this is the meta type for

In addition to these type creation methods, the constrain_size method allows changing the size constraint (for types that supports this; String, Array, Hash, and the occurrence of the last type in the Tuple type's sequence. The type_of(o) method is used by the other methods when converting the given argument(s) to type. The label(t) method produces a string representation of the type.

method	description
constrain_size(t, from, to)	constrains the size of t, where from and to are the same as for an integer range
type_of(o) is	produces a type for o using the rules shown below
label(t)	produces a string representation of the type.

The type_of(o) method allows flexible specification of type parameters in Ruby, as specified by the following table

o is_a	then
Class	if the Ruby class corresponds to one of the Puppet types, e.g. String, Integer, and then that type is returned, else a Puppet Ruby type.
PAbstractType	used as is (i.e. Puppet::Pops::Types::PAbstractType)
String	the string is the classname of a Ruby class - the corresponding type is produced
any other	the type is inferred using the type calculator (see below)

Example:

FACTORY.tuple(String, Integer)
FACTORY.struct({'a' => String, 'b' => Integer})

# i.e. instead of having to do this
FACTORY.tuple(FACTORY.string, FACTORY.integer)

In summary - the type factory creates types using convenient transformation from Ruby types to Puppet types. (For details about each method, see the yardoc for the TypeFactory). For more information about what the different types represents, see the earlier posts in this series about the Puppet Type System

The Type Calculator

The other major part of the Puppet Type implementation is the class Puppet::Pops::Types::TypeCalculator (from this point on referred to as CALCULATOR in examples). The type calculator is the type inference system.

The type calculator has the same set of methods available both as instance and class methods. If a long series of operations are to be performed, it is faster to call the singleton method to get an instance, and then use what is returned for multiple operations.

The set of methods are:

method	description
assignable?(t, t2)	is t the same, or a more general type than t2
instance?(t, o)	is o an instance of the type t
equals(t, t2)	is the type t equal to the type t2
enumerable(t)	if the type is Enumerable an suitable Enumerator is produced, else nil (currently only for integer range)
infer(o)	infers the type of o and produces a generalized type (see below)
infer_set(o)	infers the type of o and produces a value dependent type set (see below)
singleton	returns the single instance of the TypeCalculator
string(t)	produces a s string representation of the given type. This is the same as calling to_s on a type instance

In addition to these methods, there are several utility methods, mostly for use by the type calculator itself - those are not considered to be API.

The set of methods makes it possible to perform all of the operations exposed in the Puppet Programming Language.

The use of most of these methods should be easy to grasp. The assignable? method performs a type check based on two types, and instance? on a type and an instance.

The infer method infers a generalization - e.g. given [1, 3.14] infers Array[Numeric], whereas the infer_set method infers a set of value dependent types, e.g. given [1, 3,14] it will produce an Array[Variant[Integer[1,1], Float[3.14, 3.14]]] where each value in the array is encoded as a unique type. This is what allows more detailed type-questions to be answered.

The Type Parser

The third and final component in the Type System is the TypeParser. It produces a type given its string representation - e.g. if you execute the example below, you get back the same type:

a_type = FACTORY.array_of(String)
Puppet::Pops::Types::TypeParser.new.parse(a_type.to_s)

This allows you to store / pass type information in String form in a Resource parameter and convert it back to a type again in a Provider. Same thing for facts, settings, or when you get data from a source that cannot produce type instances.

Other Operations on Type

The types themselves support equality (==, eql?), and they can be used as hash keys - two types that are equal hash to the same hash-key. You can also copy a type (and all of its parameters) using the copy method (which is important as you need to consider the containment rule).

The Regexp type has a method called regexp that returns a Ruby Regexp from the puppet type's pattern.

The Struct type has a method to obtain the name/type hash as a Ruby Hash.

All types are considered to be immutable once they have been fully constructed, but this is not (currently) enforced.

Typical Usage

Say we want to check the types of arguments given to a puppet function. In this case we can perform all the type checking in one go even for complex types (just calling instance?).

accepted_t = FACTORY.tuple(String, Integer)
unless CALCULATOR.instance?(accepted_t, arguments)
  raise ParseError, "Argument type mismatch. Expected: " + 
     accepted_t.map(&:to_s).join(', ') +
     ". Got: " +
     CALCULATOR.infer_set(arguments).map(&to_s).join(', ')

That is, if we accept two arguments of String and Integer type respectively this will print out what we expected and what we got. (Here I did go through the gymnastics of turning the types back to string even if this could have been written out directly as just "String, Integer").

If the function supports multiple signatures, we can obtain the type of the given argument by calling infer_set, and then test assignability of that against the signatures - this is faster than performing multiple instance? calls, as each call will need to infer the type of the given arguments from scratch.

given_t = CALCULATOR.infer_set(arguments)
case
when CALCULATOR.assignable?(signature_1_t, given_t)
  # process signature 1
when CALCULATOR.assignable?(signature_2_t, given_t)
  # process signature 2
else
  # error, not a supported signature
end

Now the gymnastics from before makes more sense since we may want to print the various signatures and state that the given did not match any of them.

Future Work

There is a new Function API in the works that will make use of the Type System, and it will also contain providing good error messages when there is a type mismatch. Until then, manual checking can be done as shown above.

There may be changes to how the containment of parameters work - right now the same type instance may have to be repeated multiple times, and it would be beneficial to be able to declare them by name and then reference rather than contain them.

Puppet Internals - Location Processing

In this post about Puppet Internals I am going to cover how source location information is handled by the future lexer and parser. This also shows a concrete case of using the adapter pattern introduced in Puppet Internals - the Adapter Pattern.

Rationale for Detailed Position Information

It is important to have detailed position information when errors occur as the user programming in the Puppet Programming Language would otherwise have to guess about where in the source text the problem is.

Positioned information is not only wanted for syntax errors and other errors immediately detected by the parser, this information is perhaps even more valuable when errors occur later and it is important to know where a particular value originated.

To date, this output has consisted of file and line only. While this is enough in most situations there are also many cases where line alone is not enough (say using an operator like '+' the wrong way, and there are five '+' on the same line), just knowing that there is something wrong with one of the '+' on line 3 is not that great. What we want is to also know the character position on the line.

Implementation Background

In the 3x Puppet Language implementation, the positioning information was limited to only contain the name of the file and the line number. In many cases this information is wrong (or rather, quite imprecise) as it relies on the lexer's current lexing position (i.e. what it has consumed from the source text) as opposed to the position of the individual tokens. The lexer may very well have advanced past the particular point where the problem is when the problem is reported by the parser.

The first implantation of the future parser (available in Puppet 3.2) contained detailed positioning (file, line and position on line). It was calculated at the same time as the lexer did its regular processing - i.e intermixed with production of the tokens that it feeds to the parser.

This proved to be complicated as the lexer needs to look ahead and thus either maintain a complex state where it is, the location of points it will potentially backtrack to, etc. This also proved to be a performance hog. Part of the performance problem was the need to compute the length of an entire expression to enable future output of the corresponding source text.

The second implementation was much more performant. The idea was that detailed position information is really only needed when there is an error and that only a fraction of the total amount of tokens have positions that end up being recorded in the catalog). Thus, the position information could be computed lazily, only recording the most basic information; the offset and length of the significant parts of the source text. It was then possible to get rid of the complex state and intermixed position calculations and instead more efficiently scan the entire input for line breaks and building an index.

This implementation was introduced in the future parser in Puppet 3.4. Unfortunately, the idea that positioning is really only needed when there is an error was wrong in this case since the 3.4 implementation has to transform all parsed syntax tree into the 3.x AST equivalent tree, and there all position information is still computed up front and stored inside each AST node (even if never used). (Bummer).

Now, in Puppet 3.5, the new evaluator is responsible for evaluating most expressions and it will only trigger the position computations when it is required; when there is an error or when the expressions must be evaluated by the 3x evaluator (catalog operations fall in this category).

While working on the performance of the lexer / parser it was also observed that it would be beneficial to serialize the constructed syntax tree as deserialization of a tree is much faster in general than again parsing the same source text. In order to be able to support this, it was obvious that the detailed information needed to be included in the model (and not only computed and kept in structures in memory). Thus, in Puppet 3.5's future parser's AST model you will find a Program element that holds the entire source text, the line offset index, and the parse tree result. (This has other benefits as well - even a broken program can be returned - even if it can not be evaluated, it makes it easier to report what is wrong).

Requirements on Positioning Information

• Be fast to compute when parsing
• Serializable / De-serializeble
• Not be too expensive to compute when details are required
• Once computed it should continue to be available
• Provide file, line, and position on line
• Provide corresponding source text given an expression
• Handle source strings in UTF-8 (i.e. multibyte encoding)

The Implementation in Puppet 3.5

A mix of techniques were used to meet the requirements.

The central concept is that of a Locator; an object that knows about the source text string; where all the lines start, and given an offset into the string can answer which line it is on, and the position on that line. This means that only the offset and length of tokens needs to be recorded and stored in the syntax nodes.

We could store a reference to the locator in every node, but that requires one extra slot per node, and would need to be handled in de-serialization (i.e. setting thousands of references when loading a single model). The offset and length are in contrast regular Integers and are fast to serialize/de-serialize.

The parser always produces an instance of Program, and it contains both the source text and the required line index. With these two, it can reconstruct the Locator (that was originally created by the lexer / parser when parsing the source). The Program is only a data container, it does not do any offset computation - that is always handled by an instance of Locator.

Here is a diagram that shows the relationship between the Program and the Locator. It also shows how individual nodes (Positioned) and their corresponding computational / cache of positioning (a SourcePosAdapter) relate to Program and Locator. Read on to learn how those work.

Positioned

All nodes in the constructed syntax tree model inherit from Positioned (except Program which is always the entire source). Being Positioned means that there is an offset and a length (but nothing more).

If we want to know the line number and the position on the line we need to find the Locator since it knows how to compute the information. We could have implemented that in the Positioned object itself, but it would clutter its implementation and it would be difficult to change the strategy for computing. This is where the SourcePosAdapter comes in.

The SourcePosAdapter

Being an Adapter (see earlier post (1) for details) means that it is bi-directionally associated with a particular object without the object knowing about it. The responsibility of the managing the relationship is entirely on the adapter side.

An Positioned object is adapted to a SourcePosAdapter by:

adapter = SourcePosAdapter.adapt(the_object)

The same instance of the adapter is always returned (it is created the first time). It is possible to ask if an object is adapted (and get the adapter) by:

adapter = SourcePosAdapter.get(the_object)

Once a SourcePosAdapter is obtained, it can answer all the questions about position. When it is created it performs a minimum of computation. When asked for something that requires a Locator is searches for the closest object that has knowledge of it and then caches this information. When this takes place for the first time, the search always goes up to the Program (root) node. On subsequent searches a parent node with a SourcePosAdapter may be encountered and the search can stop there.

The resulting structure is what is depicted in the graph.

It is worth noting that all model objects that are contained knows about their container via the somewhat mysterious method eContainer (how that works in more detail and what the difference is between a containment, and a reference is the topic for another blog post).

Example

Say we have something simple like this:

 $a = 1 + 2 * 3

The lexer produces a sequence of tokens:

 VARIABLE, 'a', offset = 0,  length = 2
 EQUAL,    '=', offset = 3,  length = 1
 NUMBER,   '1', offset = 5,  length = 1
 PLUS,     '+', offset = 7,  length = 1
 NUMBER,   '2', offset = 9,  length = 1
 TIMES,    '*', offset = 11, length = 1
 NUMBER,   '3', offset = 13, length = 1

The Parser arranges them into a tree:

When the parser parsed the expressions, it did so by evaluation rules. Here is an excerpt from the grammar.

expression
  : ...
  | expression PLUS     expression   { result = val[0] + val[2]; loc result, val[1] }
  | expression MINUS    expression   { result = val[0] - val[2]; loc result, val[1] }
  | expression DIV      expression   { result = val[0] / val[2]; loc result, val[1] }
  | expression TIMES    expression   { result = val[0] * val[2]; loc result, val[1] }
  | ...

While there is much to talk about how the grammar / parser works, this post focuses on handling of location, so the interesting part here are the calls to the loc method. It is called with the resulting model node (e.g. an ArithmeticExpression) and then one or two other nodes which may be model nodes or tokens produced by the lexer. All of the arithmetic expressions are located by their operator (loc is called with val[1] which is a reference to the operator i.e. '+', or '*' in the example).

Once the tree is built, since all of the nodes are Positioned it is possible to adapt them with a SourcePosAdapter to get the detailed computed information.

Output

The output when there is position information is simply line:pos where pos starts from 1 (the first character on the line).

Output of source excerpt (i.e. a snippet of source and a caret-pointer to the position) is not yet implemented. Maybe someone wants to take a stab at making an implementation?

UTF-8

And, lets talk about UTF-8, or rather about lexing multibyte strings.

The new implementation (in lexer2) handles multibyte source. It does this by recording the byte offsets and using a locator that is specific to single or multibyte runtime. This proved to be much faster than using the new mulitbyte support introduced in Ruby 1.9.3. You can look at the implementation in Puppet::Pops::Parser::Locator, where there are two different implementations.

Summary

This post shows both a concrete example of using the adapter pattern, as well as some high level examples about how the lexer gets its work done in a performant way.

In the Next Post

Well, to be honest, I don't really know what the next post should be about? What would you like to learn more about next - I am considering an overview of modeling as it is a fundamental concept, but there are many other topics to choose from - the implementation of the lexer, how the parser works, how error messages are generated and formatted, and then we have the evaluator to cover... - seems like this will be a very long series.