The Topic is Deferred

You may have seen the announcements about the new Deferred data type and how it makes it possible to call functions on the Puppet agent side. The main driver behind this is to enable the agent to make direct contact with services that provide secrets. Doing that from the agent is both more secure and more timely than letting the Puppet master act as an intermediate.

What I wanted to show in this blog post is a bit more detail about the Deferred data type and how you can use it in various ways.

A Regular Call

As a very simple illustration - if you do something like this in a manifest:

notify {"time_now":
  message => TimeStamp()
}

The message will be the time “now”. The point in time when the catalog was compiled. For example:

Notice: 2018-10-04T20:18:06.542425000 UTC

A short time thereafter the catalog would be applied, but in a big catalog that could be a minute or so later when the compilation and transfer to the agent is finished. If you have a workflow where you are using a cached catalog, the time could be days or weeks in the past.

I am using “time now” as it obviously will be a different value in the future.

A Deferred call

Now, with the Deferred data type, you can make the call to the new function that creates the timestamp for “now” take place on the agent, right before the catalog is applied. Like this:

notify {"time_now": 
  message => Deferred('new', [TimeStamp])
}

The output is the “same” in that it will notify with the time “now”, only that the timestamp is taken just before catalog compilation.

In case you wonder: TimeStamp.new() is exactly the same thing as new(TimeStamp) and the same as TimeStamp(). The result is the time “now”.

In other words, we deferred the call to the function to take place in the future. The Deferred data type takes the name of a function and an array of values to use as arguments to the function when it is called.

That is really all there is to it. Being inherently simple does not mean you can do complex things. And some things that appear simple at first turn out to be more complex than what you may first think. – For example, what if you want to defer a call, but you need to format the result?

Formatting the Result of a deferred call

You are probably tempted to write something like this to to do the formatting:

$d = Deferred('new', [TimeStamp])
notify {'time_now':
  message => "The time is: ${d}"
}

But that will result in something quite disappointing:

Notice: The time is: Deferred({'name' => 'new', 'arguments' => [Timestamp]})

This happens because the string interpolation (evaluating an expression and turning it into a string) takes place while compiling (as opposed to when agent applies the catalog), and there is no way to make that reference to Deferred magically jump out of its context and make the entire string interpolation operation be deferred (take place on the agent side).

You need to use something like this:

$d = Deferred('new', [TimeStamp])
notify {"time_now": 
  message => Deferred("sprintf", ["The time is: %s", $d])
}

That will make a call to sprintf on the agent side, and that call will get the format string as an argument, and it will get a just resolved second deferred that calls new(Timestamp) (also on the agent)!

In this case, since the result is a string, we can simplify it by using an inline template:

notify { 'time_now':
  message => Deferred('inline_epp', [@(END)])
    The time is: <%= TimeStamp() %>
    |- END
}

Which when used produces something like:

Notice: The time is: 2018-10-04T20:43:46.800219000 UTC

If the end result should be something that is not of String data type it gets more complicated as we do not have a function (yet) that evaluates puppet code dynamically and returns the result of the evaluation.

Let’s look at some other fun things you can do.

Using a helper function to construct a Deferred

You may want to wrap combinations of deferred values into your own helper functions to make this more readable since the use of nested Deferred() expressions is perhaps not the easiest to read and understand. You could do something like this:

notify {"time_now": 
  message => agent_sprintf("The time is: %s", agent_time_now())
}
function agent_sprintf($format, *$args) {
  Deferred("sprintf", $args)
}
function agent_time_now() {
  Deferred('new', [TimeStamp])
}

While the complexity is still there, it is at least moved out from the “business logic”. Still, this approach only works if you have a handful of things like this - clearly you don’t want an agent_this() for every possible function this().

Resolving Deferred while compiling

A Deferred is as you know by now simply a value that when resolved at a later time will call the function it references. The Deferred in itself does not define when it will be resolved. The fact that the catalog processing on the agent will resolve all Deferred values in resource parameters is just one such future that is implemented in Puppet that will be done automatically for you. You can actually use a deferred value while compiling.

If we take this example:

$d = Deferred('new', [TimeStamp])
notify {'test':
  message => "The time is: ${d}"
}

Remember the disappointing result shown earlier when doing this: you get the Deferred instead of the result of resolving it.

Now, we modify the example to:

$d = Deferred('new', [TimeStamp])
notify {'test':
  message => "The time is: ${d.call}"
}

When running this we end up with getting the time “now” in the output since $d.call results in the deferred being resolved during the compilation.

Amazing, you can use this to do functional programming! Let’s explore that a bit.

First we define a helper function which simply calls a Deferred and returns the result, and if not a Deferred returns the value:

function resolve(Any $x) {
  if $x =~ Deferred { $x.call } else { $x }
}

Now we can implement if-then-else as a function:

function if_true(
  Any $test, 
  Optional[Any] $when_true = undef,
  Optional[Any] $when_false = undef
  ) {
  if resolve($test) { resolve($when_true) }
  else              { resolve($when_false) }
}

And we can write:

$x = true
$y = $x.if_true(10, 20)
$z = (!$x).if_true(10, 20)

Which makes $y be set to 10, and $z to 20.

The fun begins when we pass a Deferred:

function do_this() {
  notice "Doing this"
}
function do_that() {
  notice "Doing that"
}
$the_value = true
$the_value.if_true(Deferred('do_this'), Deferred('do_that'))

This does exactly what you expect, if $the_value is true we will get a notice
of "Doing this", otherwise "Doing that". Contrast that result with what
you get when not using deferred values:

$the_value.if_true(do_this(), do_that()))

When evaluating that you would get a notice from both of the functions!

If you would like to play with the functions in this example you can get them from this gist.

Introspection

It is possible to do introspection of a Deferred by calling methods on a deferred value:

$d = Deferred('myfunc', [1,2,3])
notice("The function name is: ${d.name}")
notice("It will be called with args: ${d.arguments}")

Armed with that you can start creating new deferred values based on a given deferred - say by adding or reshuffling arguments, or use it for information/logging purposes.

Deferred in Types and Providers

At this point you may wonder if you can you use Deferred inside types and providers when applying the catalog. Clearly just sticking a Deferred into a resource parameter will resolve it
before the types and providers would get a chance to see it. But what if you had a function that created a Deferred, will that work?

The answer is yes, but at the moment it will be too complicated to make that have any practical use. But since this post is about fun things you can do, and possible future things you may want to do… here is how you would do this (until it does not work):

Imagine you are implementing the if_true function as a resource type; given a condition it would select the result of resolving its if_true parameter otherwise its if_false parameter.

We cannot do just this:

my_conditional { 'test':
  if_true => Deferred('do_this'),
  if_false => Deferred('do_that')
}

since that would not get passed the automatic resolution of the two deferred values and you would end up with both functions being called.

What you can do is to create a Deferred as the result of a deferred call - this works because the automatic resolution of deferred values will not resolve a returned Deferred value.

We modify the example above to wrap the two deferred values:

my_conditional { 'test':
  if_true => Deferred('new', [Deferred ['do_this']]),
  if_false => Deferred('new', [Deferred ['do_that']])
}

Short of actually implementing the my_conditional resource type (which I will not go into), how do you see the effect of this? Well, we can try it with the if_else function I showed earlier:

notice true.if_true(
    Deferred('new', [Deferred ['do_this']],
    Deferred('new', [Deferred ['do_that']]
    )

Which will give you the result:

Notice: Deferred({'name' => 'do_this', 'arguments' => []})

Exactly what we wanted - a deferred value that survived one level of resolution.

To recap: we managed to make the agent side produce a deferred value and this value was assigned to a resource’s parameter since the automatic resolution produced the wrapped
deferred value we wanted to assign to the parameter.

Now, the next step: how to resolve the deferred value, is where it gets complicated. It can be done, but there is no nice API available. When the agent is applying the catalog there is no configured context that allows functions to be called or puppet logic to be evaluated. It is certainly possible to create such a context by doing exactly what the resolver of deferred values does when it processes the catalog. That resolver does not however have any useful public methods as it resolves the entire catalog. If you can think of use cases where actually getting this to work would be of value hit me up on a puppet slack channel. (It was simply a bit too much to explore and write up for this blog post).

Some Functionality we Deferred [sic]

You may have wondered why the Deferred takes the arguments to the function as an array - this is to allow more options to be passed in the future. We may for example be adding a return type option to make the Deferred assert that the result is of a particular data type.

We may also add the simple resolve function to puppet. (If you start using your own implementation of that make sure to namespace it to your module).

As mentioned we are considering adding a function that will evaluate puppet logic, much like
what epp does, but returning any type of value.

It would also be nice to be able to give a lambda to a Deferred to let it pass that lambda on to the function when it is calling it.

And then finally, I wonder if there are real use cases for being able to use deferred values inside types and providers. I know that there have been requests to be able to call functions. And if that is a valid request, then it is logical that using deferred values would also be great to have.