Runtime Pipeline Patterns

Introduction

Pipeline patterns are most commonly used for simple data workflows like general functional composition or user registration processes. After briefly describing the basics, this article focuses on a couple of behavioral software design patterns under the strategy pattern we'll refer to as "runtime patterns" for use with pipelines. Their power in modern applications is allowing pipelines to expand or contract dynamically. Whether it be through behavior modeling with ai, blueprint fulfilling or more basic conditional pipeline adaptation, the runtime patterns are helpful and with forethought can be simply understood and implemented.

Pipeline Basics

In its most primitive form the pipeline design pattern is just a series of functions that sequentially pass and likely operate on data. Though the functions connect much like a water pipe would (hence the pipeline name,) it's better to think of them as an assembly line as water pipes don't necessarily try to alter the water, just route it; albeit a part or whole pipeline can merely be used for monitoring data or dispatching events and not specifically mutating data.

Pipeline Composition

Pipelines are composed of stages and fixtures. Stages are the building blocks of the pipeline and are atomic, usually intended to take data in and return data. A pipelines stages may either mutate data, immutably instantiate and return new data or return the data unchanged. Stages can also cause operational occlusion and neuter a pipeline when data isn't appropriately returned. Fixtures are stages that allow ways of forking or joining stages. Care must be made when using fixtures as a pipeline should be a directed acyclic graph or DAG, and with improper directed fixtures an infinite recursion could take down a pipeline.

Pipeline Registrar

The pipeline registrar allows a stage to be inserted between two stages or at the beginning or end of a pipeline and is the most basic runtime pattern above the pipeline and is very similar to the service locator pattern with respect to pipelines. Here are a few simple possibilities for implementation and some observations for each:

1. The stages themselves can handle registration processes on before or after hooks.
   • Either the stages need to be found by a mapped name, index or some pointer or reference. This drastically increases application complexity based on the size of the pipeline.
   • The before and after hooks themselves could take either a singular function, stage class or an entire pipeline. This increases robustness.
   • Because the stages could contain other stages on before and after hooks, a pipeline could in fact be a tree of stages. This drastically increases the application complexity (especially debugging) by the factor of the trees depth.
2. The stages themselves take registration on before or after hooks through a parent pipeline class delegate.
   • Because the pipeline manages its own stages, the logical structure of a singular array to hold stages could make debugging and pipeline complexity minimal. Good representation of Single Responsibility Principle.
   • The before and after hooks themselves could take either a singular function, stage class or an entire pipeline. This increases robustness.
3. A third party registrar service takes new stage registration on behalf of a pipeline and either reconstructs the pipeline or alters the pipeline to reflect the registration.
   • We've performed dependency inversion on stages, but incidentally on the pipeline itself. This would be good if the pipelines are not classes but just functions.
   • This would also be good if there is more than one pipeline that needs to be managed. Much like a train station, it could help direct passengers to the correct train.
   • If the previous two points aren't present then there is likely no value to compensate for the added complexity.

Pipeline Orchestrator

The pipeline orchestrator pattern allows pipelines to be created, adjusted or managed according to blueprints and is similar to the template method pattern. The foundation of the orchestrator is a pipeline registrar. An example purpose of the pipeline orchestrator might be an application that creates vehicles. Each stage might be the components of the car, the blueprint would be the order in which certain components are to each other and would also establish conditional operations with fixtures. While one blueprint might be a truck and another one could be a sedan, they would both use shared methods and logic stages. The orchestrator patterns true power is in allowing blueprint counts to scale quickly while doing very little to no development on the actual pipeline itself.