Documentation
Laiki operator
Documentation
Laiki operator

Laiki operator

Introduction

The Laiki operator is a core component of the Laiki platform. It automates the workflows that give data scientists self-service capabilities: the ability to manage their own apps and configurations without depending on a sysadmin for every change.

To explain what this means in practice, let's start with ShinyProxy, one of the components of the Laiki platform.

The first level of self-service: containers

ShinyProxy is a tool for deploying data science apps in an enterprise context. It relies on containers (most commonly Docker) to run these apps.

A container packages an app together with everything it needs. That includes libraries, dependencies, and configuration, all bundled into a single, self-contained image.

This isn't an incidental design choice. Containers have been a core principle of ShinyProxy since day one, because they solve a major problem: helping data scientists get their applications in front of end-users without friction.

How it works:

  • Every app has its own container image, and each image includes only the tools that app needs. For example, a Shiny app's image contains R, the required system libraries, R packages, and the R code itself.
  • These tools are only available to that specific app, and only that app can see them.
  • When a new app is deployed, a new container image is created for it. That image can use a completely different set of tools (a different R version, different packages, and so on) without conflicting with any other app, because every image and every running app is isolated.
  • As a result, the server itself only needs Docker installed. There's no need to install R, Python, or any other data science tool directly on the server.

Why this matters:

This is a sharp contrast to traditional setups, where every tool is installed on a single shared server. In that model, all data scientists are limited to a single set of tools. If someone needs a new R version, they have to file a request and wait for the sysadmin to install it.

Containers remove that bottleneck by creating a clean separation of responsibilities: the sysadmin maintains the server and the platform, while the data scientist chooses and controls the tools used inside their own apps.

The data scientist gets full control over their app's tooling, with no need to involve the sysadmin. The sysadmin, in turn, can focus on building and maintaining the platform instead of fielding a constant stream of install requests.

We call this principle self-service data science. With ShinyProxy, data scientists already get the first level of it: the freedom to build apps with any tools they choose.

The second level of self-service: Laiki

ShinyProxy manages all of its apps through a single YAML configuration file. Adding an app means adding an entry to that file.

For a small team, this works well. A common best practice is to store the configuration file in a Git repository, so changes are tracked and deployed automatically. But as a team grows to hundreds of apps and users, a single shared configuration file becomes harder to manage.

One option is to restrict who can edit the file. But that comes at a cost: it takes control away from the data scientists who own the apps.

This problem is solved by the Laiki operator.

Laiki is an intelligent tool that understands the ShinyProxy configuration format and hands control back to the data scientist, all without sacrificing manageability at scale. Instead of one large configuration file, Laiki reads from multiple configuration files across multiple Git repositories, typically one file per app.

This unlocks many benefits:

  • Access control: data scientists can see and manage the configuration for their own apps only. They don't have access to ShinyProxy's shared, platform-wide settings.
  • Governance: through built-in workflows, Laiki can enforce rules on these configurations. For example, it can enforce certain settings or limit others, so platform owners retain guardrails even as control is distributed.
  • Automation: Laiki can integrate with tools outside the Laiki platform. For example, it can automatically create a dedicated container image repository in an external registry for each app it deploys.
  • Abstraction: by hiding platform-level details, Laiki keeps app configuration high-level and simple. This makes it easier for data scientists to understand and easier for sysadmins to maintain.

Laiki provides the second level of self-service: data scientists can configure their own apps directly, within the boundaries set by the platform owner.

Technical deep-dive

As mentioned in the introduction, Laiki starts by reading input files. Let's build on the ShinyProxy example, starting with the base configuration of ShinyProxy:

laiki-metadata:
  target-type: ShinyProxy
  name: ShinyProxy
  namespace: ShinyProxy
laiki-spec:
  proxy:
    store-mode: Redis
    stop-proxies-on-shutdown: false
    title: ShinyProxy
    containerBackend: kubernetes
    authentication: none
  # ...

Every Laiki file contains metadata that tells Laiki how to handle the file. In this case, the metadata includes target-type: ShinyProxy, which lets Laiki recognize this as an ShinyProxy configuration file. The rest of the file is a standard ShinyProxy configuration: Laiki doesn't add anything special here.

This base configuration file should not contain any applications. Instead, each application gets its own file:

laiki-metadata:
  resource-type: ShinyProxyApp
  target-name: ShinyProxy
  target-namespace: ShinyProxy
laiki-spec:
  id: shinyproxy-demo
  container-image: openanalytics/ShinyProxy-demo

Here, the metadata includes resource-type: ShinyProxyApp, so Laiki knows this file configures a single ShinyProxy app. The target-name and target-namespace fields bind this app to a specific ShinyProxy instance, which makes it possible to manage multiple ShinyProxy instances within a single Laiki setup. As before, the rest of the file is standard ShinyProxy app configuration. In this case it's minimal: just an ID and the location of the container image.

With these two simple files, Laiki is already able to run. A common approach is to run Laiki as part of a CI/CD pipeline: Laiki reads all configuration files and deploys the resulting ShinyProxy configuration. With the setup above, this is straightforward, as the app configuration is simply merged into the base configuration file.

The first workflow

This section walks through one of Laiki's built-in workflows. To follow along, it helps to have some background first.

ShinyProxy has strong support for Kubernetes, a technology for running containers across a cluster of servers. When Kubernetes runs in the cloud, the cluster can typically scale up automatically: as more users join the platform, more servers are started. When an app is launched on a freshly started server, that server first needs to pull the app's Docker image. In the R world, these images can be large, which increases pull time and slows down app startup.

ShinyProxy can avoid this by pre-initializing containers, but that isn't possible for every workload. In those cases, an alternative is to pre-pull the image on new nodes: as soon as a server starts up, it begins pulling the required images in the background. By the time an app is scheduled on that server, the image is already present, so there's no pull step left and the app starts faster.

This might sound involved, but Laiki makes it easy. Just create a YAML file with the following contents:

laiki-metadata:
  target-type: PrePuller
  name: pre-puller
  namespace: ShinyProxy
laiki-spec:

This file tells Laiki to create a target of type PrePuller. Internally, Laiki runs a workflow called ShinyProxyAppToPrePullerWorkflow, which, as the name suggests, looks at all ShinyProxyApp resources and generates the pre-puller configuration needed for them.

Managing external resources

To get the full benefit of working with containers, images should be stored in a central registry. Since every app has its own image, every app needs a repository in that registry. Platform owners often want to place some restrictions on how these repositories are created.

For example, in regulated industries (GxP), reproducibility is critical. Many registries support a feature called tag immutability to help with this. Once a Docker image has been pushed under a given tag (its name), that tag can never be overwritten. This guarantees that running image X with tag Y always produces the exact same image.

If users are allowed to create repositories directly through the registry's UI, some will inevitably forget to enable tag immutability. And in regulated industries, it isn't enough to enable a setting correctly: you also need to be able to demonstrate that it's consistently enforced. Rather than solving this with paperwork or by taking repository-creation rights away from users, the better approach is automation. Data scientists keep the ability to create Docker repositories, but automation removes the room for human error.

Laiki ships with built-in workflows for managing repositories in AWS ECR, and it can easily be extended with new workflows to manage repositories in any other registry.

For end users, creating a repository is as simple as creating a YAML file:

laiki-metadata:
  resource-type: AwsEcrRepository
  target-name: ShinyProxy
  target-namespace: ShinyProxy
laiki-spec:
  name: my-repository-name

Data volumes for apps

Many data science apps need to fetch data from external sources, whether that's a database, an object store, a file server, or something else. The Laiki platform includes a tool called Crane for managing data science artifacts, documentation, and more. In Crane, everything, including access to data repositories, is managed declaratively.

A common use case is creating a dedicated Crane data repository for a Shiny app. The data in that repository can be stored on a file server (such as SMB, NFS, or a cloud solution like AWS EFS), or in blob storage such as S3. Laiki is fully compatible with Crane, so a data volume can be added directly to an app's configuration. For example, to add a data volume to a Shiny app:

laiki-metadata:
  resource-type: ShinyProxyApp
  target-name: ShinyProxy
  target-namespace: ShinyProxy
laiki-spec:
  id: shinyproxy-demo
  container-image: openanalytics/ShinyProxy-demo
  crane-data-repository:
    mount-read-only: true
    write-access:
      users:
        - jack
        - jeff
    read-access:
      users:
        - jack
        - jeff

This configuration creates both a ShinyProxy app and a Crane data repository. Inside the Docker container of the app, the volume is mounted at /mnt/data. The users jack and jeff have both read and write access to the volume and can manage its contents through the Crane web UI. The volume is not visible to any other user.

Setting defaults using a pre-processor

Every app in ShinyProxy can specify how much memory and CPU it needs. This matters when running hundreds of ShinyProxy apps on a shared Kubernetes cluster. In practice, most apps only need a minimal amount of resources, so it would be inefficient to require every app developer to specify these values explicitly when a sensible default would work most of the time.

This can be handled by adding a new YAML file:

laiki-metadata:
  pre-processor-name: ShinyProxyAppDefaultsPreProcessor
  name: ShinyProxyAppDefaults
  target-namespace: ShinyProxy
laiki-spec:
  fallback:
    container-cpu-request: 0.25
    container-cpu-limit: 1
    container-memory-request: 256Mi
    container-memory-limit: 512Mi
  append:
    container-env:
      PLATFORM_VERSION: v1.2
  override:
    container-privileged: false

Unlike the earlier examples, this file doesn't configure a workflow. It configures a pre-processor. A pre-processor runs against a resource (in this case the ShinyProxyApp resources) and modifies it before any workflow gets to handle it.

The ShinyProxyAppDefaultsPreProcessor configuration above performs three tasks:

  • fallback: if an ShinyProxyApp doesn't specify a given resource setting, the corresponding fallback value is applied. Each line under fallback is checked and applied independently.
  • append: the PLATFORM_VERSION environment variable is added to every application. Apps can still define their own additional environment variables on top of it.
  • override: apps are prevented from running in privileged mode, even if an app explicitly sets container-privileged to true.