experimental new `App` resource kind

[jstrachan]

So I've noodled the opencompose examples and I love their concise nature. What worries me is leaky abstractions and keeping in sync with all the detail in kubernetes and keeping in sync with the kubernetes resources which keep changing over time. e.g. from a quick look at the issue tracker there's already a fair few gaps: #134, #130, #128, #126, #125, #124, #123, #122, #120, #114, #101, #93, #91, #90, #85, #81, #24

As a thought experiment I had a little play with adding something opencompose-ish to the fabric8-maven-plugin. Here's a link to the issue with some example markup and what it generates along with the design rational.

Basically I tried to adding the smallest possible new resource kind while trying to be nice and concise but letting the real kubernetes structs shine through so that there's no leaky abstraction and minimal learning required for kubernetes savvy folks. It turns out I've a fix for these issues too ;) #134, #130, #128, #126, #125, #124, #123, #122, #120, #114, #101, #93, #91, #90, #85, #81, #24

Basically I:

• created a new resource kind called App its currently Java code but a go lang equivalent would be equally trivial to do as well
• the App is-a PodSpec which solves lots of those issues referenced above for defining pods
• added the design time metadata properties like name, labels, annotations etc
• added the properties from DeploymentSpec other than the template which mostly just is the PodSpec anyways
• added a service property for defining a service for the app (which uses ServiceSpec)
• added an optional ConfigMap data for easy adding of configuration data to an app
• added optional extra PVCs we could do the same with emptyDirs too I guess - its easy to also mount external volumes/secrets/configmaps already with the current App as its a PodSpec already; the persistentVolumes property is more for defining the actual PVCs that are owned by this App
• given the closeness to the existing kubernetes structs its pretty trivial to write a faithful generator that generates the full kubernetes resources for a given App

Example

here's a sample app.yml file using the App resource:

---
name: "cheese"
labels:
  app: "cheese"
annotations:
  com.acme: "foo"
replicas: 1
containers:
- image: "foo/bar:123"
  env:
  - name: "MY_ENV"
    value: "CHEESE"
  volumeMounts:
  - mountPath: "/app/logs"
    name: "log-data"
  - mountPath: "/app/appstuff"
    name: "app-data"
service:
  ports:
  - port: 80
    targetPort: 8080
configData:
  application.properties: "foo.bar = abcd"
persistentVolumes:
  log-data:
    accessModes:
    - "ReadWriteOnce"
    resources:
      limits:
        storage: "1Gi"
  app-data:
    accessModes:
    - "ReadWriteMany"
    resources:
      limits:
        storage: "5Gi"

Generated these resources:

---
apiVersion: "v1"
kind: "List"
items:
- apiVersion: "v1"
  kind: "Service"
  metadata:
    annotations:
      com.acme: "foo"
    finalizers: []
    labels:
      app: "cheese"
    name: "cheese"
    ownerReferences: []
  spec:
    deprecatedPublicIPs: []
    externalIPs: []
    loadBalancerSourceRanges: []
    ports:
    - port: 80
      targetPort: 8080
    selector:
      app: "cheese"
- apiVersion: "v1"
  kind: "PersistentVolumeClaim"
  metadata:
    annotations:
      com.acme: "foo"
    finalizers: []
    labels:
      app: "cheese"
    name: "app-data"
    ownerReferences: []
  spec:
    accessModes:
    - "ReadWriteMany"
    resources:
      limits:
        storage: "5Gi"
      requests: {}
- apiVersion: "v1"
  kind: "PersistentVolumeClaim"
  metadata:
    annotations:
      com.acme: "foo"
    finalizers: []
    labels:
      app: "cheese"
    name: "log-data"
    ownerReferences: []
  spec:
    accessModes:
    - "ReadWriteOnce"
    resources:
      limits:
        storage: "1Gi"
      requests: {}
- apiVersion: "v1"
  kind: "ConfigMap"
  metadata:
    annotations:
      com.acme: "foo"
    finalizers: []
    labels:
      app: "cheese"
    name: "cheese"
    ownerReferences: []
  data:
    application.properties: "foo.bar = abcd"
- apiVersion: "extensions/v1beta1"
  kind: "Deployment"
  metadata:
    annotations:
      com.acme: "foo"
    finalizers: []
    labels:
      app: "cheese"
    name: "cheese"
    ownerReferences: []
  spec:
    paused: false
    replicas: 1
    selector:
      matchExpressions: []
      matchLabels:
        app: "cheese"
    template:
      metadata:
        annotations:
          com.acme: "foo"
        finalizers: []
        labels:
          app: "cheese"
        name: "cheese"
        ownerReferences: []
      spec:
        containers:
        - args: []
          command: []
          env:
          - name: "MY_ENV"
            value: "CHEESE"
          image: "foo/bar:123"
          ports: []
          volumeMounts:
          - mountPath: "/app/logs"
            name: "log-data"
          - mountPath: "/app/appstuff"
            name: "app-data"
        imagePullSecrets: []
        nodeSelector: {}
        volumes:
        - name: "log-data"
          persistentVolumeClaim:
            claimName: "log-data"
        - name: "app-data"
          persistentVolumeClaim:
            claimName: "app-data"

Comparing App to opencompose

Whats interesting is its very similar really to opencompose in feel, conciseness and markup. The main difference is the header part; opencompose looks more like docker compose at the top (the version: and services:).

We already have kcompose for migrating docker compose to kubernetes so I'd rather opencompose focus on whats best for kubernetes; using its conventions.

Though it'd be trivial to define an AppList type resource to be a collection of Apps and be more like opencompose; though the kubernetes default for lists of structs is not to use maps like:

services:
  foo:
     cheese: edam

but to use a list

kind: AppList
items:
- name: foo
  cheese: edam

so if I were re-inventing the 2.0 of opencompose I'd probably ditch this docker compose convention anyways?

Other than that things are fairly similar really; the main difference is the focus on reusing the Spec structs from kubernetes and composing them together in a concise way rather than inventing a brand new schema.

Interestingly on the services: versus AppList front; with the focus on microservices (one app that does one thing well), I think its going to be easiest for developers to have an app YAML file for each app they work on. Typically the default case will be 1 app.yml per microservice (i.e. per git repo) anyway.

A typical docker compose example often shows a front end and a database; which is great for demos. However in the real world usually the database is owned by one team and provisioned and upgraded on a totally different schedule to the front end app anyway; in which case having 2 separate app.yml files for them makes loads more sense really.

Plus developers find it hard to grok large nested yaml files anyway ;) so I do actually prefer the default of keeping apps in different files.

Closing thoughts

Note that this was just a thought experiment really. I was really just thinking at the time of Java folks and reducing their typing when hacking YAML in some validating/completing IDE. I'm still mulling over how this may or may not affect opencompose really.

One interesting further thought is with the new API Groups in kubernetes we can add new schemas to kubernetes along with controllers/operators that work on them. So we could create a new App resource (maybe even upstream it to kubernetes itself - or worst case make an API Group and controller for it) so that folks could use kubectl directly:

$ kubectl get apps
$ kubectl apply -f myapp.yml

etc which then whenever an App is created/updated/deleted it runs the canonical generator to make the resulting Deployment, Service, ConfigMap, PVCs etc and apply those chanages.

i.e. then folks could work at this App abstraction as a simpler way of working with the actual underlying kubernetes abstractions.

If we did go in this direction though, we'd maybe wanna make a minor change to App and move the metadata fields out, rename App to AppSpec and then add a new App class/struct containingMetadata and AppSpec so that it follows the conventions of all the other kubernetes resources. Then we'd add AppList and then the kubectl / REST API would be consistent.

[jstrachan]

Incidentally the java code is all code generated from the underlying kubernetes source code so these Java classes keep in sync with the kubernetes structs; this is obviously much easier to do in golang as we can then just reuse the kubernetes Spec structs directly ;)

So over the time this model should keep pace with the underlying changes in the kubernetes model nice. The only thing we need to be careful of is any conflicts between merging the DeploymentSpec properties and the PodSpec properties, along with the extensions, into the same struct- which hopefully will be fine

[pradeepto]

Note : Data Transfer Object == DTO aka struct in Go language terms.

[jstrachan]

thanks @pradeepto - I stuck to referring to them as struct in the above to avoid more confusion with other opencomposers

[jstrachan]

incidentally another argument against the services: header longer term for opencompose is that services are assumed to be kubernetes Service resources in the kubernetes ecosystem; whereas opencompose is more about making apps / microservices

[kadel]

Hi @jstrachan It looks interesting and I like the idea of "letting the real kubernetes structs shine through".

With this approach, especially with having 1 app.yaml file for every app (separate file for db and application), how would you handle things that are shared between applications? For example, Secrets that are defined once, but they can be used by multiple application.

[jstrachan]

So the first thing is Secrets should never be included in your applications distributed YAML that gets moved between dev / test / staging / production; they should always be curated on a per environment basis separately from the app. e.g. in production Secrets will be generated/managed/rotated by operations folks rather than the developer who wrote the microservice.

So today in the app.yml's container: you can expose secrets as environment variables or mount entries as files using the volume mounts in the usual kubernetes way - you just reference the secret by name (without including or generating the Secret resource).

      containers:
      - image: foo
        env:
        - name: MY_PASSWORD
          valueFrom:
            secretKeyRef:
              name: somesecret
              key: user.password

Referring to Volumes/Secrets/ConfigMaps by name describes a static reference that operational tools in each environment can check they exist before you try to install an app - but allow things to be configured separate in each environment.

One thing thats currently missing in kubernetes is a SecretClaim resource, rather like a PVC claims a PV it would be nice to include a SecretClaim to describe the name and metadata you need (e.g. what kind of key/cert) so that tools can auto-generate them or import them from known locations like Vault etc. We've used annotations in the past for this https://github.com/fabric8io/fabric8/blob/master/docs/secretAnnotations.md - one day it'd be nice to separate that out into a separate API Group / ThirdPartyResource along with a controller/operator that could automatically generate random secrets for things (or import secrets from known locations on a per environment basis).

Though I think the common case for PVCs and ConfigMaps is for them to be included in the app really as the App often owns them (its the PVs that are never included and handled by Operations). Though sometimes you may wish to reference a shared ConfigMap by name and not have it included in your app (so that it can be managed by other people, another team or just a different release process) - or have shared ConfigMaps owned by operators and not distributed with your app.

e.g. shared ConfigMaps could be managed via one git repo with its own release cycle; then each app having its own git repo (and maybe using branches or separate repos for each environment like testing/production) and CI / CD pipeline to roll out changes.

For example with openshift.io right now each app just releases its Service + Deployment (or DeploymentConfig); then operations take care of maintaining all the secrets; then our CI / CD pipelines roll out new apps (new images + YAML files of Services/Deployments/DeploymentConfigs/ConfigMaps/PVCs) to each environment.

Interestingly we'd like Routes to not be included in a YAML either; as its common to want those to be enriched with an environment specific certificate for HTTPS support; so thats something better done by a microservice/controller/operator than added to your YAML. e.g. see exposecontroller that can generate Routes / Ingress / LoadBalancer / NodePort for your Service

[jstrachan]

Longer term I'm expecting Databases and Message Queues and whatnot to not be specified as an app.yml but specified using the Service Catalog so that your app.yml just references known Service resources by name and may include some Service Catalog resources such as an Instance to help the brokers provision the Services you need (which typically create a Service and Secret for your app to use).

i.e. referring to Services and Secrets is gonna be increasingly common. The same would be true if Team A is using a service provided by Team B and vice versa; each team would have their own YAML and refer to each others services by name.

We should try ensure that the YAML focusses on capturing a single teams responsibilities; if 2 teams need to release YAML at different schedules and times then its better to separate those resources into separate YAML files in different git repos

[surajssd]

I like the consice way of defining the application using k8s structs, where we are not reinventing things.

Also the point you made about the user caring only about their application and not about anything else is a valid one.

But I think the tool might not be able to give good user experience if we assume things to be already present, like secrets. So how does a user who want to define secret use opencompose to do it?

Also since we are assuming things are present already, user will find out those things only while deploying. So we are loosely coupling the dependencies.

If you see the configs that are generated right now, specially with docker-compose or even opencompose it acts as single point of truth, and once converted it can be deployed on any cluster. No pre-reqs are assumed than a normal running cluster.

But assuming things to be present in the cluster and just referring from opencompose adds new set of issues. Since opencompose is no more single point of truth but something that has invisible dependencies. Maybe the tool could become intelligent and query cluster while deploying things that are being referred to be present on the cluster before hand.

I agree on the point where you say that in production grade deployments teams will be working on different pieces and then they will write their own configurations independently and when we add those together it will be deployed properly.

[concaf]

@jstrachan -

Hey James, thanks for bringing this up, this has opened room for a much broader view and discussions.

Thoughts -

• added the properties from DeploymentSpec other than the template which mostly just is the PodSpec anyways
• added a service property for defining a service for the app (which uses ServiceSpec)
• added an optional ConfigMap data for easy adding of configuration data to an app
• added optional extra PVCs we could do the same with emptyDirs too I guess - its easy to also mount external volumes/secrets/configmaps already with the current App as its a PodSpec already; the persistentVolumes property is more for defining the actual PVCs that are owned by this App

If we are modifying the pod spec to add other primitives, then it's not a standard Kubernetes spec anymore, right? This would mean that we have already modified the pod spec to include other primitives.

So, the way we have been writing the OpenCompose spec has been with keeping the developer in mind and the most common directives they would use, and hence adding those to the spec. Till now, it has turned out to look like a lot like a pod spec :) , but we are not limited by the pod spec and the way Kubernetes defines things.

One example would be how we are moving forward with secrets.

This is how a snippet from the OpenCompose file would look like -

services:
- name: foobar
  containers:
  - image: foo/bar:tag
    env:
    - name: foo
      value: bar
    - name: travis_creds
      secretRef: ci_secret/travis

secrets:
 - name: ci_secret
   data:
   - key: travis
     file: /tmp/travis.password

Here, we are referring to the root level secrets using the secret_name/data_name syntax. This makes the spec a lot more concise, more intuitive, and yes, different from the pod spec.
Also, at the root level secrets, we also allow taking in input from a file, which is not a part of the Kubernetes Secret object definition.

Now, this might not be the best example or syntax to move forward with, but if we move forward with extending the Kubernetes spec as is, then we lose out on the angle of "making defining microservices simpler", and we move to "making Kubernetes configurations simpler to write".
We have been doing things the former way till now, but both have their downsides and tradeoffs which we should consider.

I think its going to be easiest for developers to have an app YAML file for each app they work on. Typically the default case will be 1 app.yml per microservice (i.e. per git repo) anyway.

We could push this as a "best practice", but imo the developers should have the ability to choose if multiple microservices are defined in one file or multiple. A way to merge those would be awesome, but I'm not sure about enforcing this.

incidentally another argument against the services: header longer term for opencompose is that services are assumed to be kubernetes Service resources in the kubernetes ecosystem; whereas opencompose is more about making apps / microservices

Yep, the naming can be confusing here, we can change that. But I think this root level directive serves the purpose of having multiple microservices defined in one file, and even is decoupled teams, it's very likely that a single team owns multiple services which are coupled together - and it would be great to define them in one file.

So today in the app.yml's container: you can expose secrets as environment variables or mount entries as files using the volume mounts in the usual kubernetes way - you just reference the secret by name (without including or generating the Secret resource).

Yep, but at the same time, a developer should be able to quickly define a secret in (a separate?) OpenCompose file, so in their developer environment, they don't have to interact with the cluster. Their OpenCompose file will be all they need to deploy their application, and if they do not want to define a secret in the OpenCompose file, the tooling should log that "the cluster must contain this resource, since you didn't define it in OpenCompose"

We should try ensure that the YAML focusses on capturing a single teams responsibilities; if 2 teams need to release YAML at different schedules and times then its better to separate those resources into separate YAML files in different git repos

I really like the idea of multiple OpenCompose files coming together and merging into one application, but at the same time it should be left to the teams to decide how they want to do it, we should be providing a "multiple files merging" and "one single file" for the users to choose from.

---

I really like the idea of "app", and merging files, and borrowing more from the Kubernetes spec, it'd be great if both the approaches converge to something more awesome :)

Would love to hear your thoughts about the above mentioned points.

[jstrachan]

@surajssd I think opencompose should focus on helping application developers make apps in the correct way for kubernetes, as immutable blobs of YAML that are then used to deploy apps into any environment/namespace/cluster of kubernetes. That doesn't mean it has to solve all problems. e.g. opencompose should not create PVs - thats something for Ops folks to setup on a per cluster basis. Similarly secrets are something that are typically provisioned & rotated out of band of application YAML. A typical app may be deployed to multiple environments and clusters; each with possibly different secrets but using the exact same application YAML.

So while we could include Secrets inside the app.yml its much more helpful to nudge application developers down the happy path of just providing the stuff that application developers should provide - and omit the stuff that the cluster Operations team handle (PVs and Secrets).

However upstream in kubernetes I'd love a SecretClaim resource which we could include inside the app.yml like we do for PVCs; thats kinda like the PersistentVolumeClaim resource - to indicate to an ops person/program/cluster that there is a need for a Secret (like a PV) but not actually including the Secret - just providing the metadata to describe what kind of secret is required.

There's no real way around this loose coupling really. Its exactly the same with PVCs; you can deploy your app and if a cluster has not got suitable PVs setup (or automatic PV provisioning) then the PVCs won't bind. Ditto if you depend on a separate Service implementation which is managed by a separate microservice team. This loose coupling is going to be more and more common with the service catalog work.

However there's nothing to stop opencompose writing a tool to check if an app.yml has all the associated resources (PVs / Secrets) to be able to be created.

If you want opencompose to include everything - that also implies including PVs right? It also means you have to reject ever working with the service catalog? Also that implies checking your Secret into source control that sounds like a bad idea.

There's nothing stopping folks adding a separate secrets.yml file with some default secrets; though we should really try discourage that approach.

Incidentally for simple secrets (login/passwords) its common to use an OpenShift Template with parameter values for the user/pwd values which can be auto-generated at template instantiation time. That only works for simple user/pwd fields though; it doesn't much help for certs or SSH public/private keys or whatnot.

But really secret management is something we should tackle upstream in the kubernetes ecosystem via something like a SecretClaim

[jstrachan]

@containscafeine I'm not suggesting we modify any structs / types from kubernetes - I'm suggesting we extend those structs to add more fields. Thats easy to do in golang via struct extension or java (using base classes) - which is what I did in the Java implementation via:

public class App extends PodSpec {

Just like you've added your own properties which are not in PodSpec I've done the same in App - the difference is I've extended PodSpec and reused ServiceSpec, DeploymentStrategy, PersistentVolumeClaimSpec and reused the same properties as DeploymentSpec so that its all very consistent with kubernetes - without changing any kubernetes structs at all

I think its going to be easiest for developers to have an app YAML file for each app they work on. Typically the default case will be 1 app.yml per microservice (i.e. per git repo) anyway.

We could push this as a "best practice", but imo the developers should have the ability to choose if multiple microservices are defined in one file or multiple. A way to merge those would be awesome, but I'm not sure about enforcing this.

I'm not against allowing multiple apps to be defined in the same file - though I think it makes the developer UX much worse - but its also trivial to generate the resulting generated YAML as a single YAML to simplify installing it via kubectl / oc. Though if there was a YAML for multiple apps I'd prefer it used the kubernetes conventions rather than docker compose conventions (lists of objects).

In response to the rest of your comments on Secrets (I've not quoted it all here as its hard in github comments ;-) I'm not against having some clearly defined default secrets YAML file thats only used for developers trying things out but is never used for a real pre-production or production deployment. However that feels to me more like something we'd keep in a separate file so that you can omit the default secrets when deploying your app to pre-prod/prod.

If we had a SecretClaim resource which could supply dummy default values but those values would be omitted from real deployments - that would maybe solve all of our issues; we'd be able to put SecretClaim resources OOTB in all app.yml files like we do with PVCs; then they'd have some dummy insecure defaults that get everyone started in development (e.g. using admin/admin login/pwds and stuff) but then we'd have tools that could lookup or generate new secrets in other environments like we've done in the past in fabric8 - then its a win-win - we know what secrets need to be populated in real environments but in developer experimental environments we can get going fast. (To be clear we used to use the presence of these secret annotations to generate or load secrets from disk as applications get deployed - to avoid hard coding secrets inside application YAML files. We could do something similar if we added a SecretClaim resource (that it'd be nice to upstream into kubernetes really).

e.g. I can imagine a controller in kubernetes using the presence of SecretClaim resources to then provision Secret resources by looking things up in tools like Vault

[pradeepto]

Similarly secrets are something that are typically provisioned & rotated out of band of application YAML. A typical app may be deployed to multiple environments and clusters; each with possibly different secrets but using the exact same application YAML.

This topic is being discussed here: Issue - #47 and Proposal - #149