With the rise of cloud, cloud-native and a lot of other buzzwords, we also see the rise of all kinds of Dev(fill here)Ops practices. While I’m an avid advocate of the DevOps mindset, all the buzz around pipelines sounds old-school to me.

What is GitOps?

GitOps evolved from DevOps. The specific state of deployment configuration is version-controlled (using Git for example). Changes to configuration can be managed using code review practices, and can be rolled back using version-controlling.

https://en.wikipedia.org/wiki/DevOps#GitOps

A quick look at Wikipedia brings you the above description. It is stuffed away somewhere in the DevOps article. Like with DevOps, there is a whole religion around what tool is the best to use, and people even claim to be ‘DevOps Engineers’ whatever that may be. In most cases they mean ‘cloud infrastructure’ because DevOps is a mindset, and not a function.

It doesn’t help that some tools like Azure DevOps are actually called this way, but hey, VSTS is also not really a snappy name IMHO.

To explain GitOps in my own words and in a single sentence: Instead of pushing your code and deploy, your system pulls the software from the version control system.

A nice one-liner by WeaveWorks is: GitOps is Continuous Delivery meets Cloud Native

https://www.weave.works/technologies/gitops/

GitOps versus DevOps Pipelines

So, it is the same as DevOps Pipelines right? No it is not. From a distance it can seem to be the same. In a way your code, if configured correctly, will be deployed automatically with CICD.

The fundamental difference is that your system (for example Kubernetes) and not your DevOps CICD pipeline, is in the lead. Remember, the settings of the k8s system itself also need a repository.

Let’s look at a typical CICD setup:

As you can see in the example above, the Azure DevOps pipelines are in the lead. The Build step pulls in the code, the Release step creates the artifact and pushes it towards a registry, and also installs it on AKS. But this does not include the state of your cluster.

Let’s say your DevOps pipeline trigger failed, or someone made changes manually to the AKS applications or configuration. AKS has no idea where the code is located so it cannot check if there are any changes. This is where GitOps can be a solution. Let’s look at the following example:

In the example above you can see that (in this case Flux) is used to pull the desired state towards your Kubernetes cluster. This pulling makes sure that you are always up-to-date. And it can also detect changes so if anything or anyone changes the configuration manually, this ‘drift’ will be resolved by the desired state pulling with Flux!

Yes, this complicates your setup, but you always know that your clusters are exactly how you described them in the desired state. Especially in situations with Kubernetes, deployment and configuration can be really complex, so these tools make sure you are in control of your systems.

Example of GitOps tools

So I need Kubernetes? No. In theory any system can use the GitOps setup, but for Kubernetes a lot of tools and so called operators are readily available.

In the previous paragraph, Flux is drawn in the picture. There are many more tools out there, but we see convergence on two major tools: Flux andArgo. The resources section at the end of the blog shows more tools and resources if you want to dive in the deep end of GitOps.

There is a nice writeup (sponsored by Red Hat) on deciding between Flux and Argo. Depending on your situation it can fall either way. GitOps on Kubernetes: Deciding Between Argo CD and Flux – The New Stack.

OpenGitOps

The emergence of GitOps led to the OpenGitOps project.

OpenGitOps is a set of open-source standards, best practices, and community-focused education to help organizations adopt a structured, standardized approach to implementing GitOps.

https://opengitops.dev/

The principles of GitOps are four-fold:

1. Declarative
   A system managed by GitOps must have its desired state expressed declaratively.
2. Versioned and Immutable
   Desired state is stored in a way that enforces immutability, versioning and retains a complete version history.
3. Pulled Automatically
   Software agents automatically pull the desired state declarations from the source.
4. Continuously Reconciled
   Software agents continuously observe actual system state and attempt to apply the desired state.

“The GitOps Working Group is a WG under the CNCF App Delivery SIG.

The focus of the GitOps WG is to clearly define a vendor-neutral, principle-led meaning of GitOps. This will establish a foundation for interoperability between tools, conformance, and certification. Lasting programs, documents, and code are planned to live within the OpenGitOps project.”

The main goal of the OpenGitOps CNCF Sandbox project is to define a vendor-neutral, principle-led meaning of GitOps. This will establish a foundation for interoperability between tools, conformance, and certification through lasting programs, documents, and code.

Concluding

GitOps has been around for several years, and continues to evolve. CNCF has OpenGitOps as Sandbox project and embraced Flux and Argo as Incubating projects. The shift in mindset can be difficult, as some people still try to embrace DevOps as a principle.

Watch this space for more information and an upcoming eBook on our take and practices surrounding GitOps!

Resources and further reading

It is really easy to blow all your budget when you do not know the impact of the choices you make in the cloud. And while there are many tools available to run analysis on your bill, or showing where your money went, would it not be smarter to shift everything left?

34 per cent of large enterprises set a budget of at least $10m a year while 62 per cent of small business went for under $100k. Of those spanking between $2m and $10m on the cloud, almost half overspent (compared to 27 per cent of those budgeting until $100k

https://www.hashicorp.com/blog/cloud-budgets-busted-almost-40-overspent-last-year

In a race for speed of delivery, putting restrictions on cloud environments can be contra-productive. On the other hand money can fly out the window if you do not keep everything in check. So what is the solution?

Training and Awareness

First of all it should be obvious that anyone involved in cloud project should also be trained appropriately. Part of our work as consultants and architects is to also include the running and operational costs of a cloud solution. But to sometimes the costs can sneak into a project, especially when using systems like Kubernetes or any other cluster based system. These can have high impact on your costs because the VMs underneath those systems can be costly to run.

Automate, automate and automate!

In comes Infracost. This really great open source tool has many integrations with CICD systems and can be simply added to your Azure DevOps, Gitlab or GitHub flows (and many more).

Infracost calculates cloud costs based on Terraform. Cost estimates can be shown in the terminal or put in pull requests using our CI/CD integrations. This helps you understand the cost of services before you use them, and take action to optimize costs within your existing workflow.

https://www.infracost.io

The tool can add a simple cloud estimation to your pipelines, and this will make your DevOps teams instantly see the changes their code have on the bill and your project:

How to get started

The documentation of the tool is great, and many examples for the most used CICD tools are documented at https://www.infracost.io/docs/integrations/cicd. Did I mention they have a free Community version available? This means you can start, right now, with using and testing this tool!

Concluding

In the end it is not about tooling, it all starts with training and awareness. But it is better to check than to get caught by a surprise when the bill comes in. This tool by Infracost is a really great way to start tracking your costs and estimates at the earliest stage in development as possible, give it a spin!

In this blogpost I will show a basic primer of using KICS together with Azure DevOps, to scan Terraform Infrastructure-as-Code.

Kics an open source solution for static code analysis of Infrastructure as Code.

https://kics.io/

What is KICS, and why should I care?

KICS finds security vulnerabilities, compliance issues, and infrastructure misconfigurations in the following Infrastructure as Code solutions: Terraform, Kubernetes, Docker, AWS CloudFormation, Ansible, Helm, and OpenAPI 3.0 specifications

The open source tool is brought to you by Checkmarx but is a true open source community project.

I’ve blogged before about the importance of testing your IaC, and this tool makes it pretty easy to incorporate exactly this into your pipelines.

At this moment in time I’m experimenting with the usage and because some of the examples do not really work out-of-the-box, I’ve decided to put them in this blog.

Running KICS locally

You can run KICS locally, and the documentation of KICS shows a very easy example running it with a Docker container. This is a easy way of testing and playing around with the tool:

docker pull checkmarx/kics:latest
docker run -v {​​​​path_to_host_folder_to_scan}​​​​:/path /
    checkmarx/kics:latest scan -p "/path" -o "/path/results.json"

Setting up KICS in Azure DevOps

A more likely scenario is that you want to run KICS as part of your CICD strategy, automated and triggered by, for example, a pull request. But how to start?

First of all, you should have a repository in DevOps. In my example, I’ve created two folders with some Terraform code, and one Yaml pipeline:

In these ‘terraform-example1’ and ‘terraform-example2’ I’ve added some representative examples for Azure. Resources ranging from storage accounts, Windows Virtual Desktop resources to IAM and Virtual Machines.

As I have two example folders setup, I’ve created two pipelines. The Yaml file is the same, but using variables I can let KICS scan different folders. Basically it looks like this:

Opening up KicsExample1 and viewing the variables shows the following:

Important to note here, is that I’ve added the complete path to the working directory. My directory is ‘terraform-example1’ but as you can find in the documentation, the path on the build agent will be /home/vsts/work/1/s/terraform-example1/. For the second pipeline example, I obviously set/home/vsts/work/1/s/terraform-example2/ as variable.

Running the KICS Azure DevOps pipeline

So, for the actual magic to happen, let us run the pipeline and look at the outcome:

As we can see; due to some trickery with VSO logs, I’m able to put the feedback from KICS into the actual summary! In this case, I’ve taken the example from KICS and wrote the output errors to Azure DevOps. You can look at the example code how I did this.

Analyzing and debuging the results

In the previous paragraph we ran the KICS pipeline, and it came back with some errors. Now, we want to see what the actual problem is. For this we can drill down to the run:

One of the issues found in my first example, is the following error:

Storage Account Not Using Latest TLS Encryption Version, Severity: MEDIUM, Results: 1 
Description: Ensure Storage Account is using the latest version of TLS encryption 
Platform: Terraform

This issue is quite self-explanatory. KICS runs many queries, and you are able to tweak and adjust them.

The second issue found is on the keyvault in my example:

Vault Auditing Disabled, Severity: HIGH, Results: 1 
Description: Ensure that logging for Azure KeyVault is 'Enabled' 
Platform: Terraform

This also is based on a best practice, and it warns you about the disabled auditing. The great thing about this tool is that is also gives descriptions on how to fix your errors!

Concluding

There are many ways on securing your infrastructure as code and testing statically. In my previous blog about Open Policy Agent I also wrote about this. The great thing about KICS is that it is really easy to setup, and for my example in Terraform they have many well-documented queries and how to resolve the issues found.

Implementing KICS as your first line of defence is really easy and gives direct feedback in your DevOps process.

Give me the code!

The example pipeline has been put in to this public gist. Do not forget to pass in the variable PWD with the location of your project!

Any questions or remarks? Feel free to reach out via any of my public profiles or the contact page on this site.

If I would get paid a euro for each time someone says ‘Let’s spin up a VM for that!’ or ‘We have this VM running anyway’, I would have been a rich man.

Although I do advocate reuse, stuffing many services on a machine ‘because it runs there anyway’ will cause a huge problem if there is a shift in architecture or platform. In the end all services need to be carved out and that can be a real pain in the behind. Also it does not really scale, for obvious reasons.

Yes let’s put everything in a pod/container!

A big cluster of VMs is still just infrastructure chewing away your money in the cloud. Do not forget that in many cases, a k8s cluster, Service Fabric cluster or whatever-cluster-cloud-native-you-like in the end is mostly a bunch of VMs running 24/7 for a lot of money.

We underestimate the power of simplicity

Let’s take an example. If you want to run a simple process, for example ingest some data and put it in a store, you could go full ETL/ELT. Alternatively you can leverage simple things, like Azure Functions or LogicApps. With connectors to almost all services, it is really easy and simple to ingest data using these services.

And it’s cheap too!

Azure Functions (ran in Azure) run really cheap, as the first million(!) executions are free anyway.

What about portability?

If you start out with Azure Functions on the Azure Platform, and later on need to run on a cluster like Kubernetes, you’re in luck. Azure Functions run basically anywhere so hosting them in a container or pod, should be a piece of cake. Development is also easy exactly because of the portability.

How to start?

Microsoft has done really great work in the training area. For Azure Functions, you can simply go to: Create serverless logic with Azure Functions – Learn | Microsoft Docs. This module will get you started easily.

Want to kickstart your development? Reach out and we’ll help!

I’ve written in a previous blog that for writing maintainable code, you really should automate your tests. Currently we see Terraform really taking off as the standard to provision and deploy almost everything in cloud like Azure and AWS. In this short blogpost I will show how you can start testing your Terraform code using the Open Policy Agent.

Infrastructure-as-Code

With Terraform, engineers write declarative statements that, when applied, result in for example a network, of a database:

This obviously is great because you can automate and repeat your steps minimizing the manual labour of setting up services. Next to that, you can reuse components so all provisioned services will adhere to the same standard.

What about standards and compliancy?

In the previous example, a network will be created with some IP range, in a specific location from Google.

If you use any cloud, you should have some policies in place. It is not a free-for-all, do whatever you want, scenario. It could be very important where your data resides. In that case you don’t want to enable teams to deploy in non-compliant regions.

With code however, especially in the sample shown above, it is not directly clear what is allowed. Some examples of questions DevOps team may have are:

• Am I allowed to create a public IP address on a machine or database?
• Am I allowed to deploy services in the US or only in Europe?

Worst case, the team finds out when they deploy their services onto their cloud environment. In case of Azure, policies can block a certain location, blocking the entire deployment of their stack.

This means they have to go back to trace why something is wrong, adjust the code, go through all steps again and hope they did not miss anything.

Shift left

One solution to this problem is to test as early as possible. If a team gets feedback on their code, this will save a lot of time in the stages later on. It will also give the teams the confidence they are doing things right!

Open Policy Agent (OPA)

OPA states the following on their website:

Stop using a different policy language, policy model, and policy API for every product and service you use. Use OPA for a unified toolset and framework for policy across the cloud native stack.

Whether for one service or for all your services, use OPA to decouple policy from the service’s code so you can release, analyze, and review policies (which security and compliance teams love) without sacrificing availability or performance.

Source: Open Policy Agent website

Using OPA for testing your Terraform plans

You can use Open Policy Agent for a variety of use cases. Testing Terraform plans is only one of them, but really interesting for DevOps team.

The language for writing the OPA policy is called rego. It is easy to understand and should be relatively simple to learn. To get a grasp, take a look at the explanation of the language on the OPA site: https://www.openpolicyagent.org/docs/latest/policy-language/.

A very complete manual is located at https://www.openpolicyagent.org/docs/latest/terraform/ and helps you understand writing a test.

The great thing about the tool, is that you can use it in your pipelines, running the eval command in for example Azure DevOps. There is even an installertask for Azure Pipelines.

Getting started

If you are using Terraform, just download OPA, and go to the getting started page. Use the given example to test against your own Terraform plan outputs.

Just want to play around with the rego language? The site https://play.openpolicyagent.org/ contains several examples, and makes exploring easy, as it is an online playground without the need for installing anything!

Spoiler alert; no.

Low code vs Traditional Dev

My colleague, friend and SogetiLabs Fellow Daniel Laskewitz and I frequently talk about this topic. His field of expertise as Microsoft MVP covers the Microsoft PowerPlatform including low code systems like Power Automate (formerly Flow).

All too often people see a division between low code and traditional development using languages like C#, Java, TypeScript and Go.
In the real world however, these systems work together perfectly.

Most of the times, you cannot solve a problem with only low code. Think about scenario’s where you should link to old legacy systems or complex API calls. In those cases low code without any enhancement cannot natively connect to those systems.

Behold custom connectors

In the Microsoft ecosystems, custom connectors allow you to bridge this gap. This way, the low code system can interact with any system you write a connector for. This may be common knowledge, but the fact is that most developers do not see how big this really is.

This means you can link any PowerApp, Microsoft Flow, or LogicApps to your custom connector, and reuse those within your entire organisation.
You could even publicly publish these if you have a service you want to expose. So if you are an ISV, this can help you get more traction on your product.

Bridging the gap

In the end it all comes down to developers of any system and language understanding the capabilities of the platforms they and their companies are using. For low code developers this means sometimes calling in the help of traditional developers. And more importantly, this also means traditional developers should learn that these low code systems can help you simplify (and thus speed up!) your development by using ready-to-roll systems and connectors available to you.

As there are over 325 connectors available, that should really speed making connections up!

Get started!

Want to explore custom connectors? Look at these resources or feel free to contact me or Daniel, we strongly believe bridging this gap between low code and traditional dev is key for succes in the future of development!

Hashicorp has done a terrific job making Terraform with the ability to template any API, and deploy and manage stacks in Azure, AWS and many, many more.

I’ve been following Pulumi for a while, and their approach is a bit different than the previous stated tools.

Pulumi states their product as follows:

Modern Infrastructure as Code.

Declare cloud infrastructure using real languages. Enable developers and operators to work better together.

With ‘real’ languages they mean programming languages. At this moment JavaScript, TypeScript, Python, Go, or any .NET language, including C#, F#, and VB are supported. I did some testing with C#, and although it is in preview, it works great:

Installing Pulumi

The install is pretty simple. This page will get you up and running in no time. I personally used Chocolatey but you can download the tool for Windows, MacOS and Linux if you want.

Getting Started with Azure

The getting started guides work great. The only thing you need (in my example for Azure) is an account. The page https://www.pulumi.com/docs/get-started/azure/ show you a step-by-step guide.

Deploying resources

I’ve picked a simple setup, having one resource group in Azure, with one Storage Account. To deploy the stack to Azure, a simple ‘pulumi up’ command fires up Pulumi. Just like Terraform, you can see what will happen if you deploy this command. In this case you see that a Stack in Pulumi is created, and a resourceGroup and storage account will be setup in Azure:

After choosing ‘yes’, you will see Pulumi actually deploying your stack. In the picture below the account has been created, and we see the output plus duration (don’t worry, the storage account has been deleted so the key not valid anymore):

Updating resources

The next step shows what happens if we update something in the stack. In this case, we have updated that the storage account only allows HTTPS traffic. The pulumi up command shows us the expected changes, and shows us it will update the storage account:

After choosing yes, the update will start, and it will display the next image. As you can see, 1 resource was updated, and 2 were unchanged:

Destroying resources

To destroy your resources, pulumi destroy will tear down everything, up until the stack in Pulumi itself:

Wait, Pulumi Stack?

Pulumi defines stacks of your deploys. To quote them on this:

Every Pulumi program is deployed to a stack. A stack is an isolated, independently configurable instance of a Pulumi program. Stacks are commonly used to denote different phases of development (such as development, staging and production) or feature branches (such as feature-x-dev, jane-feature-x-dev).

So, you can look at these like your set of resources. You can group them under Projects so it can be easier to manage. The webinterface of Pulumi is clean, and you can inspect all of your stacks and settings there:

Conclusion

This short blog is just to show you that there is more than the standard way of working with the cloud. Obviously you can use various APIs, SDKs or templates, but these sometimes can be a bit overwhelming for developers. Pulumi lets you choose your language, and makes sure your developers do not need to learn another language and still use the modern infrastructure-as-code techniques!