Collaborating with multiple authors with GitHub

The great thing about GitHub is the way it brings developers together and promotes collaboration. In this example, I’m going to take a DSC module from Microsoft (xSQLServer), and combine it with some pull requests from Marko Bozikovic (@bozho). I’m then going to merge three of his pull requests so I can test 3 fixes that’s he’s contributed related to SSRS. Eventually, these will get merged into Microsoft’s code base, but I’m the impatient type and I want to use them now.

This is post assumes you’ve got some basic familiarity with Git, if not, check out my introductory articles starting here.

Workflow recap

When you want to submit a fix for a repository you don’t own, you follow this workflow:

  • fork (think ‘copy locally’) the remote repository
  • switch to the branch that you want to write the fix for (e.g. ‘dev’)
  • create a branch, by convention, this typically references the issue number & name, e.g. “issue-569-sql2016-virtual-dir-name”
  • make your changes – you’ll likely have to update tests, amend documentation, and you’ll need to make sure you pass any automated checks.
  • submit a pull request.

The pull request on GitHub includes a conversation thread where comments can be raised and addressed, see an example here.

Finally, someone with write access to repository will commit the change (in the case of xSQLServer, that’s going to be a Microsoft employee).

Clone and switch to dev branch

To begin, lets clone Microsoft’s repository, this is called ‘forking’.

To switch to the development branch, run

  • git checkout dev

Or within Visual Studio Code you can list & change branch like this:

Combining work from other branches

Now we want to bring in some code by someone else. We’ll start by adding a link to @boxho’s xSQLServer repository, that we’ll call ‘bozho’, and fetch it

This will pull in any new branches or tags created, in this case we’re interested in the fixes for issues 568,569 & 570

In order to test all three, we’ll create a branch and merge all three into it:

  • git branch dev-ssrsfixes
  • git checkout dev-ssrsfixes
  • git merge bozho/issue-569-sql2016-virtual-dir-name bozho/issue-568-optional-sqlcredential bozho/issue-570-url-reservations

Git is smart about the way it merges changes and can often do them automatically, but occasionally it needs human intervention to resolve a merge conflict, as we see above.

In Visual Studio Code, click into the git workspace, and you can see the list of changes pending. “Merge changes” need intervention, “staged changes” have been managed OK.

If you click on “CHANGELOG.md” in the above example, you’ll be able to work through the file and make the necessary alterations, in this case, it’s a simple documentation change required to merge the list of ‘unreleased’ fixes

Make your changes, stage them (click the “+” next to the file), then supply a commit message as normal to save it to your local repository:

The ‘dev-ssrsfixes’ branch now contains a copy of Microsoft’s dev branch, combined with three fixes from another developer – we can now grab this DSC resource from Explorer and test it out.

  • Copy the contents of your xSQLServer repository, excluding the ‘.git’ folder, to the target machine %programfiles%\WindowsPowerShell\Modules\xSQLServer\7.1.0.0\

Tidy up

Once testing is complete, you can remove your branch:

  • git checkout dev
  • git branch -D dev-ssrsfixes

& remove the remote from @boxho:

  • git remote remove bozho

Keeping in step with remote repositories.

As time passes, and Microsoft push fixes to the xSQLServer repository, your cloned is going to get behind. To stay up to date, you’ll need to add a link to the remote repository, which we’ll call ‘upstream’

Finally, to commit code in your local repository, run:

  • git pull upstream

e.g. if you’re working on the dev branch, it would be “git pull upstream dev”.

To update your remote repository (e.g. the copy on GitHub), run ‘git push’

  • git push
Advertisements

Version Control for IT Pros

In this series, I’m going to write about using PowerShell with Visual Studio Code (“VS Code”) on Windows.  If you’re on Mac or Linux, don’t feel left out, VS Code & Git are available on these platforms too, and it is an excellent editor for many languages, so if version control is new to you too, why not follow along as well?

Version control of scripts, software code, or other text-based assets such as websites is an important safety net, it’s ubiquitously used by programmers, where its value as a record of change and as a collaboration tool is accepted & well understood.

I’ve met many IT pros are either unaware or choose not to use version control. If minor changes are required, they might be made ad-hoc by editing files directly. If it’s an important script or a significant change, the old version might be renamed or placed elsewhere. Folders might get cluttered with old versions, or it might be hard to identify the right version. Change might well be recorded somewhere (inside some ITSM tool, perhaps), but typically it is hard to get a detailed line-by-line record of how a particular file has evolved over time, and who might have made those changes.

Good version control becomes even more critical when you start to break your scripts into shareable modules, since changes to a module upon which others depend may have significant consequences. Generally, as IT pros become more dependent on automation and scripting, managing these resources becomes increasingly important.

Visual Studio Code, a great IDE for coders, web developers & IT Pros.

Visual Studio Code is a relatively new free & open-source IDE from Microsoft, more feature rich than a basic text editor, but much lighter than the developer-centric Visual Studio. The download for VS Code is pretty small at around 34MB.

The IDE was originally developed as a cross-platform editor for the TypeScript language, and it has an extremely rich extension system, one of which is for PowerShell (it has heaps more). This PowerShell extension is, in many ways, more flexible than PowerShell’s baked-in ISE. It has matured very well in recent months, filling in functional gaps found in earlier versions.

There are many benefits to using this IDE above the basic PowerShell ISE, but in this article, we’ll focus on version control integration using Git.

Did you just call me a Git?

Git is a free & open-source distributed version control system for text files (mostly).

It is ‘distributed‘, because, everyone who works on the code has a full copy of the current files and all version history. This sounds quite inefficient, but there are huge projects using this mechanism and it scales surprisingly well. The distributed nature it means you’re not completely dependent on the availability of a centralised repository.  Since you can work completely offline, you can carry on hacking on your multi-million-dollar idea when stranded on a desert island, at least until the battery runs out.

The version control’ provides a way to track changes within a folder hierarchy. These changes & their meta-data are all self-contained in a .git folder at the root of the folder being managed (the .git folder should never be interfered with).

Each top-level folder to be managed is called a repository, typically a repository holds the code for a particular project or set of related activities. Since all the version information is self-contained, these repositories are completely portable – they can be moved around or stored in different places on different workstations.

One of the most exciting parts about Git is the development of social-media-style websites, where developers & IT pros can co-author code, collaborate, track issues in a very open and transparent way. You’re probably already heard of GitHub, or at least come across it – it’s where the brave new Microsoft open-source their software, including, for example, PowerShell, ASP.Net Core & Visual Studio Code. Repositories stored on another system are called remote repositories, or just ‘remotes‘.

I’m going to use GitLab to illustrate hosting a remote repository, which is very similar. GitLab provides free repositories that can be made ‘private’ to a group of individuals, which is a paid-for service on GitHub.

  • If your company is already using a private Git repository, you’ll need to request access from your organisational admins. They will typically send an invite to your company e-mail address, with instructions to follow.
  • If not, you can follow along by signing up, and you can use my repository.

The skills you learn here are directly transferrable to GitHub – I use both simultaneously, GitHub for personal projects that I open source, GitLab for my work, and Visual Studio Code switches between GitHub and GitLab seamlessly, as I move between different projects.

Let’s get installing

Let’s get some software installed, we’ll make this quick to avoid boring the *nix people. I assume you’re using chocolatey (if you’re not, you should install it). The following will install git + visual studio code along with a couple optional-but-highly-recommended utilities (PowerShell for Git aka ‘poshgit’, and a helper to cache login details for GitHub and GitLab in the Windows credential store)

choco install git.install -y

choco install git-credential-winstore -y

choco install poshgit -y

choco install visualstudiocode -y

OK, so, now you’ve finished chuckling about being a ‘poshgit’, and have software installed, let’s move on!

Identify yourself

Before we can start using Git, we need to identify ourselves, we’ll do this being using the Git command line tool, before switching to Visual Studio Code:

Find Git CMD:

Configure it using the name and e-mail that you signed up for with GitLab

git config –global user.name “My Name”

git config –global user.email “my@email.com”

Create an empty folder to store the repositories, it should be a local directory on your computer. I typically go for something short like C:\Work to avoid typing too much, if you’re on *nix then stick it somewhere easy like ~/work.

Remember, this repository will be stored in GitHub, so as long as you synchronise reasonable frequently that should protect you from lost work.

Clone your first repository

When you want to work on an existing repository, you clone it. This takes a copy of the remote repository and copies it to your machine.

To do this, you need the repository URL, in the case of GitHub or Git Lab, this is the same as the website address for the repository.

I’ll be illustrating this using the repository here: https://gitlab.com/kewalaka/vscode-training.  This contains a small collection of simple Powershell scripts.

On the home page for VS Code, select “Clone Git repository”

If you’ve closed the Welcome page, you can get to it from the Help menu -> Welcome.

Enter the Repository URL:

Chose the parent folder (note, it will create a subfolder with the name of the repository)

Select “Open Repository”

The contents of this GitLab repository will be saved to your file system, and you can see them in the tree view on the left of the IDE.

If you’re the curious sort, you can confirm the presence of the repository by looking inside the hidden ‘.git’ folder:

Creating a brand-new repository

Let’s say you’re starting a new project ad there’s no existing repository. The easy way to do this is to first create the project on GitLab, here: https://gitlab.com/projects/new

  • Enter the project name.
  • Enter a description
  • Choose the visibility level. Note that “Internal” means that any user with an account on GitLab will be able to clone this repository.

Having created the project, you can then clone it from Visual Studio Code using the same mechanism above.

Branching out – keeping your production code safe

Central to Git is the idea of ‘branching’, this is typically combined with a workflow that allows collaboration on independent issues and features, whilst reducing the risk of changes to production.

The workflow possible is as complex as your organisation can imagine, but one approach looks like this (also illustrated below):

  • There are three branches – master, hotfixes and development
  • New code is developed and tested in the ‘development’ branch
  • Once signed off, it is merged to the master branch, which is for production ready code
  • For larger projects, specific releases are typically ‘tagged’, this allows the team to identify the code that belongs to a specific version of software.
  • If a production issue arises midway during development, the hotfix branch is used to clone production and a patch is created and pushed to both the master & development branches.

By default, if you don’t branch, you’ll edit the master branch directly. This might be OK for very small projects, or if you’re just starting out. Even if you’re just working on a folder library of scripts, wouldn’t it be great to know which are production ready, whilst allowing work to be done in development to improve them? Always treating the master branch as production ready makes that possible.

You can create a new branch or switch to an existing one in Visual Studio code by using the command bar (Ctrl+Shift+P), and typing git checkout branch-name

If the branch doesn’t exist, it will be created in your local git repository.

Don’t worry if you’re not quite following this, we’ll pick up branching in the next post.

What’s next?

So, you’ve begun the journey towards protecting your precious scripts using version control. In the next post, we’ll look at how to start making changes using the above workflow.

Launch SQL Server Management Studio (SSMS) as a different user

Many organisations require that administrators have a separate user account for work on production systems, different to the one they use for day to day work.

Unfortunately, newer versions of SSMS display this when you try to launch (or ‘runas’) SSMS as another user.

Neither option, “Restart under different credentials” or “Cancel the task and return to Microsoft SQL Server Management Studio”, will allow you to run SSMS as another user account.

One option is to turn off UAC (in case you don’t know what that is – it’s “User Access Control” – the box that takes over your screen when you try and do something needing admin rights, e.g. install software)

Whilst there is no shortage of opinions and holy wars on the usefulness of this security feature, I believe that in this age of ‘drive-by website attacks’, turning off security protections puts your machine at greater risk. If you agree & would like a quick fix, read on.

Resolution

To allow you to run SSMS as a different user, you need to make one change to the SSMS shortcut.

First, find SSMS on the start menu, right click on it, and select “Open File Location” to find the application shortcut:

Right click on the “Microsoft SQL Server Management Studio” shortcut, select Properties. Under the compatibility tab, click ‘settings for all users’, & tick “Run this program as an administrator”

Then, you’ll be able to run SSMS as another account.

How do you launch an app using another account?

In order to launch a program as another user, you need to copy the application shortcut to your desktop, or right-click on the shortcut and select “Pin to taskbar”.

Having done this, hold down shift and right click on the SSMS shortcut, and select ‘Run as a different user’

Put in your admin credentials, then you will be able to successfully launch SSMS, using your admin account.

If you can’t see ‘Run as different user’, make sure the shortcut is on your desktop or pinned to your taskbar.  Note, the start menu doesn’t seem to offer this function.  Make sure you have shift held down before you right click on the shortcut.

Chocolatey, a sweet package manager

chocolatey

Coming from a Linux system, Windows software installation appears rather confused and fragmented. On Ubuntu (and other Debian derivatives), we have:

  • apt-get install – to install a piece of software
  • apt-get update – to get the latest software versions available
  • apt-get upgrade – to upgrade every piece of software on the system

Using similar commands, it’s even possible to upgrade the entire operating system (apt-get dist-upgrade)

On Windows, we;

  • find the software from the vendor’s website. Hopefully, we pick the right download.
  • download it & run the setup
  • answer a bunch of questions & “click next” a few times, whilst being careful to avoid any agreements that try to install spyware.
  • wait for the software to nag us about upgrading (sometimes)

Automating these installs comes with a number of challenges, there are multiple install technologies with different ‘switches’, e.g. MSI, NullSoft.

Chocolatey is the beginning of a new way – it starts to bring the simplicity of *nix software installation to Windows. Once it is installed, software can be installed with a simple command, e.g.

  • choco install google-earth (to install Google Earth)
  • choco install jre8 (to install Java Runtime Environment)

These commands download software from a vendor’s website, and silently install without any user intervention, similar to apt-get, yum or rpm.

Chocolatey can also upgrade every piece of software under its management, using a single command:

  • choco upgrade all

Let’s just bask in how awesome that is before we continue.

The freely available online repository has, at the time of writing, 4778 packages available. It is straightforward to create an internal repository – only a web server is required, which can be used to host packages which cannot be publicly redistributed, or to avoid having to trust an external resource for software installs.

The client installation is via a simple script – it’s not quite a simple as perhaps it could be (an MSI would be nice), but it is worth it.

Once installed, the simple command-line syntax makes it ideal for scripting, there are resources for Powershell DSC, Puppet and Chef (others too).

A licensed version provides a number of additional features, such as:

  • create chocolatey packages using a simple GUI
  • take software currently installed on an existing PC and make it managed by chocolatey.

Prevent software from upgrading

In the event a piece of software causes an issue, it can be ‘pinned’, for example, this pins the Git Credential Manager to the current version

This will then not be upgraded until the pin is removed:

What about a GUI?

There is a basic GUI, but Chocolatey’s strength lies in making software installs simple to automate. This simple mechanism makes it straightforward to build an ‘appstore’, or integrate it into an existing business workflow.

Why I think chocolatey is great

  • Chocolatey makes software installs easy, it has a community that keeps common, frequently updated, software such as Google Chrome up to date
  • The simple syntax makes it easy to automate, there is no need to load complex Powershell modules, or connect via PS Remoting to a complex console.
  • It is easy to create packages
  • You can use the public chocolatey packages to get started and there are nearly 5000 installers already available.  Why repackage Chrome, Java, etc yourself?
  • If you want to host your own repository for your own software, the infrastructure behind chocolatey is really simple – it’s just IIS, packages are just files. This can be scaled to deal with larger organisations using existing mechanisms, e.g.
    • The website can be load balanced to provide high availability
    • Packages can be distributed to remote sites using DFS-R (or similiar).
    • A simple workflow can be established to connect some users to an early release source, for testing and QA.
    • It’s simple to introduce basic processes to manage software – e.g. create a scheduled task for choco upgrade all to keep all your software up to date.
  • Using the business edition
    • end users can install software themselves without admin rights (Youtube video)
    • it is even easier to create packages (Youtube: there is a simple GUI too), often it automatically works out the automated switches required to install software.
    • packages can be automatically created from software installed on existing systems (Youtube video)
    • .. and much more
  • Combine it with Boxstarter and automate building your next dev machine!

Dealing with passwords in PowerShell

password

Oftentimes, it is necessary to store a password or some other secret within a script. It’s not uncommon to see the password in plain text in a script, but that does create more than a few concerns.

If you use a source code repository to version control your scripts (which you do, right?), this means your passwords are probably stored in plain text in this repository, which is OK(ish), until you start using a public one like GitHub, or if you’re the paranoid sort that thinks the NSA tracks the elasticity of your socks via your usage of cloud computing.

So, what should we do?

I’ve created a password repository on GitHub that explores various alternatives that I’ve used to date.

Get in touch on there, or on here, if you’d like to discuss or improve.

Benefits realised, risks managed, using DSC

Desired State Configuration is a framework developed by Microsoft for managing systems, that provides significant benefits compared to traditional automated builds, or manual installs.

Benefits extend beyond the initial build, into the operation of a system, for example, it can provide:

  • A means to ensure on-going compliance, reduce risk & increase security
  • Improvements to the quality of change & release management.
  • A method to document the state of a system, and with good process, ensure these documents automatically remain relevant & up to date.
  • A community that encourages collaboration + continuous improvement with other IT pros and developers, on a global scale.
  • & many more benefits.

The technical concept is not new, having roots in Linux & Unix-based systems, but in recent years has become fully integrated into the Windows too.  Although it is Microsoft-centric, the framework provides support for managing other operating systems, devices, applications, and systems.

What makes it different?

Many systems are built using an imperative approach, meaning, we script the specific steps to take, in a specific order, to take a system from one state to another. This might, for example, install and configure software required for a business application.

Once this process is complete, there is no further need for the script, except if it needs to be repeated. In fact, in many cases, re-running this process on an existing system can be dangerous, as it may attempt to re-do activities already complete, potentially leaving a system in a broken state.

Desired State Configuration (DSC) is declarative, but, what does that mean? To understand, we will break DSC into a few component parts:

  • Templates
  • Resources
  • Environmental settings
  • The “Make it so” engine

Let’s first describe these in the context of a restaurant:

040117_2012_Benefitsrea1.png040117_2012_Benefitsrea2.png040117_2012_Benefitsrea3.png040117_2012_Benefitsrea4.png

When creating declarative systems, you create a template that specifies what you want, e.g.

  • Create a web server
  • Place some content in it
  • Set it up for a specific domain name “mycoolwebsite.nz”
  • Make sure it is secure (e.g. turn https on).

A component built into Windows is supplied with this template & the necessary resources and makes it happen on the system.  We’ll talk about these resources in a bit. This maître d’ component is called the Local Configuration Manager, if you’re a Star Trek fan, you could think of it as Captain Picard on the bridge of the Enterprise, saying “make it so”.

Having declared what we want the system to be, the “make it so” engine in Desired State Configuration will continually check the system against the template, to make sure the configuration remains in the state that you desire.

For example, if someone or something was to make a change (e.g. turn off “https”), the next time the system was compared against its template, it would be found to be out of step. DSC can either ‘monitor’ and report changes or alternatively take direct action to re-apply the template, restoring the system to its desired state.

Re-iterating, in this declarative model, we expect to run the same script repeatedly, this effectively becomes an on-going quality and compliance check by which the system can be measured.

Many resources are free, and openly developed

Hopefully, you can see how important the resources are. They describe how the actions in the template will be undertaken.

These resources can perform all sorts of actions, here are a few examples

  • ensuring the presence of files in a location
  • setting a registry key
  • creating a database login,
  • ensuring a piece of software is installed,
  • configure complex configuration in systems such as Active Directory, SQL, Exchange
  • install a package on a Linux server
  • .. and there are hundreds more, many posted on the Powershell Gallery

Microsoft has released the source code for these, and many more resources, in an open forum where they can be freely used, shared and improved. More are created and shared by a thriving global community.

The permissive sharing model allows resources to be used in many scenarios, with testing and feedback supplied by a global network of contributors.

Virtually every resource comes with examples and samples templates, these illustrate both simple scenarios, and also very complex ones (e.g. a highly available SQL Server), and provide guidance to help your team get started.

A template to match all your environments

The environment in which the template is deployed can be specified separately, this allows scenarios such as:

  • The ability to build the same application template but in different environments.
  • Ability to scale, or size, each environment depending on its purpose. E.g. a production environment might have many web servers to cope with the load.

By using DSC, you can ensure consistency across many different environments, serving different purposes. This provides greater assurance that each environment is functionally the same, reducing the likelihood of issues as changes are tested & deployed.

Since DSC is built on Microsoft’s ubiquitous scripting language, PowerShell, its reach and flexibility is extremely broad.

Furthermore, since DSC has a simple, text-based, format, makes them easy to share & differences can be straightforwardly compared, and historically tracked (using version control).  There are high-quality editors available to simplify the creation of these templates & assist in checking their quality.

Benefits realised.

Hopefully, having read this, you can understand how a declarative approach can be used to achieve the benefits set out at the start of this article

Where to start?

If you’re a manager, ensure you reward people who are automating using tools that encourage collaboration. This is the first step. Encourage your team to look at declaratively modelling a specific system. There are many products, such as Chef, Puppet, that can integrate with DSC and ensure you are able to scale your new declarative world. DSC can also be used by itself, and that is probably the best way to begin.

A great place to begin technically is the free Microsoft virtual academy courses. Even though they are a few years old, they remain relevant.

If you’re interested in a more technical explanation of declarative vs imperative approach, I highly recommend this article from Puppet Labs.

If you’d like to follow my journey, then sign up to my RSS feed, and /or join me in the conversation below.