Automating Google Cloud Platform Snapshots with PowerShell

As part of a project I am working on, I needed a way to automate disk snapshot creation and retention on Google Cloud Platform.  I found several good examples of how to do this in BASH scripts etc.  However, I was unable to find anything in native PowerShell that I liked.  So, I wrote some scripts and decided to publish them here and on GitHub in the hope of helping someone else.

  • create-snaps.ps1 automates snapshot creation.
  • remove-snaps.ps1 automates snapshot cleanup based on the age of the snapshot.

Obviously, you will need to schedule these to run at some interval.  I’ve used Windows task scheduler for that purpose, and it seems to work great.

Obvious Warning: Please use these scripts at your own risk. I accept no responsibility for your use of them. However, if you run into any issues please let me know so I can work to improve them.

create-snaps.ps1

# Set the path and file name for PowerShell transcripts (logs) to be written to.
$LogPath = "c:\logs\powershell\snaps\"
$LogFile = Get-Date -Format FileDateTimeUniversal
$TranscriptFileName = $LogPath + $LogFile +".txt"

# Start the transcript.
Start-Transcript -Path $TranscriptFileName

#Set the GCP project.
$Project = "put-your-gcp-project-here-12345"

#Set the zone(s) where the disks are that you would like to take snapshots of.
$Zones = "us-east1-d", "us-central1-c"

#Record the date that the snapshots started.
$StartTime = Get-Date

#Go snapshot all of the disks in the zones identified above.
foreach ($Zone in $Zones) {
$DisksInZone = Get-GceDisk -Project $Project -zone $Zone | foreach { $_.Name }

foreach ($Disk in $DisksInZone) {
Write-Host "=========================================="
Write-Host "$Zone "-" $Disk"
Write-Host "=========================================="
Add-GceSnapshot -project $Project -zone $Zone $Disk #In the future we could clean this output up a bit.
}
}

#Record the date that the snapshots ended.
$EndTime = Get-Date

#Print out the start and end times.
Write-Host "=========================================="
Write-Host "Started at:" $StartTime
Write-Host "Ended at:" $EndTime
Write-Host "=========================================="

#Stope the transcript.
Stop-Transcript

#Send the PowerShell transcript (log) by email. You can delete this entire section if you don't want log copies delivered by email.
#Google Cloud Platform blocks direct outbound mail on port 25. Reference: https://cloud.google.com/compute/docs/tutorials/sending-mail/

#Mail Server Settings
$smtpServer = "mail.yourdomainname.com"
$smtpPort = "2525" #Don't put 25 here it will not work. See link above.

$att = new-object Net.Mail.Attachment($TranscriptFileName)
$msg = new-object Net.Mail.MailMessage
$smtp = new-object Net.Mail.SmtpClient($smtpServer, $smtpPort)

# Set the email from / to / subject / body / etc here:
$msg.From = "gcpsnapshots@yourdomainname.com"
$msg.To.Add("you@yourdomainname.com")
$msg.Subject = "GCP Snapshot Report"
$msg.Body = "Please see the attached PowerShell transcript."

# Attach the log and ship it.
$msg.Attachments.Add($att)
$smtp.Send($msg)
$att.Dispose()

remove-snaps.ps1

# Set the path and file name for PowerShell transcripts (logs) to be written to.
$LogPath = "c:\logs\powershell\snaps\"
$LogFile = Get-Date -Format FileDateTimeUniversal
$TranscriptFileName = $LogPath + $LogFile +".txt"

# Start the transcript.
Start-Transcript -Path $TranscriptFileName

#Set the project.
$Project = "put-your-gcp-project-here-12345"

#Record the date / time that the snapshot cleanup started.
$StartTime = Get-Date

#Choose what snaps to remove. Essentially, the script takes the current date / time subtracts 30 days and sets a variable ($deletable). Is delatable even a word? Anyway... Any snaps older than that variable get removed. Obviously, you could tweak this number of days to fit your needs.
$deleteable = (Get-Date).AddDays(-30)

#Log what date and time we set $deleteable to.
Write-Host "Deleting snapshots older than:" $deleteable

#Delete the actual snaps.
$snapshots = Get-GceSnapshot
foreach ($snapshot in $snapshots) {
$snapshotdate = get-date $snapshot.CreationTimestamp
if ($snapshotdate -lt $deleteable) {
Write-Host Removing snapshot: $snapshot.Name
Remove-GceSnapshot $snapshot.Name
}
}

#Record the date / time that the snapshot cleanup ended.
$EndTime = Get-Date

#Print out the start and end times.
Write-Host "=========================================="
Write-Host "Started at:" $StartTime
Write-Host "Ended at:" $EndTime
Write-Host "=========================================="

#Stope the transcript.
Stop-Transcript

#Send the PowerShell transcript (log) by email. You can delete this entire section if you don't want log copies delivered by email.
#Google Cloud Platform blocks direct outbound mail on port 25. Reference: https://cloud.google.com/compute/docs/tutorials/sending-mail/

#Mail Server Settings
$smtpServer = "mail.yourdomainname.com"
$smtpPort = "2525" #Don't put 25 here - it will not work. See link above.

$att = new-object Net.Mail.Attachment($TranscriptFileName)
$msg = new-object Net.Mail.MailMessage
$smtp = new-object Net.Mail.SmtpClient($smtpServer, $smtpPort)

# Set the email from / to / subject / body / etc here:
$msg.From = "gcpsnapshots@yourdomainname.com"
$msg.To.Add("you@yourdomainname.com")
$msg.Subject = "GCP Snapshot Cleanup Report"
$msg.Body = "Please see the attached PowerShell transcript."

# Attach the log and ship it.
$msg.Attachments.Add($att)
$smtp.Send($msg)
$att.Dispose()
<pre>

The infinite value of awesome interactions.

The last few days, I had an interaction with a company that really impressed me.

Backstory:
I was looking for a couple of basic managed network switches for a small project.  Essentially, I wanted basic enterprise switch functionality on a shoe string budget.  I emailed a friend (Jason B.) who does a lot more Cisco work than I do and he suggested I take a look at the Cisco 3750E series.  He event sent me the link below:
https://www.cablesandkits.com/cisco-3750e-series-24-port-switch-wsc3750e24tds-p-8042.html

I took a quick look and this switch seemed appropriate.  I was about to go out of town, so I moved on to other things without placing the order. I figured I’d order them when I got back in town.

Curious:
For the next few days I started seeing ads for this exact switch on what seemed like everywhere I went online.  I run a couple of ab blockers so I only see ads on pages that have convinced me to turn them off or in applications where my ad blockers are not effective.  However, suddenly that little Cisco 3750E seemed like it was chasing me all over the internet.  Open Skype – boom – Cables and Kits Cisco 3750E ad.  Open a web page where my ad blockers were off – boom – more ads just like the one below.

I understand at a technical level exactly how this works.  However, I don’t think I have ever been so well targeted all the way down to the specific product level in my life.  These were not generic Cables and Kits ads, or ads for generic Cisco gear – these were ads reminding me to go buy the exact switch I had looked at.  To make this even more impressive – this is not a new switch – they don’t have thousands of these – when I first looked, they had three in stock.

Snark:
After a few days of this, I was feeling a bit snarky.  So, I tweeted this:

We had some good back and forth, and @cablesandkits mentioned they would be sending me a small gift.  I figured I’d get a sticker etc.  Yawn.

Wow:
So, I go away on my trip and essentially forget about all of this.  Then I come home one day and there are two boxes from Cables and Kits waiting for me.  They actually sent me the switch, a bunch of great Cables and Kits swag and a handwritten note personally signed by 15+ team members!  How awesome is that!  I was totally blown away.

Impressive:
The Cables and Kits Digital Marketing folks are doing an awesome job.  I’m a busy guy. Most of us who work in I.T. are.  It’s very hard for a sales / marketing message to pop out above the noise. Cables and Kits took one short visit to their site and transformed it into an experience that really got my attention.  It’s a bit ironic that awesome ad targeting led to what was ultimately a very human interaction. The Cables and Kits folks did a great job with both methods of interaction.

Takeaways:

  • More than anything else, your reputation matters.  Without my friend having previously had positive interactions with Cables and Kits, he would have never sent me that link in the first place.  If Jason B. said this switch was a good fit for my needs, and that Cables and Kits was a good place to get it, that was good enough for me.
  • Each interaction with a client, or potential client matters.  Cables and Kits took the tiny bit of data they got when I clicked that link, and they used it diligently.  When they had the opportunity to have a deeper interaction through Twitter, they were paying attention and took advantage of it.  They then chose to ramp things up and interact on a more personal level.
  • A culture where you can do cool / non-standard stuff matters.  Over the years, I’ve bought and recommended literally tons of Dell gear.  If my poor Dell rep wanted to send me a hat and a mouse pad, I seriously don’t think he could get it done.  Cables and Kits must have a corporate culture that empowers and encourages people to do this sort of thing.
  • The impact of interactions like this are real, and measurable.  Do you think I’ll buy from Cables and Kits in the future?  Do you think I’ll recommend Cables and Kits to my clients and friends?  Absolutely!

Hats off to the fine folks at Cables and Kits, for the switch, the swag, and the reminder of the infinite value of awesome interactions.

Do what you love.

Disclaimer: I have no idea if this post will ever be helpful at all to anyone else.  Sometimes writing something like this helps me make sense of my own thoughts.  Since I wrote it down, I thought I would share it.

A commonly repeated bit of wisdom is that it is a wise career choice to “Do what you love.”  I’ve heard this advice for years and at first glance, it seems good.  However, as you really dig into it, it can be difficult to figure out what it is that you really love.  What would it look like to actually do what you love for a living?

Is this what I really love?

I love to go to my Aunt’s lake house and ride the SeaDoo around the lake.  It’s one of my favorite leisure activities.  Should I do that for a living?  Is that what I love?  Thoughts like this go through my brain…  Perhaps I can make a living riding a SeaDoo.  Perhaps I could become a SeaDoo racer.  Do what you love – right!  YOLO!

Hang on a moment Mr. YOLO man.  Let’s pause and count the potential cost of doing that.  I might need to move away from my extended family to some warmer climate where you could do this year-round.  I might need to exercise a lot to get in great physical shape to race competitively.  It’s nearly impossible I would be able to earn much of a living doing this in the beginning, so for several years I would need to train to be a SeaDoo racer and work at another job to support my family.  My free time and a lot of my family time would be totally consumed by this.  Hmm.  This is sounding less awesome very quickly.

Once you look at what you think you love in the bright light of reality, the picture changes a bit.  I do enjoy riding the SeaDoo.  However, I don’t love it nearly enough to make all of the sacrifices I would need to make in order to make an actual career out of it.

Now what?  Give up?  Head back to the proverbial salt mine to spend the rest of my days doing something I really don’t love or perhaps even hate?  Nope.  Dig deeper. Here are a couple of things I have observed recently that have made me think differently about this topic.

Observation #1 – Olympic Swimmers

Recently we watched some of the 2016 Summer Olympics.  I was amazed watching the swimmers.  Think about what they did to even make it into that Olympic pool.  They exercised like crazy.  They ate healthy – probably extremely healthy.  The practiced over and over again, nearly perfectly, day after day for years upon years.  They sacrificed a lot of big things in their life only to make it into the Olympics.

Watching them, I sat and thought at first, “Wow.  That is cool. They sure are fast!”  Then I thought a bit deeper about what being that fast must have required and I thought: “You know what – these people are crazy!  Why spend so much of your life for so many years on end to become THAT good of a swimmer?  Who cares!”

I for one frankly do not care nearly enough about swimming to do that.  I would not be willing to invest even a small fraction of the effort that the person who came in dead last must have invested, even if you could assure me that by doing so I could be an Olympic gold medal swimmer.  It’s simply not something I care that much about.  Those swimmers must really love something about swimming.  They have paid a tremendous price to get to this point.  I don’t think it is a price anyone would pay, if they did not love it.

Observation #2 – Wonky I.T. Security Topics

Last week, somehow I came across something that peaked my interest in an I.T. security book called the Art of Memory Forensics.  So, I paid ~$50 and ordered it from Amazon.  It came in on Friday and I proceeded to give it and the great tool that it is written about a ton of my weekend free time.

Why?  Because memory forensics is awesome!  Well – at least I think it’s awesome.  It’s a tool that can help me do something I actually love even better.  It open up another possible angle to attack the problems that I wake up thinking about from.  It is a small piece of a grander puzzle that has had me fascinated for years.

If most of you were to read this same 858 page book, you would hate it.  You would be bored to death.  You would not be willing to invest a fraction of the time that I will happily invest on this topic.  You would probably not do it even if you could become one of the best memory forensics people in the world.  You would nearly die of boredom or confusion or perhaps both during the first few hours.  Why?  Because most of you don’t care at all about this topic.  You don’t care so much that I bet your brain would nearly refuse to focus on this for long enough to really learn much about it.  You don’t care about memory forensics in the same way I don’t care about swimming fast.

What I have learned

Why do I love topics related to Computer Security?  Honestly, I have no idea.  I just do.  Perhaps it is just what God put me on this Earth to do.

Looking back over my life, I can see a clear interest in this topic all the way back to when I was a kid.  I remember one year, my family was at the beach and somehow I ended up reading a bunch of Tom Clancy books.  Perhaps they happened to be at the house we rented for the week.  Spy stuff, military stuff, tapping undersea cables to gather intelligence on the bad guys – all of it seemed so fascinating!  I did not get nearly the amount of sun that my brothers and cousins did that year.

After hearing I was a Tom Clancy fan, my High School Principal pointed me to a book by a guy named James Bamford titled The Puzzle Palace.  I had never heard of the Nation Security Agency before this book, but I read every word.  Again – totally fascinating.

Fast forward my life story a bit and I ended up in an I.T. career.  As my career has progressed, I’ve always gravitated to areas that fit within the broad categories of Information Assurance / Information Security.  I love configuring firewalls.  Seeing an IPS alert on a blocked attack is an actual thrill for me.  I love well planned and well configured backup systems.  Quickly restoring data that was destroyed by a crypto ransomware attack knowing that the capability to do that mean the criminal will not get a dime of my client’s money makes me happy on the inside.  I imagine I love these things as much as those crazy swimmers love swimming.  It’s just IN me.  It’s what I actually love doing.

Could I do something else?  Sure, but I might not enjoy it enough to get really good at it.  For me, this is the area that is so fascinating that I will willingly invest my free time and personal money to learn even more.  For me this is a marker.  It’s a hint.  It’s an indicator.  It’s a pointer that points to what I must really love.

My $.02 worth of advice for you if you are wondering about what you really love.

Sometimes people struggle to figure out what they love.  If you can’t figure out what it is that you love, look at your life and ask yourself this question:  What is it that I am so fascinated with that I will happily spend my free time and my own money learning more about?  For some of you it is music.  For others, it’s real estate.  Perhaps for some of you it is cooking.  For others it’s helping hurting people put their lives back together.

Look for patterns.  If you’ve been fascinated with something for years and you’ve spent your own money and your own free time to learn more about it and/or do more of it – pay attention.  That might be your thing.

If you think you might want to make a career out of it, pause and run it through the sacrifice filter first.  Ask yourself “Do I really love this enough to sacrifice what would be required to become good enough at this to make a living out of it?”  Be warned – the sacrifice required will probably be even higher than you expect upfront.

If you are not willing to sacrifice enough to make a career out of something, that’s ok.  Perhaps whatever it is can still be a great long term interest for you.  For me it has been helpful to have thought of some things and then intentionally set them aside as career options.  Doing this frees you up just to enjoy them as interests without getting stuck thinking about a career move that you know deep down you are not willing to actually make.  Set them aside and over time move on to the next thing you identify.  Rinse and repeat.  Eventually, you might hit on the thing you really enjoy where the cost to actually do it fits with what you are willing to sacrifice.

Back to me for a moment…

Do I do exactly what I love 100% of my work time.  Nope.  I honestly doubt anyone does. However, I get to do enough of it that the sacrifices are worth it.

If I’m lucky, I’ll make it to my Aunt’s place this weekend to ride around on the SeaDoo.  However, when I get home and clean up, I’ll probably be thinking about some  I.T. security related topic while I’m in the shower.

If anyone reads this far – I’ll be amazed.  If you do, I sure hope you can find and do work you love as a career too.  If you want to read more on this topic, here are a few links to folks who have shaped my thinking on this.

Windows 10 – Unintentional Upgrades

In the last week or two I have gotten a significant number of calls from clients who have had PCs unintentionally upgraded to Windows 10.  While I generally like Windows 10, I do not believe that Microsoft should be doing what they are doing and essentially upgrading people automatically by using deceptive practices.  So far, I have not seen this happen on machines that were joined to a Windows Active Directory domain.  However, I have now seen it 10+ times on machines not joined to a domain.  Here is what I think you should know.
Microsoft is misbehaving:
If you receive the free Windows 10 upgrade notification and click the X, rather than simply closing the upgrade offer app, Microsoft considers this your acceptance of the upgrade and scheduled the upgrade.  This is absurd and inexcusable.  Here is an in depth story about this.
How to prevent an upgrade from happening if you wish to stay on Windows 7 / 8 / 8.1 etc:
Simply download and run the Never10 app.  A good description of how to do this can be found here:
How to roll your PC back to Windows 7 / 8 / 8.1 if you have been unintentionally upgraded to Windows 10 against your wishes.
Fortunately, this is very simple and so far seems very reliable.  Here are directions directly from Microsoft.
I hope this is helpful to you.

Google Cloud Platform (GCP) vs Amazon Web Services (AWS) vs Microsoft Azure – Cloud IaaS – Price Comparison

I’ve decided to share some public cloud Infrastructure as a Service (IaaS) compute instance cost analysis that I recently created as part of a project for one of my clients.  When choosing an IaaS provider there are obviously many things to consider beyond just compute instance pricing.  Other factors such as storage, network bandwidth, snapshot and replica options and many other factors (and costs) come into play.  Each of these providers offers many different services that may be of differing value to potential customers.

Conclusions (up front for you TL/DR folks):

  • The commonly accepted wisdom is that these providers are locked in a price war and that they have all closely matched each others pricing.  Nothing could be further from the truth.  Instances from Microsoft Azure are dramatically more expensive that Amazon Web Services and Google Cloud Platform no matter how you slice the data.  Google Cloud Platform and Amazon Web Service pricing looks close if you compare total three year costs.  However, how you get those numbers to be close (write AWS big checks upfront) is dramatically different.
  • Based on the numbers we chose for cost of capital (5%) and likely future IaaS price cuts (15% /yr), AWS does in many cases offer the lowest cost three year option IF you are willing to pay substantial amounts upfront.
  • Google Cloud Platform offers extremely competitive pricing with no upfront purchase needed.
  • Windows is expensive.  In some cases the cost difference in a Linux instance and a Windows instance exceeds the cost of the Linux instance itself.  Think about that for a moment.  The cost of your OS choice can more than double the cost of your instance.  I love Microsoft.  I love Windows.  I hope this changes.

Update – 12/6/2016 – A Microsoft rep posted this comment on my LinkedIn post of this article.  Keep this in mind as you compare prices.

If potential Azure customers talk to their local Microsoft sales rep they can chose to buy via a so called “Compute Pre Purchase” option. It will give you up to >45% savings for modern compute instances depending on the location and instance family. You need to decide for a location, instance type and pay for one year upfront but still might be appropriate for many use cases. Microsoft will very soon offer an easier way to leverage those savings and offer more options as well as longer term periods, etc. very soon.

Methodology:

In order to simplify some of the discussion for this purpose of this post, we’ve made the following assumptions.

  • We will look at only four similar instance sizes.
  • We will not consider storage, bandwidth or other costs.  Perhaps that will be a discussion for another post.
  • We will look at the cost difference between running Linux and Windows instances.
  • We will consider and attempt to model the different purchase options available from each provider.
  • We will compare the costs for running these compute instances for both one and three year terms.
  • We will assume 100% sustained use during the entire period considered.

Instance sizes:

  • Small – At least 1 CPU core / ~4GB of RAM
    • Specific instances we chose to compare: AWS: t2.medium / GCP: n1-standard-1 / Azure: GP A2
  • Medium – At least 2 CPU cores / ~8GB of RAM
    • Specific instances we chose to compare: m4.large / GCP: n1-standard-2 / Azure: GP A5
  • Large – At least 4 CPU cores / ~16GB of RAM
    • Specific instances we chose to compare: AWS: m4.xlarge / GCP: n1-standard-4 / Azure: GP A6
  • Extra Large – At least 8 CPU cores / ~30GB of RAM
    • Specific instances we chose to compare: AWS: m4.2xlarge / GCP: n1-standard-8 / Azure: GP A7

There is no perfect way to compare things that are not identical.  So, we have chosen what we believe to be fairly similar instance types to compare.

Provider Pricing Model Discounts:

Each provider offers ways to purchase instances in order to save some money.

Amazon Web Services: AWS offers a variety of purchase options.  These options can result in significant savings.  Explaining how reserved instances work is beyond the scope of this article.  For more detail on this topic go here: https://aws.amazon.com/ec2/purchasing-options/reserved-instances/ .  In general, the longer term you are willing to commit to and the more you are willing to pay upfront, the higher the discount you can get.

Azure: Microsoft Azure offers a flat 5% discount if you are willing to pre-pay for 12 months of service upfront.  https://azure.microsoft.com/en-us/offers/ms-azr-0026p/.  The 5% Microsoft discount is frankly not very enticing compared to the significant discounts you can get from AWS for prepayment, and compared to the discounts you get from Google for simply using instances on a sustained basis.  Since a three year upfront purchase is not possible, when we modeled Azure three year costs we did so by estimating the cost of three annual purchases.

Google Cloud Platform: Google offers great discounts for sustained use.  You don’t need to pre-purchase anything, you get the discounts automatically.  The discounts are very substantial.

Cost of capital:

For purposes of this post we also wanted to consider the cost of capital.  It is also not reasonable to compare spending a large sum of money upfront with spending no money upfront and simply paying for what you use on an ongoing basis.  So, for purposes of this discussion we are going to assign a relatively arbitrary 5% annual cost of capital to options where prepayments are considered.

Expected future IaaS price reductions:

The costs of public cloud IaaS continue to drop.  For purposes of these calculations, when we look at one year costs we will assume that no price drops will happen during the middle of our one year term.  For purposes of our three year estimates, we will assume that a 15% price reduction will happen at the end of year one, and another 15% price reduction will happen at the end of year two.   Obviously, these are best guess estimates and we could easily be wrong.

Shameless Plug:

If your business needs help figuring out how to best architect public cloud infrastructure, we would love to help.

Raw data:

Linux – 1 Year

Note: A 5% cost of capital has been used for these calculations where an upfront purchase was required.

Linux-1year-small

Linux-1year-medium

Linux-1year-large

Linux-1year-extra-large

Linux – 3 Years

Note: A 5% cost of capital has been used for these calculations where an upfront purchase was required.  A 15% annual cost reduction has been estimated.

Linux-3year-small

Linux-3year-medium

Linux-3year-large

Linux-3year-extra-large

Windows – 1 Year

Note: A 5% cost of capital has been used for these calculations where an upfront purchase was required.

Windows-1year-small

Windows-1year-medium

Windows-1year-large

Windows-1year-extra-large

Windows – 3 Years

Note: A 5% cost of capital has been used for these calculations where an upfront purchase was required.  A 15% annual cost reduction has been estimated.

Windows-3year-small

Windows-3year-medium

Windows-3year-large

Windows-3year-extra-large

Learning to code: Week 4 – ASP.NET MVC view model basics

This week, I made limited progress learning to code.  I was out of town on a work project Sunday – Wednesday.  Since I got home, I’ve been slammed with catch up tasks.  However, I was able to read a few good resources while in the air or airports on MVC view models.  I wanted to link to them here for reference.

This next week, I hope to put these resources to good use for a potential enhancement to the MVC framework app Mr. Miyagi and I are working on.

More next week…