Protecting your kids online: What worked for our family.

TL;DR:  If you are a parent who wants to better manage how your kids use the internet – go buy a Circle with Disney box.  You can thank me later.

Back Story: One night last month just before going to bed I decided to review my nine year old daughters iPad browser history.  We are not some sort of crazy – spy on everything your kids do – sort of folks.  In fact in retrospect, we may have been too permissive on this previously.  However, I do think as parents monitoring and managing how your kids use the internet is part of your job as a parent.  So, periodically I check the kids internet history.

It had been a while since I had done that on my daughters iPad.  On this night, I was horrified by what I found.  Earlier that very evening she had been watching some generally kid friendly stuff on YouTube.  However, looking back through the history you could see how her curiosity led her down a bad path.  That bad path ended with her on the web site of a hate group, “learning” about a very sensitive topic.

Waves of emotion washed over me – anger, sadness, and a lot of determination to do something to prevent this in the future.  I showed my wife the iPad history and I went and got my daughter.  She came down to our room and we sat on our bed together.  For the next half an hour or so, my wife and I unpacked the topic she was curious about, shared our feelings and beliefs on the subject and explained that she had gotten some really bad information from a really bad source.  After a long, emotional discussion – we prayed together and she went on back up to her room to sleep.

I felt so bad.  How could I – a dad who is I.T. Infrastructure Engineer for a living – not have done a better job of protecting her from some of the awful stuff online.  After all, I install enterprise grade equipment for corporate clients that filters out this sort of stuff.  My first inclination was to buy a smaller version of what I generally deploy for clients and install it in my home.  However, even smaller versions of that equipment are very expensive and honestly total overkill for home use.  Then I remembered a friend mentioning that he had purchased and deployed Circle with Disney on his home network.  From what he had told me previously – it was working great for him.  So – at  11:43 PM on Saturday night, desperate to take some sort of action to solve this, I fired up my phone and ordered a Circle device for $99 from Amazon.  My expectations were pretty low.  So much consumer I.T. equipment is total garbage.  However, I was determined to do something.

Progress: A couple of days later, the Circle box arrived and I unpacked it.  Setting it up was very easy.  It would be easy for those of you who are less technical as well.  If you are comfortable with using an app on your phone, and you know your home wireless network name and associated password – you can do it.   You simply add the Circle app to your phone, and follow the directions.  The initial process is very well designed and the initial connection process is very easy.  Within a few moments, I had the device on my home network.

Circle App – Home Screen

The way the device works is by discovering devices on your network, and associating those devices with either a particular person, or the house in general then associating each of those entities with some basic use controls.  In our house we ended up with a lot of devices either associated with my wife, myself, or our house generally.  Those profiles and therefore those devices don’t have any sort of filtering or controls applied.  If my wife wants to look at crazy stuff online – who am I to stop her?  Ha!  Seriously – the device just ignores traffic from those devices.  This is great.  The last thing I needed was a consumer grade device interfering with my work use of our home internet connection.

I then associated the devices my kids use to profiles I setup for them.  Circle has some great pre-built filtering profiles that were very easy to customize.  Currently, both of my kids are setup with minimally customized versions of the “Kid” profile.  I turned all of this on without telling either of them.  I wanted to see if they would even notice.  They noticed.

Circle App – Profile Settings Screen

Issues: Shortly after the next time my daughter went online, she noticed that she could no longer see videos from one of the YouTube channels she was watching previously.  Circle was forcing her YouTube views into YouTube Restrictive Mode.  So, as far as she could tell – these videos were just totally gone.  Being the extremely reasonable parent that I am – ahem –  I explained that I had added some filtering and I asked her to come down so we could watch the video in question together.  We made it about five minutes in, and after being subjected to a good bit of profanity and a lot of generally tasteless content – I told her we would not be unblocking this.  This was a catalyst for a good follow up discussion with her about being careful what content you consume online.  Being the generally reasonable daughter she is, and knowing there is no way on earth I would change my mind, she went back to watching kid appropriate stuff.  Bingo!


Circle App – Per Profile – Filter Settings Screens

A bit later my son came down.  He is a bit older than my daughter, and he has a good history of being very reasonable with what he watches online.  We have had a lot of discussions about this.  He was incensed because he could not watch some of the YouTube gamer videos he likes to watch.  Given the age difference, and his generally good history of managing what he watches well – I turned YouTube Restrictive Mode off for him.  However, I reminded him we could and would monitor his activities online.  Him knowing this encourages reasonable use.

The only other issue we have run into since we installed Circle about a month ago was with a site my son uses for school.  Circle was blocking it.  I simply went in and added the site domain to his custom filter and allowed it – problem solved – all from my phone within a few seconds.

Circle App – Per Profile – Custom Filter Screen

Unexpected Benefits: One other issue this has solved for us is device use near bedtime.  We have times of day set (different on school days and weekend) where the kids internet access just turns off.  The kids are welcome to read, draw, whatever but screen time is over for the day.  Previously, each night we would tell the kids “Time for bed…” and they would reply with “Four more minutes please!” or something like that.  Now they know – at 8:45PM on school nights the devices just stop being able to get online, so they plan accordingly.  Bedtimes around here have gotten a bit easier.

Circle App – Per Profile – Bedtime Screen

We also have a setting in place that only allows the kids to be online one hour per day – unless my wife or I add more time by hitting the reward button.  To be honest – we hit that button quite a lot.  The way we use this currently is to make sure they do their school work etc before we add more time.  So – they can come home from school and chill online for an hour if they want – then it’s time for them to get homework etc done before they ask us for more time.

Circle App – Per Profile – Time Limits Screen

Summary: The end result of deploying Circle has been nearly ideal.  The kids are protected from a lot of content that is not appropriate for them.  In addition my wife and I have dramatically better control of how they use the internet in general.  We are also well positioned to manage this in the future.  As they get older – we can easily open this up further and further.

Caveat Emptor: I have no illusion that we are perfect parents.  Perfect parents would have managed this whole thing better from the beginning.  My goal here has been to be transparent and share what I’ve learned in the hopes that sharing this will help others.  What works for our family may not work well for your family.  I totally get that.  However, I will humbly suggest that if you are a parent who believes that its part of your responsibility to manage your kids internet use, that this would be a great tool for you to consider.

Oh yeah – one last thing: You know what my little girl sees when she goes to hate speech sites now?  This:


To the team of engineers and developers who built and manage the Circle platform: You all are awesome!  Great work!  Thank you!

Automating Google Cloud Platform Snapshots with PowerShell

As part of a project I am working on, I needed a way to automate disk snapshot creation and retention on Google Cloud Platform.  I found several good examples of how to do this in BASH scripts etc.  However, I was unable to find anything in native PowerShell that I liked.  So, I wrote some scripts and decided to publish them here and on GitHub in the hope of helping someone else.

  • create-snaps.ps1 automates snapshot creation.
  • remove-snaps.ps1 automates snapshot cleanup based on the age of the snapshot.

Obviously, you will need to schedule these to run at some interval.  I’ve used Windows task scheduler for that purpose, and it seems to work great.

Obvious Warning: Please use these scripts at your own risk. I accept no responsibility for your use of them. However, if you run into any issues please let me know so I can work to improve them.


# Set the path and file name for PowerShell transcripts (logs) to be written to.
$LogPath = "c:\logs\powershell\snaps\"
$LogFile = Get-Date -Format FileDateTimeUniversal
$TranscriptFileName = $LogPath + $LogFile +".txt"

# Start the transcript.
Start-Transcript -Path $TranscriptFileName

#Set the GCP project.
$Project = "put-your-gcp-project-here-12345"

#Set the zone(s) where the disks are that you would like to take snapshots of.
$Zones = "us-east1-d", "us-central1-c"

#Record the date that the snapshots started.
$StartTime = Get-Date

#Go snapshot all of the disks in the zones identified above.
foreach ($Zone in $Zones) {
$DisksInZone = Get-GceDisk -Project $Project -zone $Zone | foreach { $_.Name }

foreach ($Disk in $DisksInZone) {
Write-Host "=========================================="
Write-Host "$Zone "-" $Disk"
Write-Host "=========================================="
Add-GceSnapshot -project $Project -zone $Zone $Disk #In the future we could clean this output up a bit.

#Record the date that the snapshots ended.
$EndTime = Get-Date

#Print out the start and end times.
Write-Host "=========================================="
Write-Host "Started at:" $StartTime
Write-Host "Ended at:" $EndTime
Write-Host "=========================================="

#Stope the transcript.

#Send the PowerShell transcript (log) by email. You can delete this entire section if you don't want log copies delivered by email.
#Google Cloud Platform blocks direct outbound mail on port 25. Reference:

#Mail Server Settings
$smtpServer = ""
$smtpPort = "2525" #Don't put 25 here it will not work. See link above.

$att = new-object Net.Mail.Attachment($TranscriptFileName)
$msg = new-object Net.Mail.MailMessage
$smtp = new-object Net.Mail.SmtpClient($smtpServer, $smtpPort)

# Set the email from / to / subject / body / etc here:
$msg.From = ""
$msg.Subject = "GCP Snapshot Report"
$msg.Body = "Please see the attached PowerShell transcript."

# Attach the log and ship it.


# Set the path and file name for PowerShell transcripts (logs) to be written to.
$LogPath = "c:\logs\powershell\snaps\"
$LogFile = Get-Date -Format FileDateTimeUniversal
$TranscriptFileName = $LogPath + $LogFile +".txt"

# Start the transcript.
Start-Transcript -Path $TranscriptFileName

#Set the project.
$Project = "put-your-gcp-project-here-12345"

#Record the date / time that the snapshot cleanup started.
$StartTime = Get-Date

#Choose what snaps to remove. Essentially, the script takes the current date / time subtracts 30 days and sets a variable ($deletable). Is delatable even a word? Anyway... Any snaps older than that variable get removed. Obviously, you could tweak this number of days to fit your needs.
$deleteable = (Get-Date).AddDays(-30)

#Log what date and time we set $deleteable to.
Write-Host "Deleting snapshots older than:" $deleteable

#Delete the actual snaps.
$snapshots = Get-GceSnapshot
foreach ($snapshot in $snapshots) {
$snapshotdate = get-date $snapshot.CreationTimestamp
if ($snapshotdate -lt $deleteable) {
Write-Host Removing snapshot: $snapshot.Name
Remove-GceSnapshot $snapshot.Name

#Record the date / time that the snapshot cleanup ended.
$EndTime = Get-Date

#Print out the start and end times.
Write-Host "=========================================="
Write-Host "Started at:" $StartTime
Write-Host "Ended at:" $EndTime
Write-Host "=========================================="

#Stope the transcript.

#Send the PowerShell transcript (log) by email. You can delete this entire section if you don't want log copies delivered by email.
#Google Cloud Platform blocks direct outbound mail on port 25. Reference:

#Mail Server Settings
$smtpServer = ""
$smtpPort = "2525" #Don't put 25 here - it will not work. See link above.

$att = new-object Net.Mail.Attachment($TranscriptFileName)
$msg = new-object Net.Mail.MailMessage
$smtp = new-object Net.Mail.SmtpClient($smtpServer, $smtpPort)

# Set the email from / to / subject / body / etc here:
$msg.From = ""
$msg.Subject = "GCP Snapshot Cleanup Report"
$msg.Body = "Please see the attached PowerShell transcript."

# Attach the log and ship it.

Do what you love.

Disclaimer: I have no idea if this post will ever be helpful at all to anyone else.  Sometimes writing something like this helps me make sense of my own thoughts.  Since I wrote it down, I thought I would share it.

A commonly repeated bit of wisdom is that it is a wise career choice to “Do what you love.”  I’ve heard this advice for years and at first glance, it seems good.  However, as you really dig into it, it can be difficult to figure out what it is that you really love.  What would it look like to actually do what you love for a living?

Is this what I really love?

I love to go to my Aunt’s lake house and ride the SeaDoo around the lake.  It’s one of my favorite leisure activities.  Should I do that for a living?  Is that what I love?  Thoughts like this go through my brain…  Perhaps I can make a living riding a SeaDoo.  Perhaps I could become a SeaDoo racer.  Do what you love – right!  YOLO!

Hang on a moment Mr. YOLO man.  Let’s pause and count the potential cost of doing that.  I might need to move away from my extended family to some warmer climate where you could do this year-round.  I might need to exercise a lot to get in great physical shape to race competitively.  It’s nearly impossible I would be able to earn much of a living doing this in the beginning, so for several years I would need to train to be a SeaDoo racer and work at another job to support my family.  My free time and a lot of my family time would be totally consumed by this.  Hmm.  This is sounding less awesome very quickly.

Once you look at what you think you love in the bright light of reality, the picture changes a bit.  I do enjoy riding the SeaDoo.  However, I don’t love it nearly enough to make all of the sacrifices I would need to make in order to make an actual career out of it.

Now what?  Give up?  Head back to the proverbial salt mine to spend the rest of my days doing something I really don’t love or perhaps even hate?  Nope.  Dig deeper. Here are a couple of things I have observed recently that have made me think differently about this topic.

Observation #1 – Olympic Swimmers

Recently we watched some of the 2016 Summer Olympics.  I was amazed watching the swimmers.  Think about what they did to even make it into that Olympic pool.  They exercised like crazy.  They ate healthy – probably extremely healthy.  The practiced over and over again, nearly perfectly, day after day for years upon years.  They sacrificed a lot of big things in their life only to make it into the Olympics.

Watching them, I sat and thought at first, “Wow.  That is cool. They sure are fast!”  Then I thought a bit deeper about what being that fast must have required and I thought: “You know what – these people are crazy!  Why spend so much of your life for so many years on end to become THAT good of a swimmer?  Who cares!”

I for one frankly do not care nearly enough about swimming to do that.  I would not be willing to invest even a small fraction of the effort that the person who came in dead last must have invested, even if you could assure me that by doing so I could be an Olympic gold medal swimmer.  It’s simply not something I care that much about.  Those swimmers must really love something about swimming.  They have paid a tremendous price to get to this point.  I don’t think it is a price anyone would pay, if they did not love it.

Observation #2 – Wonky I.T. Security Topics

Last week, somehow I came across something that peaked my interest in an I.T. security book called the Art of Memory Forensics.  So, I paid ~$50 and ordered it from Amazon.  It came in on Friday and I proceeded to give it and the great tool that it is written about a ton of my weekend free time.

Why?  Because memory forensics is awesome!  Well – at least I think it’s awesome.  It’s a tool that can help me do something I actually love even better.  It open up another possible angle to attack the problems that I wake up thinking about from.  It is a small piece of a grander puzzle that has had me fascinated for years.

If most of you were to read this same 858 page book, you would hate it.  You would be bored to death.  You would not be willing to invest a fraction of the time that I will happily invest on this topic.  You would probably not do it even if you could become one of the best memory forensics people in the world.  You would nearly die of boredom or confusion or perhaps both during the first few hours.  Why?  Because most of you don’t care at all about this topic.  You don’t care so much that I bet your brain would nearly refuse to focus on this for long enough to really learn much about it.  You don’t care about memory forensics in the same way I don’t care about swimming fast.

What I have learned

Why do I love topics related to Computer Security?  Honestly, I have no idea.  I just do.  Perhaps it is just what God put me on this Earth to do.

Looking back over my life, I can see a clear interest in this topic all the way back to when I was a kid.  I remember one year, my family was at the beach and somehow I ended up reading a bunch of Tom Clancy books.  Perhaps they happened to be at the house we rented for the week.  Spy stuff, military stuff, tapping undersea cables to gather intelligence on the bad guys – all of it seemed so fascinating!  I did not get nearly the amount of sun that my brothers and cousins did that year.

After hearing I was a Tom Clancy fan, my High School Principal pointed me to a book by a guy named James Bamford titled The Puzzle Palace.  I had never heard of the Nation Security Agency before this book, but I read every word.  Again – totally fascinating.

Fast forward my life story a bit and I ended up in an I.T. career.  As my career has progressed, I’ve always gravitated to areas that fit within the broad categories of Information Assurance / Information Security.  I love configuring firewalls.  Seeing an IPS alert on a blocked attack is an actual thrill for me.  I love well planned and well configured backup systems.  Quickly restoring data that was destroyed by a crypto ransomware attack knowing that the capability to do that mean the criminal will not get a dime of my client’s money makes me happy on the inside.  I imagine I love these things as much as those crazy swimmers love swimming.  It’s just IN me.  It’s what I actually love doing.

Could I do something else?  Sure, but I might not enjoy it enough to get really good at it.  For me, this is the area that is so fascinating that I will willingly invest my free time and personal money to learn even more.  For me this is a marker.  It’s a hint.  It’s an indicator.  It’s a pointer that points to what I must really love.

My $.02 worth of advice for you if you are wondering about what you really love.

Sometimes people struggle to figure out what they love.  If you can’t figure out what it is that you love, look at your life and ask yourself this question:  What is it that I am so fascinated with that I will happily spend my free time and my own money learning more about?  For some of you it is music.  For others, it’s real estate.  Perhaps for some of you it is cooking.  For others it’s helping hurting people put their lives back together.

Look for patterns.  If you’ve been fascinated with something for years and you’ve spent your own money and your own free time to learn more about it and/or do more of it – pay attention.  That might be your thing.

If you think you might want to make a career out of it, pause and run it through the sacrifice filter first.  Ask yourself “Do I really love this enough to sacrifice what would be required to become good enough at this to make a living out of it?”  Be warned – the sacrifice required will probably be even higher than you expect upfront.

If you are not willing to sacrifice enough to make a career out of something, that’s ok.  Perhaps whatever it is can still be a great long term interest for you.  For me it has been helpful to have thought of some things and then intentionally set them aside as career options.  Doing this frees you up just to enjoy them as interests without getting stuck thinking about a career move that you know deep down you are not willing to actually make.  Set them aside and over time move on to the next thing you identify.  Rinse and repeat.  Eventually, you might hit on the thing you really enjoy where the cost to actually do it fits with what you are willing to sacrifice.

Back to me for a moment…

Do I do exactly what I love 100% of my work time.  Nope.  I honestly doubt anyone does. However, I get to do enough of it that the sacrifices are worth it.

If I’m lucky, I’ll make it to my Aunt’s place this weekend to ride around on the SeaDoo.  However, when I get home and clean up, I’ll probably be thinking about some  I.T. security related topic while I’m in the shower.

If anyone reads this far – I’ll be amazed.  If you do, I sure hope you can find and do work you love as a career too.  If you want to read more on this topic, here are a few links to folks who have shaped my thinking on this.

Windows 10 – Unintentional Upgrades

In the last week or two I have gotten a significant number of calls from clients who have had PCs unintentionally upgraded to Windows 10.  While I generally like Windows 10, I do not believe that Microsoft should be doing what they are doing and essentially upgrading people automatically by using deceptive practices.  So far, I have not seen this happen on machines that were joined to a Windows Active Directory domain.  However, I have now seen it 10+ times on machines not joined to a domain.  Here is what I think you should know.
Microsoft is misbehaving:
If you receive the free Windows 10 upgrade notification and click the X, rather than simply closing the upgrade offer app, Microsoft considers this your acceptance of the upgrade and scheduled the upgrade.  This is absurd and inexcusable.  Here is an in depth story about this.
How to prevent an upgrade from happening if you wish to stay on Windows 7 / 8 / 8.1 etc:
Simply download and run the Never10 app.  A good description of how to do this can be found here:
How to roll your PC back to Windows 7 / 8 / 8.1 if you have been unintentionally upgraded to Windows 10 against your wishes.
Fortunately, this is very simple and so far seems very reliable.  Here are directions directly from Microsoft.
I hope this is helpful to you.

Google Cloud Platform (GCP) vs Amazon Web Services (AWS) vs Microsoft Azure – Cloud IaaS – Price Comparison

I’ve decided to share some public cloud Infrastructure as a Service (IaaS) compute instance cost analysis that I recently created as part of a project for one of my clients.  When choosing an IaaS provider there are obviously many things to consider beyond just compute instance pricing.  Other factors such as storage, network bandwidth, snapshot and replica options and many other factors (and costs) come into play.  Each of these providers offers many different services that may be of differing value to potential customers.

Conclusions (up front for you TL/DR folks):

  • The commonly accepted wisdom is that these providers are locked in a price war and that they have all closely matched each others pricing.  Nothing could be further from the truth.  Instances from Microsoft Azure are dramatically more expensive that Amazon Web Services and Google Cloud Platform no matter how you slice the data.  Google Cloud Platform and Amazon Web Service pricing looks close if you compare total three year costs.  However, how you get those numbers to be close (write AWS big checks upfront) is dramatically different.
  • Based on the numbers we chose for cost of capital (5%) and likely future IaaS price cuts (15% /yr), AWS does in many cases offer the lowest cost three year option IF you are willing to pay substantial amounts upfront.
  • Google Cloud Platform offers extremely competitive pricing with no upfront purchase needed.
  • Windows is expensive.  In some cases the cost difference in a Linux instance and a Windows instance exceeds the cost of the Linux instance itself.  Think about that for a moment.  The cost of your OS choice can more than double the cost of your instance.  I love Microsoft.  I love Windows.  I hope this changes.

Update – 12/6/2016 – A Microsoft rep posted this comment on my LinkedIn post of this article.  Keep this in mind as you compare prices.

If potential Azure customers talk to their local Microsoft sales rep they can chose to buy via a so called “Compute Pre Purchase” option. It will give you up to >45% savings for modern compute instances depending on the location and instance family. You need to decide for a location, instance type and pay for one year upfront but still might be appropriate for many use cases. Microsoft will very soon offer an easier way to leverage those savings and offer more options as well as longer term periods, etc. very soon.


In order to simplify some of the discussion for this purpose of this post, we’ve made the following assumptions.

  • We will look at only four similar instance sizes.
  • We will not consider storage, bandwidth or other costs.  Perhaps that will be a discussion for another post.
  • We will look at the cost difference between running Linux and Windows instances.
  • We will consider and attempt to model the different purchase options available from each provider.
  • We will compare the costs for running these compute instances for both one and three year terms.
  • We will assume 100% sustained use during the entire period considered.

Instance sizes:

  • Small – At least 1 CPU core / ~4GB of RAM
    • Specific instances we chose to compare: AWS: t2.medium / GCP: n1-standard-1 / Azure: GP A2
  • Medium – At least 2 CPU cores / ~8GB of RAM
    • Specific instances we chose to compare: m4.large / GCP: n1-standard-2 / Azure: GP A5
  • Large – At least 4 CPU cores / ~16GB of RAM
    • Specific instances we chose to compare: AWS: m4.xlarge / GCP: n1-standard-4 / Azure: GP A6
  • Extra Large – At least 8 CPU cores / ~30GB of RAM
    • Specific instances we chose to compare: AWS: m4.2xlarge / GCP: n1-standard-8 / Azure: GP A7

There is no perfect way to compare things that are not identical.  So, we have chosen what we believe to be fairly similar instance types to compare.

Provider Pricing Model Discounts:

Each provider offers ways to purchase instances in order to save some money.

Amazon Web Services: AWS offers a variety of purchase options.  These options can result in significant savings.  Explaining how reserved instances work is beyond the scope of this article.  For more detail on this topic go here: .  In general, the longer term you are willing to commit to and the more you are willing to pay upfront, the higher the discount you can get.

Azure: Microsoft Azure offers a flat 5% discount if you are willing to pre-pay for 12 months of service upfront.  The 5% Microsoft discount is frankly not very enticing compared to the significant discounts you can get from AWS for prepayment, and compared to the discounts you get from Google for simply using instances on a sustained basis.  Since a three year upfront purchase is not possible, when we modeled Azure three year costs we did so by estimating the cost of three annual purchases.

Google Cloud Platform: Google offers great discounts for sustained use.  You don’t need to pre-purchase anything, you get the discounts automatically.  The discounts are very substantial.

Cost of capital:

For purposes of this post we also wanted to consider the cost of capital.  It is also not reasonable to compare spending a large sum of money upfront with spending no money upfront and simply paying for what you use on an ongoing basis.  So, for purposes of this discussion we are going to assign a relatively arbitrary 5% annual cost of capital to options where prepayments are considered.

Expected future IaaS price reductions:

The costs of public cloud IaaS continue to drop.  For purposes of these calculations, when we look at one year costs we will assume that no price drops will happen during the middle of our one year term.  For purposes of our three year estimates, we will assume that a 15% price reduction will happen at the end of year one, and another 15% price reduction will happen at the end of year two.   Obviously, these are best guess estimates and we could easily be wrong.

Shameless Plug:

If your business needs help figuring out how to best architect public cloud infrastructure, we would love to help.

Raw data:

Linux – 1 Year

Note: A 5% cost of capital has been used for these calculations where an upfront purchase was required.





Linux – 3 Years

Note: A 5% cost of capital has been used for these calculations where an upfront purchase was required.  A 15% annual cost reduction has been estimated.





Windows – 1 Year

Note: A 5% cost of capital has been used for these calculations where an upfront purchase was required.





Windows – 3 Years

Note: A 5% cost of capital has been used for these calculations where an upfront purchase was required.  A 15% annual cost reduction has been estimated.





If you wake up and see IPs you support routing to China, it’s going be a rough day.

If you wake up and see IPs you support routing to China, it’s going to be a rough day.  Today – was a rough day.

  • At 4:35AM EDT my network monitoring system alarmed that a clients site-to-site VPN connection was down between the clients office in NC, and our data center in Atlanta, GA.
  • At ~ 6:15AM EDT I woke up and saw the alarm.  I immediately begin testing / collecting data.  It quickly became obvious that this was a routing issue.  Connectivity between some networks (Road Runner and several others) to our clients data center IPs was broken.  Curiously – traffic from Road Runner / Time Warner Cable was routing out to a router in Los Angeles, CA then dying.
  • In order to open trouble tickets for a routing issue, you need trace routes.  So I collected several showing networks that worked and ones that did not – in both directions.  Then I opened tickets with Road Runner / Time Warner Cable (the clients ISP) and the data center (who provides us IPs as part of a BGP mix of bandwidth they maintain and optimize).
  • After some additional troubleshooting while waiting to hear back on my trouble tickets, I noticed that a new BGP advertisement which included our IPs was published at nearly the exact same time that the site-to-site VPN failed.  I’ve sanitized the screen shot to protect the innocent (my client) and the guilty (a Chinese ISP).  The red blocks contain IP details I’ve intentionally removed.
  • After some troubleshooting we were able to determine that a Chinese ISP had published a bogus BGP advertisement. The Chinese ISP wrongly advertising the a /20 block of IPs (which included some of ours).  They actually own a /20 that was one character different from the block they advertised.  It appears they simply made a typo somewhere and caused all of this.
  • Our data center NOC team reached out to the Chinese ISP NOC to see if they could get them to remove this wrong advertisement.
  • At 10:25AM EDT our monitoring system recorded the site-to-site VPN coming back online.
  • When I arrived at the client site (where I was scheduled to be today anyway) – I tested and the bogus BGP advertisement had been removed.

So – what is the take away from this?  What can be learned?  Here are a few things – several of which I knew intellectually previously and I know at more of a gut level now.

  • False BGP advertisements can create a real mess.  I knew this previously – but it never impacted me as harshly as it did today.  Want to read more on how bad this can be – check out the BGPMON blog here:
  • It seems some ISPs filter or manage BGP more carefully than others.  For example – Level 3 never seemed to be effected by this bogus BGP update.  Time Warner / Road Runner apparently accepted it nearly immediately.  I’m no BGP guru at all – but wow improvement is needed here.
  • In the future before I open a routing issue ticket, I’ll take a look not only at trace routes, but also at BGP advertisements.  Huge thanks to Hurricane Electric for a great looking glass tool that ultimately helped me get to the bottom of this.