Book Report: Kaizen Express

20180625_150222

Kaizen Express: Fundamentals for Your Lean Journey

Themes: Kaizen, Toyota Production Systems, Lean, Efficiency

Would I recommend this: No. This was hard to follow. I don’t think this book was meant to be read through cover to cover. It seemed like a companion textbook to a class. Half of each page was in English and the other half in Japanese, a translation of the content or maybe the other way around.

Note: Any examples below are generated by me.

Key Concepts

Waste

Waste is any activity consuming resources without producing customer value. The resources can be equipment time, human work time, or raw materials. Customer value in this case is any product or service the customer is willing to pay for.

Example

Not-waste: Engineers writing code for product or code for testing the product. The product is sold to the customer and increasing or maintaining the quality of the product ensures the customer will keep paying for it.

Waste: Engineers waiting for the computer to update during the time when they could be developing code. This could be done during a time period where developers aren’t using their computers.

Muda

There are different types of waste. The Japanese term for this is “muda” and each type of waste is written in a different form of Japanese script.

Type 1( むだ): Waste within a process that can be resolved with point Kaizen or within the process by an individual. Example: Blocking software updates that could be scheduled to run off hours.

Type 2( ムダ): System level waste that needs to be addressed via changes in different components. Example: High levels of inactive time between code review and production deployment.

Type 3 (無駄): Organizational or policy level waste that needs to be addressed by management. Example: Enforced time sheet logging for all employees when there is no variation in day to day time reports.

Accumulation, Batching, or Stockpiling

Accumulation between different processes, batching in large quantities, or stockpiling products or product components leads to waste. When you have large batches, it can lead to accumulation between processes or you may end up stockpiling a lot of one resource or product in anticipation of demand. Any type of accumulation hides defects.

Example

You have a table factory and a production line creating screws for the tables. You’ve run the screw line to produce a large amount of screws so you can shut off the line when you have several hundreds of thousands and start up the table assembly line to use this accumulation or stockpile of screws. Unfortunately, you find out shortly after starting the table assembly line that one of the dies was misaligned and all the screws aren’t usable. You may have found out minutes later, days later, or months later. Not only did this waste the material in the screws, but also the time on the machines, potentially some material from the tables, and the time of the humans operating the machinery.

Jidoka

The idea of Jidoka is self-stopping machinery such that it does not need to be constantly monitored by a human. One of the common examples of this in everyday life is motion detectors. Cars, garage doors, and assembly lines use motion detection to detect anomalies in places they should be and automatically stop or create an alert.

Example

Garage doors in the past were known to crush children or animals that were caught under them and needed to be watched carefully as they lowered. These days (in North American houses), new garage door openers have pressure detection when lowering as well as detection of objects near the ground that would interfere with closing. We don’t need to watch the garage door close anymore. Similarly, with Jidoka, it frees up human work time to allow us to do other things and check back only when needed.

Just-In-Time

As named, just-in-time describes only producing items or taking action as they are needed. Recall in the “Waste” section that accumulation can hide defects. Instead of batching and stockpiling, you only create things just-in-time.

Example

Your team is expanding and you know you needs lots of engineers so you hire as many as you can in anticipation of future growth. Later on, after hiring 20 engineers, you find what you actually needed was 10 engineers, 5 data scientists, and 5 quality assurance engineers but you weren’t aware that the demand or requirements would change in the future. If you had hired “just-in-time”, you would have been able to save time and money.

Discussion

What if some things can’t be done just-in-time? Take the example of packing for travel. If you’re at home, you are fine to buy toothpaste as you need it but when you travel, you want to pack your full supply for that duration because you may not be able to find the toothpaste you are looking for. This ties into more of the philosophy of Toyota Production Systems, lean, and kaizen. This restriction should push you to find a way to build a system that can support just-in-time. In the toothpaste example, this might mean that you learn how to make your own toothpaste out of baking soda, which is more readily available internationally than your specific brand.

Efficiency

There are different types of efficiency:

  • Apparent Efficiency: You can increase production without increasing the number of operators or equipment.
  • True Efficiency: Producing the number of parts that can be sold with the minimum number of operators and equipment.

Examples

Apparent Efficiency: You are in a cookie factory. Apparent efficiency is being able to double your batch of cookies without needing more workers or bigger bowls.

True Efficiency: If you can only guarantee selling 10 cookies per hour, you operate with the minimum required operators and equipment to make 10 cookies per hour. This also applies to resources. If you only consume the flour, eggs, and other ingredients needed to produce 10 cookies per hour, you reduce waste in a system that has enough apparent efficiency to produce 20 cookies per hour.

Local vs. Total Efficiency

Efficiency should always be looked at in terms of the whole system. If you have high local efficiency, this leads to an accumulation between the efficient process and the next process downstream. As we know by now, accumulation leads to waste.

My favorite example of this is when software developers punch out code review after code review but there aren’t enough reviewers to keep up. This results in a build up of unreviewed code. The volume of code to review makes reviewers less likely to focus on individual details and more likely to miss bugs. This also increases the chances of conflicts with other local development and creates wasted time for other developers that later need to merge the changes as they come through. Perhaps counter-intuitively, this local efficiency of producing code results in less global efficiency because the high volume of code to review leads to more bugs and more time to develop for other developers.

Machine Efficiency

A machine’s operating rate is the time a machine is used in a given time frame. The operating ability of a machine is the time the machine works well when it is needed. If you optimize operating rate, you will get overproduction and accumulation. Accumulation leads to waste.

The overall machine effectiveness is the availability (rate) times the performance (rate) times the quality (rate). Translates to: is it working (performance) well (quality) when I need it (availability)?

Continuous Flow

Continuous flow shows the production process as the life-cycle of a single item through a series of steps. In contrast, there is batch processing, which sees a collection of sub-processes that produce batches of items that move to other sub-processes.

Why is this good? Consider the idea of just-in-time above: if we are able to optimize our end-to-end system to produce a single item quickly, we will be able to produce items just-in-time more effective. Further consider the idea of efficiency above: if we are able to understand the true efficiency of producing a unit and we have actual efficiency in our system, we can scale to only produce items according to demand and reduce waste.

From Flow to Pure Continuous Flow

So you’ve got a continuous flow system and you want to make it better or achieve pure continuous flow. There are a few ways to look at your continuous flow system to try and improve it further. A lot of these concepts are phrased for manufacturing but it can be translated to other applications.

  • Process Village Layout: Relocate sequential processes to be together in product families. Example: put all your assembly stations for tables close together in a table factory rather than intermixing them with chair assembly. You can see this concept in how houses are laid out. We have kitchens for food related activities and bathrooms for hygiene related activities.
  • Minimize Worker Movement: If you see workers or operators spending a lot of time walking and not producing something, you may want to consider your layout. An example of this is when bathroom layouts place sinks or wash basins far from the toilet stalls. In software, this can be physical distance, like the space between offices and meeting rooms. It can also be virtual distance. How many times do you need to SSH and enter your password through various redirects to find the document you need to review? That could be minimized with dashboards or single-sign on (SSO) solutions.

Pull vs. Push

When it comes to pull vs. push, it helps me to visualize a rope with markers every few inches or centimeters. If you are in a tug-of-war style situation where there are people pulling every few meters or feet on the rope, how do you make sure as many markers make it to the end of the line of people as possible? If you push the rope onto the next person, it will bunch up and only create movement where you are. If you pull the rope from the person down the line, it will not only pull your section but also the sections all the way down the line to the beginning. So, if you want to get a steady stream of markers making it to the end of the line, should you be pushing or pulling?

Pulling means you are starting a processes based on a signal from the “end of the line” or the customer. When you are pushing, you are pushing what you produce downstream and eventually onto the customer whether they want it or not. If they don’t want the product at the rate you are producing, you end up with overproduction and accumulation. I think at this point if you’ve been reading sequentially, you know what that means: waste!

The 5 Ss

The 5 Ss are a guide to organizing the work space for maximum efficiency and effectiveness.

  • Seiri: Separate needed from unneeded.
  • Seitan: Arrange in order of consumption.
  • Seiso: Keep clean and inspect regularly.
  • Seiketsu: Don’t clutter or litter.
  • Sitsuke: Sustain and maintain discipline.

The Wikipedia page explains this much more clearly than the book did. I’ll provide some examples in the context of a software development process following sprints:

  • Seiri or Sorting: Only consider work or bugs that can be accomplished within a sprint and only work that is ready to be picked up.
  • Seitan or Order and Simplify: Prioritize and ensure all items are understandable and as small as they can be.
  • Seiso or Shine: Regularly revisit your sprint tasks to ensure they are still relevant and have been updated to reflect current state.
  • Seiketsu or Standardize: Ensure all individuals are participating in maintaining the first 3 Ss and have daily stand-ups to go through the exercise of maintaining your sprint work.
  • Sitsuke or Sustain: Continue to follow and maintain regular processes on the team such as spring planning, stand-up, and retrospective.

Kanban

Kanban is a tool for visualizing a pull system. It is not synonymous with Toyota Production System. The way kanban is used in production systems is a way to signal that work needs to be done and each signal or kanban card contains instructions and details needed to complete the work order. It doesn’t always need to be a card but it does need to indicate a “pull” is needed and should contain just the information needed, not more.

Rules of Kanban:

  • Processes should be triggered by customer action and a pull from the customer side.
  • Suppliers only produce what is requested on the card and not more.
  • No stock or items should move in the system unless they are accompanied by a kanban card.
  • Defects should not make it downstream.
  • Limit the number of kanban in progress to lower stored inventory and find problems more quickly.
  • Each item should not pass through the hands of the same person more than twice.

Heijunka

The idea of heijunka is to target producing product quantities in shippable units at each stage. In batching models, you might product all components of type A, then all of type B, and then pack them up. As you know, accumulation causes waste and hides defects. Heijunka will have the effect of shorter lead times, which helps with just-in-time production, and smaller inventories since you don’t need to stockpile as much before shipping a unit to a customer. This might seem like an obvious thing to do unless you consider that several production lines need to be converted between products. You can have a line that can do either screws or bolts but not both at the same time. This is similar to having one engineer doing both development and quality assurance.

Example: Waterfall vs. Agile

You can see this idea in waterfall vs. agile software development methodologies. In waterfall, you will do all design, then all development, then all testing. In agile, you do design, development, and testing for one deliverable unit or product and then start on the others. In waterfall, you see waste when implementation needs to change the design and a cascade of alterations that may result in a redesign of all components. In agile, by completing one unit entirely, you are able to adjust the development of other units before design and implementation has started. You just eliminated waste!

Pacemaker

You may think “oh, not the cardiac pacemaker” but it’s not so far off. The pacemaker in kaizen regulates continuous flow, as a heart regulates blood flow. All requests from the customer go through the pacemaker so it can regulate rates of demand and enforce heijunka. It is the first place in the system where heijunka can be enforced.

In the software world, the Project or Program Manager would be the pacemaker, taking all customer requests and scheduling them into a roadmap for the product or project the development team is working on. The thing that’s key in this example is that the manager will not dump all the work on the development team at once and instead will try to create a sequence that will then be given to the development team either monthly or quarterly to deliver. In an agile development setting, this would happen in sprints as well and the manager of the project/program would contribute work and assist in prioritization.

Setup Reduction

One of the driving factors for batching in some production systems is the time cost of switching equipment over to make a different part. To reduce the need to batch and accumulate (oh no! waste!), aim to reduce and shorten changeover times as much as possible. Changeover time is defined as the difference in time from the last good old item produced to the first good new item produced.

Example

You are running a catering company that makes cookies and meatballs. As such a small and specific company, you have only one working surface and you need to make sure none of the cookies or meat products touch each other. As a result, each time you change between making meatballs and cookies, you have to clean your entire surface. This will drive you to instead do all of one and then all of the other. However, as we know by now, this will lead to accumulation and potential defects. If it turns out one of your meat sections or set of eggs for your cookies were rotten, the entire set made on the shared surface with shared equipment might need to be thrown away. If the batch were smaller, this would be less waste. To reduce setup time, you can consider speeding up the cleaning process by wearing latex gloves that can be switched between batches and using parchment paper to protect the surfaces being used so it can easily be tossed and replaced.

Great, so we’re not making cookies and meatballs, we’re making software. Let’s talk about that.

Some companies have dedicated architects, dedicated development teams, and dedicated quality assurance teams. Moving a product from one to the other takes time with hand-offs, back and forth communication, scheduling of meetings, and reviews, especially in a waterfall model. This is a way to interpret the changeover time in software. How do we make this faster? What if the same person were the designer, implementer, and tester. This way, we wouldn’t pay any cost of hand-offs for switch over and testing or redesign can be done as the product develops at shorter intervals, looking more like agile. Another way to reduce the time is making the hand-offs smaller in incremental components rather than all at once, again, more agile. If there is accumulation (waste!) between the design and development or development and test phases, this is likely to hide bugs.

Andon

Andons are tools meant to visually highlight anomalies and operational status of each station. Through use of these you should be able to visually survey the work space and see if anything is going wrong. On the operator side, andons are activated simply and quickly by a button or signal cord that sometimes will stop the local production line to prevent further propagation of error.

Day to day we can see andon-like signals all over the place:

  • Grocery store check-out areas have a green light above them if they are open and are off if not. With a glance you can identify which ones are working. Associates can turn off the light if there is a problem at the register.
  • Highways with dynamic lanes show which lanes are operational and which ones are not. If an accident occurs, a lane can be shut down by using the signal to communicate to drivers not to use it.
  • Your car’s dashboard lights that tell you if your tire is flat, your oil needs changing, or if your gas tank is empty.

These are strictly andons because they are themselves a way to communicate to customers.

In software we typically see these mechanisms in operational areas with dashboards for our testing suites, environment health, or deployment health. The dashboards typically have green for “good”, red for “needs attention immediately”, and yellow for “needs attention but probably not immediately”. Further, these states are automatically detected or manually set when something wrong occurs. The faster you detect anomalies and stop the production line or rogue zombie process, the fewer defects and waste go down the line.

Maximize Human Work Time

You can get more efficiency by maximizing human work time. By watching for gaps in activity, you can find ways to creatively fill them. An example in the book shows that production lines that auto-eject boxes and wait for them to be taken allow one worker to operate two lines. By contrast, lines that need a worker to watch for the right timing to remove a product from a conveyor belt can only have one line per worker.

I don’t see this a lot in software because micromanaging at this level is typically stifling. However, I often see this when there is a “planning phase” of little development, they fill the development team time with operational improvement tasks to ensure they are keeping busy. In organizations that are more agile, there is typically an interleaving of operational and product work. That’s a different discussion though.

Zone Control

The idea of zone control is to make all defects localized and not allow them to propagate to other zones. Between zones there should be a defined inspection or quality assurance. This can be likened to running integration or health tests before deploying code to a new service host. This reduces the impact of a bug to a single host or stage instead of all versions of the system.

Standardized Work

All repeated work should be standardized and part of a cycle or defined procedure. Any work that occurs out of the cycle destroys continuous flow. Standardizing a cycle or procedure can be through written example or through enforced checks, like a machine not starting if a prerequisite is not met.

An example of how non-standardized work destroys flow is when the standard development process is skipped for a bug fix. Often when I’ve seen this done it means all tracking data for why that change was made is lost and often validation has been skipped. This inevitably leads to regressions later on and very complex bugs.

Visual Management

All components should be in plain view with their status clearly visible. The standardized process should be defined visually within this system. This visual management system should make it easy to assess status, error areas, defect rate, and performance of each area.

Kanban is a great example of a visual management system. In children’s schools, visual management systems are typically well-used to show where each student’s belonging are and what activities each student is doing. Old hotels use key hanging racks to show whether a room is occupied and if the occupant is in.

Genryo Management

Genryo management refers to how you can expand or contract your production with efficiency. It allows you to make profits and lower costs as needed. It contains a few sub-concepts:

  • Shojinka or Labor Linearity: Adding labor will increase production and decreasing labor will decrease production.
  • Capital Linearity: Create a production line such that adding more machinery will increase production and removing machinery will decrease it. This is meant to avoid the initial cost of a lot of machinery leading to over investment and potentially to overproduction (and accumulation… and waste!)

Why would this be good? Consider working on a development team of 5 people that produces 1 feature per month. If I add 5 more people, does that mean I will get 2 features per month? In practice, no, we don’t see this happening unless the two features are completely independent of each other. In an ideal world, this is how it would work. In this example, the effect of adding a new developer is more complex that just adding more work capacity. This illustrates how all the other aspects of kaizen like standardization through documentation, continuous flow, and others need to be in play for genryo management to be effective.

Examples

  • Shojinka: At the motor vehicle licensing office, adding an attendant to accept applications will result in more processed applications. Removing an attendant will decrease processed applications.
  • Capital linearity: This is often seen in scaling services with commodity hardware. Adding more hosts means more customer requests can be processed. Decreasing the number of hosts results in less.

Process Study

A process study will help you understand where you can improve. This involves listing all the steps in your process and tracking all the items that pass through it. By timing all the steps and understanding the touch points of items through the system, you can get an idea of where and how much you can improve. Process studies should be repeated until consistency in timings and performance are reached.

Employee Involvement

Employees should be motivated to connect to the flow of the production system. They will be able to contribute improvements with their expertise and familiarity that managers and designers will not see. Understand that you can develop abilities in employees that will help with improvement and growth of kaizen. Remember the discipline or sustain aspect of the 5Ss: every employee should be contributing to sustain. Hold kaizen workshops to allow any and all employees to participate regularly. Kaizen isn’t a fix once and forget solution; continually maintain kaizen and grow an improvement culture.

Glossary

Kaizen: All of this together makes up kaizen.

Takt time:  How often to produce a component based on customer demand. Takt time = (available time per shift) / (demand rate per shift). Ex. 8 hour work day / 4 code reviews per day = 2 hour takt time for completing code reviews or a developer averages completing a code review every 2 hours.

Toyota Production System (TPS): a philosophy of lean production founded by Taiichi Ohno

5 Whys: Keep asking ‘why’ until you get past the obvious problems and reach the real root cause of an issue.

Poka-Yoke: Any process or procedure should be designed to be hard to get wrong. This is a design style to reduce defects through human error. Make it hard for mistakes to happen.

Other things about this book:

  • There are tons of forms and worksheets showing how to implement some of these guidelines in the Appendices
Advertisements

An Independent Venture: Before You Go…

At this point, I’ve decided to stop working full time. I considered a few different things before starting self-employment.

Disclaimer: I haven’t fully tested this advice so stayed tuned to see if this worked or not.

Motivation

If I’m actually going to do this thing, I’ve made sure I have thought out, defensible reasons to drive me and keep going through the hard parts. It is scary to quit something stable that works well and that “everyone” tells me is the greatest thing that ever happened to me. There will be a lot to learn with surprises along the way. People before me have been successful in this and I can be too… as long as I have the fortitude and grit to keep going.

Goals

Have them. This feeds into motivation as well and can help with some unexpected decisions. By choosing work based on goals, I might actually achieve them. For example, if I want to be primarily remote, I’m going to have to keep that in mind when Some Large Tech Company offers me a wonderland of benefits, high pay, and perks… in exchange for on-site work only. Then I’m back to square one.

Examples of goals I’ve considered:

  • A flexible work schedule
  • Freedom to choose projects and clients you want to work with
  • Freedom to learn a new skill and use it on the job right away
  • Able to work in any physical location
  • Variety and excitement in your work life
  • Being able to meet a diverse group of different people regularly
  • Far more interesting tax returns

You might think, “these don’t sound like goals, they sound like tenets“. Call it want you want – “North Star”, “Tenet”, “Guiding Principle”, or “Goal” – you (or I) need to be aware of the factors that help make or break decisions with arranging work,  life, and the fact that they’re about to become the same thing… if they weren’t already.

Digression on Goal Specificity: Flexibility

To clarify the level of detail in my goals, I will go into a more thorough explanation of what flexibility as a goal looks like to me. I have achieved the goal of flexibility if:

  • I can choose the number of hours I work proportional to how much money I want to make. If I work 10 hours, I get $10; if I work 20 hours, I get $20.
  • I can shift my work hours by days, weeks or months in order to create significant gaps in my work schedule for something like travel. I can work January to March and take April to travel to Japan for cherry blossom viewing.
  • I am able to choose my location of work 80% of the time. I can work from my home, someone else’s home, or a hotel for a large duration of a project (8 days of a 10 day project).

By working towards this goal I want to find out the following things:

  • How many hours per week is ideal for me within a range of +/- 5 hours
  • What are my optimal work times during the day or week
  • Am I more productive with short bursts of concentrated work or steady streams of work over longer periods of time.

As I drill down into these ideas of what I want to accomplish, I turn it into an experiment to make me a productive human in a way the full-time workplace couldn’t. If by some ironic coincidence I find out that I’m optimal for a concentrated, non-stop, 40-hour work week year-round, I suppose I’d go back to working for large tech companies.

As with development, it helps to establish a target even if I don’t get there. It helps orient  a scale, time frame, and organizes my values so I can say ‘no’.

Money

It turns out I need money or “funding” stashed away somewhere if I want to quit my job and not work for an extended period of time. Short story: I determined how much money I spend in a given period of time (let’s go with a month) and how much time I ramp up time I need. So, if I spend $10 per month and plan a month of travel and then 3 months before earning sustained income, I get $40. There are many “rules of thumb” out there that will give values like “3 months worth of pay”.

I also considered running out of money with the goals unmet. This means finding a safety net or backup plan, like Mom’s basement. I also needed an idea of when to hit the “eject” button. I don’t wait until you have $1 left before realizing I need $500 to move my stuff to Mom’s basement. Or more realistically, pay for my internet bill to do that phone interview to get the job I now can’t afford to turn down.

There is a ton of discussion on finances out there and I’m not an expert so I won’t go into it more. If you have recommendations on your favorite finance resources, I’d love to hear about them.

As a side note, starting a low-rent apartment complex called “Mom’s Basement” just for kicks suddenly sounds like a great idea…

Brand

“What is your personal brand?” is a question I usually avoid. However, if I am selling my skills directly to clients, I need to have one and it better be the one I picked. Picking a brand usually relates to “what do you want to be known for?” or “what do you want your customers to ask you to do?”. If people keep requesting SQL database migrations even though I really want to be a Ruby on Rails developer, something is wrong with my goals and my brand. People will learn about me from word of mouth and my work portfolio. If I accept and do jobs in an area, that area will be part of my brand.

Work style also contributes to how clients and customers see you. Do I insist on fully remote? How often do I meet with customers? Do I often deal with individuals directly or through others? Am I charismatic or overly technical? Whatever I want to be seen as, I try to be intentional about how my actions form my brand. It will impact the type of work and clients I get. It’s hard to change a personal brand once it has started to morph into a monster of its own.

All Your Base

Finally, once I have motivation, goals, money, and brand more or less ironed out, I need to get going. I think of this as “establishing a base” and it consists of the following:

  • Physical work location
  • Work setup (computer, developer accounts/software, contract templates, work email, comfy chair, fancy desk, desk toys…)
  • Client pipeline and advertising
  • Methods of professional development

Honestly, I really just need the first three. I added the professional development because I will be better able to advertise and network if you are current in your area. It’s also yet another venue to get work through conferences or courses.

Another disclaimer: this is how I got into the mindset of being ready to embark on self-employment. I heard other people reach it accidentally or unintentionally. I’ve heard stories of less preparation. I’ve never heard stories of more preparation. I suppose that says more about me than anyone else.

 

Tales from the Git Keeper: No Professional Growth for You

This is a post about wanting to grow as an engineer and having my management block me from doing so.

I frequently go to tech events for networking, learning, or some mix and I’m an active member in several online tech communities as well. I came across a tech event that was being advertised in several of my groups and it turned out that it was in the same city I worked. I had just started at the company I was working at a few months before and hadn’t yet encountered conferences as part of developer growth. I was about to find out what the organization I was in thought of that.

Firstly, I thought I should make sure the content of the conference was relevant to my company in an effort to convince them to sponsor my attendance or at least let me take that day as a work day and not a vacation day. I collected the following data about my organization and company goals:

  • My company at the time had an company Town Hall describing how moving to “the cloud” (AWS in this case) was going to help them go global.
  • My organization at the time was working on reducing the overhead of devops in their service maintenance and was looking into solutions for implementing Continuous Integration and Continuous Deployment
  • The organization needs to maintain both their on-premises datacenter and new AWS clusters at the same time.
  • The organization wants to move to building a platform instead of a product.

The bold words are meant to prepare you for the next segment of my proposal.

Second, I tried to relate the content of the talk to the organization goals. These are some of the titles of the talks scheduled at the conference:

  •  What does it take to build an enterprise SaaS product?
  • I may need hybrid cloud, but where do I start?
  • DevOps in an immutable world
  • Serverless and Containers: a Match Made in the Cloud
  • Building a Notification System at Scale
  • From Infra Ops to Platform Development: T-Mobile’s Approach to Cloud
  • Does your SaaS automate enough?
  • Not Your Mother’s Cloud: Best Practices for Enterprise Hybrid Cloud – from On-prem, Cloud, Containers, and Beyond

I’ve highlighted some terms in case it wasn’t clear that just the titles of the talks directly related to what my organization was advertising to be it’s goals. If you had, for some reason, built a heuristic function to rank conferences on how relevant to your organization they were based on your organizational goals, I thought this conference would rank pretty highly.

Thirdly, I tried to quantify the cost of this so I could be transparent to the company what I was asking for. Here’s what I got:

  • Cost at the time: $345 for a single ticket
  • Duration: 1 day
  • Location: Across the street kitty-corner to my office building (I could see the conference building from my desk if I chose to stand up)

As a finishing touch, I noted that my company was listed as one of the Gold Sponsors of the event. That’s right. I was asking to attend a nearby event at relatively low cost to learn about technology related to the goals of my organization that was sponsored by my company. Let’s see how that went.

[Instant Messaging]

Me to direct manager (DM): Hi DM, I’m really interested in this tech conference that’s across the street from us and sponsored by us. You can find the info here at http://www.notareallink.com. Do you think I’d be able to either get funding to go or be able to go without taking time off work?

DM: That looks great! I think we should try to send more people to that conference and even I want to go. It looks really relevant and it’s right nextdoor. I will talk to our director to see how funding for conferences work.

That went really well. I didn’t have to negotiate or bargain at all. Unfortunately, my direct manager, DM, at the time had only started 2 weeks after me and was optimistic that such a prominent tech company would of course sponsor sending its employees to go to a conference they themselves are funding. Sadly for both of us, he was wrong.

[A week later]

DM: I brought up the conference thing at our manager strategy meeting.

Me: How did that go?

DM: Well… I tried asking about it and then our director said it wasn’t important so we’d talk about it at the next meeting.

Me: I see… well, I bought my tickets because early bird pricing was only on until 2 days ago.

DM: That’s okay, at the very least our Sr Manager said we can’t fund it but we can look the other way if you go on your own.

Me:… I’ll do that.

Okay, so that wasn’t the best news but there’s still hope that they might reimburse my tickets and possibly send a few other people, right?

[Another week later]

DM: So, I asked about the conference again…

Me: Ah… that sounds like it didn’t go well.

DM: Well, I was a little surprised by the response. The director said we would need to make a case for its relevance and we’d consider sending people but we’d have limited budget.

Me: Does he know we’re sponsoring this?

DM: He didn’t seem to care.

Me: Awesome.

While both my manager and I remained confused at the lack of investment my organization had in its developers, I carried on and made sure to give ample notice to my managers and teammates that I would be out all day for one Wednesday to go to the tech conference next door. Tickets sponsored by myself.

I thought that was it. Sure, it was disappointing that the company wasn’t supportive of professional development but at least I would be able to go and not lose a vacation day. A week before the event, I overheard a manager say we were doing a mandatory all day off-site the following week. Knowing I had sent out my notice of my absence a month before, I was sure it wouldn’t conflict with my plan to go to the conference. Just in case, I sent a reminder to my managers:

[Email]

Me: Dear Managers, The agenda for the conference is up and can be found here: http://www.notarealsite.com. Some of the topics are fairly relevant to our organization, particularly those about migration from on-premises to cloud and hybrid cloud. If there is a talk that you’d like me to attend, please let me know and I will try to make it.

Skip Manager (SM): That conference conflicts with our all day off-site that I was planning to schedule tomorrow. Do you really have to go to this conference or can you go to another one?

Me: Given that this one is very close (across the street), only one day long where others on this topic are typically 3 days long, and I’ve already paid for it, yes, I will have to insist on going.

[Time Passes]

SM: I’ve rescheduled the off-site to the day after. The talk on hybrid architecture sounds good to me.

Not only was it a bit of a struggle to get people to tolerate my going to a job relevant conference, they also try to stop me from going because in favor of plans that don’t really exist yet. So much for the CEO’s proclamation that the company “attracts and develops top tech talent”.

I ended up going to the conference and enjoyed it greatly. I wrote up notes and made sure to send them out to my teammates and managers. No one cared. I didn’t go for them though so that’s fine by me. As for the offsite, it was rescheduled and delayed two more times before they eventually decided I no longer needed to attend. I’m glad they had all that hammered out before they asked me to cancel my conference attendance.

Event Review: GeekWire Cloud Tech Summit 2018

Screen Shot 2018-06-27 at 9.10.06 AM.png

Event Info

Event Location: Maydenbauer Center, Bellevue, WA

Event Cost: $350 (early bird + group) to $500 (late bird solo) USD. PyLadies and Stripe were generous enough to sponsor my attendance.

Approximate number of attendees: 300 – 500

Event Duration: 9 hours (8am to 5pm) + after party

Slides and videos

Highlights

  • Talks were short and manageable in length (30 min each) so I could move around
  • All topics were relevant and understandable
  • I learned about cloud-edge computing and more about Kubernetes
  • This was my favorite swag: Tech Tarot Cards
  • Great dedication to diversity by giving out gender pronoun pins, providing diversity activities and events
  • Excellent venue: lots of space, tables, electric outlets, and Wi-Fi

Lowlights

  • Nothing. I fully enjoyed this event.

Would I go again?

Definitely. I recommend this to anyone wanting to get breadth on current cloud topics in the Seattle area.

Details

Caution: This is a long one.

Fireside Chat with Mark Russinovich, CTO Microsoft Azure

  • New Azure features announced recently: Azure IoT Edge, Data Lake integration, New China regions
  • Azure IoT edge: a lot of computing will move out to the “edge” to optimize for bandwidth consumption and responsiveness of computed results.
  • Github: Micrsoft will not change what’s already there and fully invests in and supports the open source community
  • How are people using blockchain in non cryptocurrency applications? Non-centralized security where multiple parties can see and verify a set of actions in a distributed networks.
  • What is Quantum? It is a type of computer that can solve intractable problems in computer science that cannot be solved by a traditional computer.
  • Is Microsoft planning to invest in quantum computing? Yes, Microsoft is getting ready for it along with an SDK and Q#, a language for quantum computing.
  • Where do you think the best space to invest in technologically is? AI and ML – we’re at a breakthrough point where it’s still unknown how much we can do with it.
  • Live poll: People are very unsure about what to think of Microsoft acquiring GitHub.

Did I like the talk?

No. This seemed like a giant advertisement for Microsoft and I could have done without. The comments on block-chain and quantum compute were interesting but very much sidelines.

3 Back-to-the-Future Trends in AI Systems – Carlos Guestrin, Apple and Amazon Professor Of Machine Learning at University of Washington

  • We are in a period of resurgence in AI. Previously we saw a dip in its popularity.
  • In the 80s, AI was primarily rule based systems that were not as useful or correct.
  • Now, we are using machine learned models we are able to go into applications where we cannot enumerate all rules in spaces where there are a lot of possibilities. For example, we previously could beat humans in chess and now we are able to move to use Go without needing to describe the full space of the game.
  • Spark uses large amounts of data with low compute per datum and massively scales out. We are moving towards Deep Learning where we have huge amounts of data (where huge > large) and there is high compute per datum that requires compute grids and node scalability.
  • We moved from specialized hardware to specialized with Spark. Now with deep learning we need to move to more specialized hardware once again.
  • AutoTVM: How do you make sure you choose the right coding patterns to optimize for your hardware when training models using a tensor language over different hardware? This project is meant to learn the right parameters to maximize use of hardware and optimize efficiency.
  • We are trying to move to simpler adoption of machine learning. Currently you need a lot of expertise in order to understand data, use the tools, and manage the hardware needed for training.
  • Two trends: pre-trained models where generic data is sufficient and the need for developers to create models for their specific scenario.
  • Turi Create: allow developers to create their own domain specific data to easily train models via transfer learning from existing generic model training.
  • Another trend is moving towards making the internal workings of machine learning and AI available to everyone instead of being a black box.
  • Providing transparency into how a result was reached helps understand what features of the input are affecting the output as well as understanding where error might be coming from.
  • The data you choose to use with define the user experience.
  • Example: initially when film was being developed, they created chemical exposure guides that optimized for only light skinned models. As a result, dark skin colored people in photos were not visible due to over exposure. We need to think about the culture and values the data we select to train on shows.

Did I like this talk?

Yes, I enjoyed the examples provided and the organization of the talk. I may have already know the content but it was phrased in a organized and easily understood way.

Rook: Portable Storage on Kubernetes – Bassam Tabbara, Founder and CEO of Upbound, Inc.

  • Kubernetes can be thought of as the cluster operating system or the container orchestrator. It also does life-cycle management, scaling, load balancing, scheduling, etc.
  • It exposes a portable cloud API and thus supports the “write once, run everywhere”. You can for the most part containerize and deploy using Kubernetes and run in AWS, Azure, or other places fairly easily.
  • When we start to manage storage beyond the lifecycle of a container (or other unit), we have to look outside of Kubernetes. Similarly, different forms of storage also need to be managed: blocks, volumes, queues, key-value stores, time-series, etc.
  • Kubernetes allows portable volume extraction that is independent of pods. This is currently the only stateful workload abstraction.
  • You need to interact with storage services that exist outside of your Kubernetes managed cluster for the most part.
  • Rook is a cloud native storage orchestrator. It is an extension to Kubernetes hosted by the Cloud Native Foundation.
  • It helps storage systems run like a service on top of a Kubernetes cluster by encapsulating storage system management logic.
  • Can create a configuration that specifies size, monitors, and scaling that will create all components within the cluster that is needed using the operator pattern via Kubernetes operators.
  • Rook operator defines a desired state and loops through state activating different components until the desired state is achieved and continues to do so to maintain desired state.
  • This is an open source project looking for contributors!

Did I like the talk?

No. I understood what the problem and solution were but somehow I felt like I was missing something when he was explaining the examples. Most of them were “here is the configuration and then it just happens”.

Serverless and Kubernetes: The best of both worlds by Aparna Sinha, Google

  • They key to a serverless platform is such that not many people are needed to maintain the servers and more time can be spent on developing the application.
  • A lot of time in application development is spent on understanding monitoring, logging, orchestration, etc. that the application developer needs to spend time on.
  • What is serverless at Google? Zero ops, auto-scaling, managed security; event-driven programming model, horizontally scaling with microservices; pay for usage.
  • Serverless can be applied to more than compute including data storage and other components.
  • Two main ideas: decouple applications and automate infrastructure
  • This translates to: containerize the application + automate deployment + build in metering
  • Kubernetes is container infrastructure as a service:
    • It contains load balancing, networking, DNS and server discovery
    • Environment agnostic management
    • Has flexible, composable primitives that can be built together to craft your application. Ex. application + storage + deployments
  • Kubernetes is a platform for automating management
    • Can provide a desired number of replicas and Kubernetes will make this happen
    • Provides auto-scaling based on system state like queue length or traffic rate
  • Kubernetes is a declarative extensible API
    • You can define different components or controllers you want to use in your own application
    • Example: you could create a custom Spark controller that will allow you to treat Kubernetes as a Spark orchestrator
  • Demo: the demo goes through creating an extension deploy a function via Kubernetes in such a way that makes the extension appear as part of the Kubernetes API. Using that function, write a hello.py function and deploy it via Kubernetes.

Did I like the talk?

Yes. I finally know what Kubernetes does (you’d think I would have looked it up by now). However, having worked with these types of orchestrators before it wasn’t too hard to figure out.

From Infra Ops to Platform Development: T-Mobile‘s Approach to Cloud – Nicholas Criss, Sr. Manager, Cloud Center of Excellence, T-Mobile

  • Background: T-Mobile aims to be a market disrupting mobile carrier that strongly favors magenta the color
  • Moving from waterfall development to agile development
  • In 6 months moved up from last to first in JD Power survey
  • Moved to the cloud and didn’t just follow “lift and shift” but tried to adopt the new philosophy of cloud development to make sure they didn’t fall behind competition
  • Having already moved to the cloud, the company was seeing that they were able to adjust very quickly to sudden demands from business needs. Example: They were able to spin up tons of video processing servers very quickly and scale down as needed. Without being on the cloud it wouldn’t have been possible.
  • Migrated through different stages:
    • Before cloud: no cloud integration
    • AWS console management: Manual management of cloud infrastructure
    • Some automation of cloud infrastructure
    • Managing infrastructure as code: all infrastructure was automated in code
    • Software Developers: The software developers can easily (with minimal error) leverage fully automated infrastructure and eliminate the need for infrastructure teams
    • Platforms: We got tired of having to relearn through repeating the same mistakes on different teams – make all these learnings a part of the platform. For example: scaling, fault tolerance, and general operational readiness.
  • We moved from “we’re mostly security, compliance, and operations” to “we are software developers”
  • Build our compliance and operations into code rather than Sharepoint docs that no one will ever read
  • If you can’t implement something as code, it won’t happen or will happen incorrectly
  • No more projects, only products – focus on building customer focused value instead of meeting deadlines and completing tasks
  • Moved from being a telecom company that sometimes has interesting technology to a software company that happens to build telecommunications software
  • Platforms that T-Mobile built internally to make software development better: Security management, policy enforcement platform, enhanced Hashicorp Vault service that optimizes for T-Mobile integration, enhanced container management platform, deployment platform
  • Building out platforms to take away the painful parts of developing makes people more excited to develop and come to work
  • They see these growth points as ways to interact with and give back to the community by open sourcing their projects
  • Question: How did T-Mobile make this happen instead of just talk and no management support?
    • From top level, management is supporting training and education
    • From the bottom, there is the ability to innovate and remove pain by moving to a continuous improvement culture. There is a push to adopt new things to make old pain go away.
    • There are still some holdovers of the old model with devops teams continuing to do deployments but there is progress and benefits being seen.
  • Question: How do you get started by targeting small things to start building interest?
    • Break off small pieces and build up the great infrastructure
    • However, speaker encourages moving directly to going fully to containers and platform and don’t waste time through the in betweens
  • Question: How do you migrate people to your platforms without the additional cost of making it work?
    • We don’t force people to use a particular solution but instead we encourage them to use our solutions where they make sense
    • It’s a continuous journey to make this work

Did I like the talk?

Yes. It was a rough start with some of the disjointed examples but it improved a lot by highlighting the journey steps and the growth that happened each time.

Diversity and Inclusion Panel Sponsored by Stripe

  • Panel:
    • Uma from Stripe Infrastructure Team manager
    • Amy is a system software engineer at a local startup
    • Dana is a manager at Google
  • Q: How did you choose your field?
    • Uma: I was one of the “accidental tech people”. In India, my grades were not high enough for medicine and the only other option was engineering. Software was chosen because it was less intense than the hard engineering. I found that I enjoyed it and continued.
    • Amy: Tried to do pre-med and didn’t like it. Through a chat roulette interaction via Omegle, decided to try computer science!
    • Dana: Father introduced her to computers and started playing with programming and got into that way.
  • Q: What’s a problem in your field that you are excited about?
    • Uma: Since we are growing to be global we are solving a lot of problems of scale and growth.
    • Amy: Working on Kubernetes, we want to enable larger companies to be able to migrate easily to new technology.
    • Dana: We focus on talking about how many 9s we want to support. We need to push harder to have better quality because with millions of users, even 0.01% is a lot of people.
  • Q: Amy, what are some ways you can demonstrate leadership without being in a formal leadership role?
    • Amy: I am not a manager and I don’t want to be one. I still want the ability to influence others. I achieve that through deep understanding and making my knowledge available through youtube or automating and sharing.
  • Q: Dana, as you started moving into your tech lead role, what kinds of pros and cons were you thinking about? What advice was helpful?
    • Dana: Tech lead can mean a lot of things and in silicon valley it can mean a tech manager. What I considered before moving into this position was to talk to other women in a tech lead manager position at Google. I recommend you make sure to talk to other women in these positions before making this move. What I got was: the hybrid tech manager role is a challenging one at Google. You won’t be able to do everything you want to do and you need to learn to let go of some things. However, you get a broad impact. You need to be comfortable not coding and still having an impact.
  • Uma: You moved from being an engineer to manager and then from manager of engineers to manager of managers. What motivated you to do this switch?
    • Uma: First, I didn’t plan to be a manager and was offered the opportunity to try being a manager after coming back from maternity leave. My manager supported me trying it out and would keep the IC option open if it didn’t work. I found that I really liked it and wanted to keep mentoring and growing other developers. Moving to the manager of managers position was due to seeking a new challenge with a level of abstraction. However, people don’t tell you how lonely it is being a manager since you lose the casual chat of other ICs.
  • Q: When you were earlier on in your career was there support that you wished you had and how would you recommend people find it?
    • Uma: Women need to ask more for access to things. Try to share what you learn with others to help them be aware of what it out there.
    • Amy: The money that people are willing to offer you is a measure of how much they value you. Having a diverse group of people you can talk to to understand the potential range of values and opportunities that could exist. Try to find mentors or sponsors in different demographics and in different pay levels.
    • Dana: I was lucky enough to get interesting products until I moved to Google. You need to find the thing you want to work on and make a difference on your own. Look for opportunities to do something you haven’t done before.
  • Q: When it comes to negotiating, what advice do you have to figure out what to ask for?
    • Dana: I was underpaid at my last job. Checked Glassdoor at a point where I was ready to potentially leave and asked them to pay me competitively or I would go. And they paid me more. Use recruiters as an opportunity to reassess you value and gain leverage.
    • Uma: Continually research and update your data on what you should be paid. Don’t base your targets on your emotional response to the money. A lot of things are negotiable like working from home and you can negotiate for what is important for you.
    • Amy: Always be on the job market to keep collecting data on what you are valued at. Make sure to try to get a pay band so that you are aware of how much you can negotiate for and where the lower end of tradeoffs might be. If you can, be on social media and get yourself out there.
  • Q: Do you notice any behaviors that you wish under-represented groups would display more?
    • Dana: If you haven’t noticed, I’m trans and had spent some of my career as a different gender. I have been socialized to have some of that other gender’s traits and as a result I don’t back down even though I have a feminine communication style. I will call people out and stand up for myself. I would like to see more women doing that. The way people behaved towards me when I changed to female. It was harder and I had to adapt. I encourage everyone to stand up for themselves.
    • Amy: Ask to be treated equally and have the same as what your peers. Don’t change your communication style because other people don’t like it.
    • Uma: Risk-taking in a more calculated way is something I’d like to see more people doing. Build a support network so that you can take risks.
  • Audience Question: Do you think that offering training to help people learn these skills that would allow them to get an even footing culturally would help?
    • Uma: Yes, at Stripe we had a training around negotiating that really helped understand that everything can be negotiable. Further, in backgrounds where saying no to managers is hard, building 1:1 relationships that helps understand the new ways interact with peers and manager.
    • Amy: Managers should constantly be emphasizing that they are equals and not superior so that directs are comfortable say no or constructive discussion. It is really helpful that my managers communicates that I’m allowed to challenge them.
  • Audience Question: How do you thank people for treating you like an equal?
    • Dana: Thank them. It’s okay to tell them and explain to them why it matters to you.

Did I like this Panel?

Not particularly but that’s only because I’ve already heard this advice a number of times.

Fireside Panel with CIO of Providence Health Janice Newell and Charu Jain CIO at Alaska Airlines

  • Q: Can you tell us about a service story that went horribly wrong?
    • Charu: a lot of our cloud providers are on their own journey as well and even though you do everything right on your end, one day you find out that something isn’t there when you expect it to. We’ve learned to trust but verify even with some of the large providers?
    • Q: How do you verify?
    • Charu: Work with them to prove that it is working and pair on testing.
    • Q: What about a great thing?
    • Charu: We were building a cloud infrastructure to support more bookings. After moving to a cloud based solution we found that we were able to do things instantly that weren’t possible before.
    • Janice: I have a horror story from us. During the Wannacry virus, we found out that one of our cloud providers that had advertised high security and a whole bunch of other features. It turns out they didn’t actually have those features. The virus then took down voice transcription for doctors for over a month. This created a huge problem for the customers. We ended up moving all of our physicians off of the SaaS based solution to an on-prem solution. We then went back to verify and inspect all the features and fault tolerance the company said they offered and were horrified to see how terrible it was. We pushed back and had to get them to improve their systems. This is an example of the trust but verify example is similar to what Alaska had to do.
  • Q: We’ve seen a trend of a shift from Hybrid (on-prem + cloud) to an idea of multi-cloud where you use multiple cloud solutions (Azure + AWS). Are you thinking of multi-cloud and how would you use it?
    • Charu: Our strategy is stated as multi-cloud. We are currently on Azure and have multiple SaaS providers. We are trying to get better at one thing before maturing into a multi-cloud environment.
    • Janice: We are using containers to try to adopt that flexibility.
  • Q: How has cloud adoption impacted things organizationally and culturally? Devops? Team makeup between specialist, generalist, or devops?
    • Charu: We are working on a phased plan where we move one team and then take the learnings and roll that out to more teams. We are choosing teams first that have the most legacy technology that needs to be moved further. Developers feel empowered to drive these changes. We stood up automation teams to help start up the infrastructure and that has migrated to a cloud center where we help teams understand developing for the cloud.
    • Q: Has there been resistance along the way?
    • Charu: Change has always met resistance. People who have embraced it are setting the example.
  • Q: What is your approach to monitoring?
    • Charu: We are still in early stages. We are trying to make our monitoring complete but also not too noisy.
    • We need to measure to know how our services are doing.

Did I like this panel?

No, it seems more managerial/business level than technical. There was a lot of repetitive back and forth so it was hard to pull out relevant information.

Machine Learning At Scale On Microsoft Azure, Paige Bailey, Microsoft

  • Artificial Intelligence: any technique that allows computers to mimic human behavior
  • This means anything that is coded to behave as a human which started off as rule based systems
  • As the behavior becomes more complex and can’t be enumerated, you can use machine learning to let the computer learn a large space than the rules defined by a human
  • Types of Machine learning:
    • Do you have labelled data?
      • Yes – Supervised Learning
        • Categories: classification – determine what type of thing and object is. Example: KNN.
        • Quantities: regression – estimate an output based on input. Example – linear regression.

        No – Unsupervised Learning

        • These problems are harder
        • Do you want to Group Data?
          • Yes – Cluster Analysis – Example: K-means
          • No – Dimensionality reduction – Example: PCA

        No – Unsupervised LearningThese problems are harderDo you want to Group Data?

        • Yes – Cluster Analysis – Example: K-means
        • No – Dimensionality reduction – Example: PCA

        Example walkthrough: you are given a CSV with age, credit history, and employment; determine if you should give someone a loanYou can make a decision tree to classify this person into yes or no bucketsAside: Jupyter notebooks are excellent for machine learningFeature: think of this as a column in a CSVRecord: think of this as a row in a CSVDemo: showing a simple training example in a Jupyter notebook. (Author’s note: many tutorials are available online that will walk you through this)Machine learning is helping us learn without explicit rules; deep learning is now helping us identify features to learn on and extrapolate huge amounts of potential cases. Example: MagentaAfter this there were more examples and a discussion of how you can save your company money by learning to do machine learning on your own and also bump your salary due to the additional skillsGPUs are very good for training neural networksData science virtual machine using X2Go: if you don’t want to have to worry about managing your own machine learning environment, this will maintain all versions of libraries and languages to help you move faster with tons of educational Jupyter notebooks out of the box including topics like TensorFlow. They are ready, waiting, and free!

    • No – Unsupervised LearningThese problems are harderDo you want to Group Data?
      • Yes – Cluster Analysis – Example: K-means
      • No – Dimensionality reduction – Example: PCA

      Example walkthrough: you are given a CSV with age, credit history, and employment; determine if you should give someone a loanYou can make a decision tree to classify this person into yes or no bucketsAside: Jupyter notebooks are excellent for machine learningFeature: think of this as a column in a CSVRecord: think of this as a row in a CSVDemo: showing a simple training example in a Jupyter notebook. (Author’s note: many tutorials are available online that will walk you through this)Machine learning is helping us learn without explicit rules; deep learning is now helping us identify features to learn on and extrapolate huge amounts of potential cases. Example: MagentaAfter this there were more examples and a discussion of how you can save your company money by learning to do machine learning on your own and also bump your salary due to the additional skillsGPUs are very good for training neural networksData science virtual machine using X2Go: if you don’t want to have to worry about managing your own machine learning environment, this will maintain all versions of libraries and languages to help you move faster with tons of educational Jupyter notebooks out of the box including topics like TensorFlow. They are ready, waiting, and free!

  • No – Unsupervised LearningThese problems are harderDo you want to Group Data?
    • Yes – Cluster Analysis – Example: K-means
    • No – Dimensionality reduction – Example: PCA

    Example walkthrough: you are given a CSV with age, credit history, and employment; determine if you should give someone a loanYou can make a decision tree to classify this person into yes or no bucketsAside: Jupyter notebooks are excellent for machine learningFeature: think of this as a column in a CSVRecord: think of this as a row in a CSVDemo: showing a simple training example in a Jupyter notebook. (Author’s note: many tutorials are available online that will walk you through this)Machine learning is helping us learn without explicit rules; deep learning is now helping us identify features to learn on and extrapolate huge amounts of potential cases. Example: MagentaAfter this there were more examples and a discussion of how you can save your company money by learning to do machine learning on your own and also bump your salary due to the additional skillsGPUs are very good for training neural networksData science virtual machine using X2Go: if you don’t want to have to worry about managing your own machine learning environment, this will maintain all versions of libraries and languages to help you move faster with tons of educational Jupyter notebooks out of the box including topics like TensorFlow. They are ready, waiting, and free!

  • Example walkthrough: you are given a CSV with age, credit history, and employment; determine if you should give someone a loanYou can make a decision tree to classify this person into yes or no bucketsAside: Jupyter notebooks are excellent for machine learningFeature: think of this as a column in a CSVRecord: think of this as a row in a CSVDemo: showing a simple training example in a Jupyter notebook. (Author’s note: many tutorials are available online that will walk you through this)Machine learning is helping us learn without explicit rules; deep learning is now helping us identify features to learn on and extrapolate huge amounts of potential cases. Example: MagentaAfter this there were more examples and a discussion of how you can save your company money by learning to do machine learning on your own and also bump your salary due to the additional skillsGPUs are very good for training neural networksData science virtual machine using X2Go: if you don’t want to have to worry about managing your own machine learning environment, this will maintain all versions of libraries and languages to help you move faster with tons of educational Jupyter notebooks out of the box including topics like TensorFlow. They are ready, waiting, and free!

Example walkthrough: you are given a CSV with age, credit history, and employment; determine if you should give someone a loanYou can make a decision tree to classify this person into yes or no bucketsAside: Jupyter notebooks are excellent for machine learningFeature: think of this as a column in a CSVRecord: think of this as a row in a CSVDemo: showing a simple training example in a Jupyter notebook. (Author’s note: many tutorials are available online that will walk you through this)Machine learning is helping us learn without explicit rules; deep learning is now helping us identify features to learn on and extrapolate huge amounts of potential cases. Example: MagentaAfter this there were more examples and a discussion of how you can save your company money by learning to do machine learning on your own and also bump your salary due to the additional skillsGPUs are very good for training neural networksData science virtual machine using X2Go: if you don’t want to have to worry about managing your own machine learning environment, this will maintain all versions of libraries and languages to help you move faster with tons of educational Jupyter notebooks out of the box including topics like TensorFlow. They are ready, waiting, and free!

Did I like this talk?

Yes and no. I already knew all of this so it wasn’t super helpful but the presenter did a great job starting from 0 and showing people good examples in the time given.

Container and Serverless at Edge Connected to Cloud – Ying Xiong, Chief Architect, Cloud Platform, Huawei

  • What is edge cloud? Computing resources at the edge managed by the cloud owned by provider or customers with bidirectional communication between both.
  • Example scenario: security cameras where processing can be done on the camera itself and send data back to cloud
  • Challenges: resource constraints and frequent network partitioning with no public access; we also need to support edge to edge communication
  • Kubernetes on the Edge: what happens when and edge node loses the network connection? We can’t update it or support serverless on it.
  • To solve these problems:
    • We need to create robust communication between edge and cloud with high reliability and scalability
    • Build an edge resource controller that manages the edge from cloud
    • Support asynchronous metadata syncing and local storage to allow edge to operate on it’s own until connectivity is available
    • Similarly, build a component called KubeEdge that will manage the edge applications and listen to events to allow self maintenance on edge nodes
    • Use an event hub as a centralized listener to aggregate and queue events locally on the edge
  • Created a custom TCP/IP to optimize for edge-cloud communication for reliable bi-directional multiplex channels that can scale
  • Needs to support AuthN with certificates
  • This provides a Kubernetes plugin to manage and maintain edge nodes and also provides Kubernetes on the node to support app status syncing
  • KubeEdge is an optimized version of kubelet on edge for managing the edge node lifecycle and maintenance. It can work offline.
  • Not yet open source but that is the plan.

Did I like this talk?

Yes. Before going to this conference I hadn’t hear the term “edge cloud” and it turns out it is very aligned with my technical interests.

Fireside Chat with Matthew Prince, Co-founder & CEO, Cloudflare

  • When Cloudflare started we were competing with companies that were trying to find newer and better hardware to accomplish their goals; we did the opposite: we searched for ways of making the commodity hardware scale cheaply and linearly.
  • Over 150 datacenters for Cloudflare with fully owned hardware in rented spaces; ISPs are interested in moving Cloudflare closer to them since it creates a faster internet and better customer experience.
  • We have a world-wide infrastructure and we are suffering from low performance that isn’t improving from existing DNS providers due to lack of competition. We were able to create a secure and performant competitor in response.
  • Discussion on opening business to public offering, costs and profits, competitive advantages, customer successes, various marketing and business data
  • Insider threats are a big security challenge going forward
  • “Our scale is so large that attackers can’t DDOS attack”
  • Question: You made a decision to stop supporting a Neo-Nazi customer. How have you dealt with this internally?
    • “I never thought I’d know so much about Neo-Nazis”
    • “It’s lots of fun to kick Neo-Nazis off your service”
    • There’s no way for consumers to understand when they are using CloudFlare. It seems creepy if we curate what users can or cannot see.
    • We had to decide what the role of deep cloud infrastructure companies is
    • The point at which we decided to do this is when the customer started to attribute their political view to the company
    • There is still no clear answer on how to handle these cases

Did I like this talk?

No. This appeared to be a very long advertisement for Cloudflare. There were tidbits of interesting notes about network structure. Some really great quotes came out of this one.

Power Talk by Peter DeSantis Vice President, AWS Global Infrastructure and Customer Support followed by a Fireside Chat

  • At a large scale you are able to invest in things that wouldn’t be cost efficient at lower scale
  • AWS has expanded to many region and at an accelerating rate
  • What is a region? A set of availability zones?
  • What is an AZ (availability zone)? A fully isolated infrastructure with one or more datacenters, a unique power center, and a meaningful distance from each other
  • Every region has at least 3 and up to 6 AZs
  • Connectivity:
    • Between datacenters within the AZ
    • Between AZs within the region
    • Between AZs and the public internet across the region
  • Built on top of a dense fibre backbone with high redundancy than Amazon manages
  • Also creating inter-region Direct Connect that allows customer to get a fast lane between regions
  • Compute with AWS EC2: Security, Performance, and Familiarity (look and feel is like any other server)
  • Going over the history of EC2 architecture
    • First started off with customer instances on top of a hypervisor that managed security, network, and hardware
    • In the next big iteration, they moved over to Nitro infrastructure
    • Through these moves they were able to improve network latency by 20%
    • This also enabled moving to commodity hardware – except for security and some other core features
    • Could continue to look for hardware solution, use FPGA chips, or go with a custom ASIC – eventually acquired Annapurna labs to realize the last option
    • Thus equipped, security improved and allowed nearly 100% of compute resources to be available to customers on the instance. This achieved the familiarity goal.
  • Now have also opened offerings for bare metal instances as well as Nitro VMs

Did I like this talk?

Yes. Despite knowing a lot of this information at a high level, I hadn’t seen in presented in this particular nutshell. I liked how it was phrased and the progression through the topics.

Event Infrastructure

  • The event used a phone app to support planning your agenda, quick connect with other attendees, integration with sli.do for live polls
  • Regular video on projectors but also a live keyword note taking screen which was actually very helpful
  • The breakfast was great: hot toasted bagels and a hot oatmeal bar for breakfast
  • Event support for developers is great: they provide desks and tables with a high density of plugs available
  • The sponsors presented before, between, and after each talk. It was a little overwhelming and felt like I was watching Youtube with ads turned on. However, the comfort of the event was worth it. Similarly, all “ads” were relevant so it wasn’t too bad.

Networking Notes

  • Very easy to network at this event: all I needed to do was sit down at a table and people would start up conversations about technical topics
  • Despite trying to avoid doing so, people are very interested in hearing the terrible things about the company I currently work for
  • I encountered some discussion because I perhaps foolishly brought my motorcycle gear with me (I did ride the motorcycle to the event) and started some conversations about whether or not to lock your helmet to your bike. This then lead to discussions about companies that let you work from home and some nice connections. If you learn one thing from this: bring or wear an item that will inspire people to talk to you.

Book Report: Six Sigma For Dummies

I’m a huge fan of continuous improvement, the “lean enterprise” philosophy, and just making things more efficient. There are different things that draw people to software development and one of them is automating repeated and wasteful tasks in their lives. This is part of what Six Sigma is about applied to an entire organization. In this book review I won’t give all the content of the book but rather an overview of topics covered and how well I think it was communicated.

The book.

51ixq96w5sL

Summary

  • This book goes over what Six Sigma is and how it is a data driven and criteria focused methodology for improvement
  • The topic is a business and organizational topic but can be applied to any process including doing chores or planning a road trip
  • There are a lot of relatable examples that can be used to teach the topic if you’d like to share these ideas with your team, friends, or family
  • The second half of the book goes into statistics, data presentation, and data interpretation
  • Measurement is the first step to improvement
  • There are some good Lord Kelvin quotes about measurement referenced in the book
  • Overview of the DMAIC process

Recommend?

I recommend this book to those who:

  • Already have a Six Sigma leader in their organization and want to know what it’s about in a single reference
  • Want a deeper understanding of which data visualizations are relevant and which are not
  • Those who want an overview of the key components of Six Sigma (only read the first half)

Detailed overview

  • Six Sigma is about measuring inputs to your system, identifying variance, and controlling or reducing the variance in your process.
  • This can be applied to any type of process including manufacturing, software development, or hiring.
  • You must measure your all targets whether they are a quality bar (ex. physical measurement or latency) or the time it takes to produce an item.
  • Both the quality and efficiency measurements are needed along with other factors you want to improve or maintain at a certain level.
  • This is a practical and understandable use of statistical and experimental methods.
  • Focus on eliminating special cases of variance or error rather than start with the common cases. Once you eliminate special cases then any improvements from common cases will be more impactful and less overshadowed by exception scenarios of variance.
  • You need to measure all inputs and outputs along with your goal to figure out what your variance is and to see improvement over time.
  • Try to find the factors that go into your process that have the greatest impact. Improving these first will provide you with more leverage to convince the organization to change.
  • Follow the DMAIC process in order to achieve improvements.
  • There are different types of improvements or change:
    • Thinking for breakthrough – affecting cultural or thought change
    • Processing for breakthrough – improving existing processes
    • Designing for breakthrough – change or improve how new processes or products are created
    • Managing for breakthrough – change leadership and hiring to create a culture of improvement
  • Six Sigma makes everyone a leader for improvement who cuts through assumptions and fights for betterment.
  • This type of organizational change needs to be driven from top down. You may have small scale success in an isolated team but you can’t do anything on a larger scale without management driving it.
  • All aspects of an organization must be involved including non-production roles like finance or HR to define their expectations and results.

DMAIC

  • A tool to turn an everyday problem into a logical plan and solution
  1. Problem definition
  2. Identify impacted processes
  3. Create a data driven definition for the problem
  4. Create a data driven solution goal
  5. Create a verification plan for the solution
  6. Design a practical solution based on the data driven goal
  7. Implement and see results!
  • Typically improvements from this process should benefit all customers and stakeholders.
  • There are different brainstorming techniques the book goes into that can be used with different groups of people to identify problem areas that feed into the definition phase.

Identifying the problem and target:

  1. Determine what needs to be improved (or the Y)
  2. Identify the impacted projects or processes
  3. Determine a baseline for Y of what is currently observed
  4. Quantify the cost and impact of this problem
  5. Write a problem statement: improve [metric] from [baseline] to [goal] in [time] with [impact] to [business goal]
  6. Identify key individuals and build a team: the best person to lead an improvement is the owner of the process.
  7. Obtain approvals from stakeholders and launch the project.
  • If you have more than one Y to improve this may indicate your scope is too broad.
  • Initial baselines may not be accurate but will improve with more measurements over time.
  • Long term metrics are more valuable and relevant than short term or short-timespan metrics. Snapshots can be misleading and you’ll only identify trends of improvement over longer periods of time.
  • Avoid biasing the project by putting parts of the solution into the problem statement.
  • Create specific and measurable statements to convince management of the value and to ensure all parties impacted can understand the value.
  • Identify the entitlement in your process: this is the best possible performance you can achieve by adjusting the inputs you can control.
  • Work to identify hidden processes which likely represent areas for improvement.

Statistics

This book went over various concepts in detail with examples of each. I will only list them out here:

At this point in the book they started to go into different ways of plotting the data, different types of visualizations, the pros and cons of each type of visualization, how to interpret the data from each visualization and I became very bored, very quickly. I stopped reading.

An Independent Venture: So Long and Thanks for All The Fish

If you’ve ready some of the other posts on my blog like this one or this one, you might wonder “why didn’t you quit?”. Short answer: quitting is hard. I only have myself and two cats to support and there’s already a migraine inducing list of things to consider. I can only imagine how much more challenging it would be for someone with a spouse or children to support as well. For what it’s worth, I will go over considerations I have in mind when quitting, whether I have a job to jump to or not.

Health and Well-being

Vacation

There are a few things to consider about vacation if you have any left at the time you are planning to leave.

Cashing Out: Does your company pay out any remaining vacation left that you haven’t taken? If not, then you may want to take that paid vacation time before leaving.

No Extension Policy: Your company may have a policy where you cannot extend your time working for the company by taking all your remaining vacation within some number of days of you quit date. Even when I worked at a company with this policy, it wasn’t enforced.

Health Insurance

End of Coverage Date: A lot of companies pay for their employee health insurance monthly and typically you are covered from the 1st of the month. This means that if you quit on the 1st, you are covered for the full month without being employed. Unless there’s a good reason not to, try to quit after and as close to the first as possible if you need to stay in the US after quitting.

FSA: If you have a Flexible Spending Account, you will lose all the money when you quit because your health plan will change. Need a new pair of glasses or were you considering Lasik? Use up that FSA before you leave.

HSA: If you have a Health Savings Account, you can keep it open and keep the money in it. However, you may be responsible for the monthly fees to the bank. Check those out with your HSA provider to see if you’re willing to keep those up. Alternatives for this include rolling your HSA funds into an IRA. You would need to check with your HSA provider first. If you plan to leave the Unites States, you will need to pay the penalty to withdraw all the money if you don’t plan to leave it in the states.

COBRA: When you quit you’ll get a letter about COBRA. You can enroll in continued coverage with your employer’s health plan for a limited amount of time. Usually this is expensive. Plan to research this along with your other health plan options before quitting.

Fitness and Education Benefits

If you company has fitness or education benefits like paying for a gym membership, you will not only see your membership end but you may need to pay back some money. Double check the policies to see if you need to pay money back and how much. Similarly, check to see at what point your gym membership will be cancelled. Maybe you can spend more time at the gym after you quit if it continues.

Employee Assistance Program

Many more companies are offering something called an Employee Assistance Program or EAP. This offers some small number of appointments for free with a therapist or psychologist. Feel free to pass this up but there are some therapists that specialize in life transitions and career planning. If you want to chat with someone about quitting and bounce your ideas off them, a couple of free session before actually making the change might keep you more sane during it.

Records

Before losing contact with your physicians or other medical providers, make sure you have their contact information to retrieve records in the future. Similarly, request any necessary vaccines and proof of vaccination before losing health coverage. Also another potential way to drain that FSA money if you want a yellow fever vaccine, for example. Do also have a printed copy of your prescriptions to make it easier for any new physicians in any country to resume your medical care.

Money, Money, Money

You will have all kinds of money implications including but not limited to:

You can go very deeply into all of these so I will try to boil them down as much as I can:

  • If you received any bonuses, check to see if you need to pay them back and how much you need to pay back.
  • Have enough money to cover your living expenses in savings for at least 3 months. There are many resources online that can help you understand what goes into that number or if 3 months makes sense for you. Example.
  • If you have stocks awarded from your company, check what their vesting dates are and how much you will lose. You will lose all unvested stocks.
  • Opening new lines of credit may change if your company has an agreement with banks to provide discounts or other advantages to it’s employees. You may want to carefully plan for any new lines of credit you’ve just opened or plan to open to see if you benefit from using the company benefits.
  • Taxes! We love taxes! If you make a decent salary and get stock from your company, you deeply fear selling your stock due to the jump in your tax bracket for that year. The plus side of being unemployed? Lower income. Consider planning to sell stocks or otherwise liquidating assets during the time you are unemployed to keep your taxes low.

Immigration

Humans

If you have a Green Card or a visa (not the credit card), you may not know when you will stop being allowed to work in the US. For visas, it varies based on the visa type. Check your visa type and it will specify for how long you may remain in the US before you must leave and reenter. For example, H1B – you must leave within 60 days and re-enter on a visitor visa if you want to come back without another work visa.

Green cards are more interesting. You may be abroad outside of the United States for 1 year and re-enter without needing to report your absence or lose the card. You may additionally request another year to extend your absence from the US to 2 years but any longer than that and you lose the card. If you don’t request and extension, you will lose the card after 1 year away. How do they tell when you’re away? I have no idea. You can try to game this system but the consequences are extended periods of being banned from entering the US if you lose that game.

Finally, if you have dependents on your visa, they will face similar status changes and risks as you do.

Animals

If you would like to move to another country, state, or city with your animals, make sure you have copies of their vaccines and medical records before moving. If you have young animals that are still considered babies of their species, there are different import/export laws for them. Even in other states or cities they may have different banned breed lists, animal registration policies, or spay neuter policies.

Nice to work with you; let’s stay in touch

Giving Notice

Check about your company policies around resignation. Many tech companies use “at will employment” contracts which means we can quit whenever we want. However, if you want to stay in the good books of your employer, find out what the unofficial guidelines are. If you are not able to ask anyone at work, websites like Glassdoor or Blind (if you can sift through the toxicity) discuss these topics for each company.

There is a stereotype of “2 weeks notice”. Do not assume that your company follows that. Typically you want to budget enough time to tie up loose ends for work but not enough time for them to give you more work. Further, when hearing that an employee is leaving, some managers will choose to pile on the work even harder to get the most out of them before they leave. Be ready to say no if this happens to you.

Finally, and perhaps most importantly, some companies will require you to leave immediately under company escort out of the building if you disclose you are leaving to work for a competitor. Make sure you moved your office couch out before you give your notice in that case. Make some excuse about getting it reupholstered with Pokemon fabric or something similarly question blocking.

Staying in Touch

There are a few different ways of staying in touch with people. My favorite is using LinkedIn. This allows me to give my contact information out to people and still being able to control who I accept as a contact.

Using your email address is up to you but I find that people rarely reach out that way. LinkedIn is far less personal and lacks commitment enough that people are more likely to use it. Similarly, Facebook or phone numbers are also a little too personal and you might find out about that secret crush the guy near the printers has had on you and wish he didn’t just get your phone number. If you do prefer phone and want to keep your personal number private, Google Voice numbers are another way to share numbers and maintaining control over who can reach you. They are also another way to maintain contact with those in the US if you are not going to be in any particular country after quitting.

How to do the quitting

After having gone over all these really painful planning and paperwork points, what about the actual conversation with your boss? You will need to stay tuned for another blog post on that topic!

Explanation of title.

Networking Group Review: New Tech Northwest

New Tech Northwest is a tech focused networking group in the Seattle, WA area. They hold monthly networking and tech talk nights in Seattle downtown and on the Eastside. The have different ways of being connected such as the events themselves, a slack channel, and a newsletter. They advertise as being a friendly and effective networking group.

Things I like

  • This group uses Slack. It’s a great way to keep in touch with people and network without committing to a full 3 hours or more of time.
  • The website, emails, and events are well put together and you are not flooded by ads.
  • The events have diverse sponsors, talks, and people that are different every time. The booths have ranged from city councils to local consulting companies to start-ups looking for beta testers. There are also only about 3 -5 booths per event so it’s very manageable. The attendees range from developers to data scientists to marketers to “business developers”*.
  • You are guaranteed to be encouraged into speaking to at least 2 people each event. The talk starts off with: high five two people that you don’t know and tell them about something cool you are doing and ask how you can help.
  • The talks are carefully timed and allow for Q&A from attendees.
  • They have a color coded name tag system where they show who is looking for work, who is hiring, who is a sponsor, and who’s just there to hang out.
  • I get to find out about topics and companies I never would have heard of like this one.

*I don’t know what a “business developer” is but a fellow attendee was adamant that I should do it.

Things I don’t like

  • The Slack channels are pretty quiet and there’s not a lot beyond a few job postings or asks for beta testers.
  • Networking by talking to people at random is hard and uncomfortable.
  • The color scheme. I don’t know what it is about the green that bothers me.

Who would I recommend this to?

  • People who want to network at events but want to be able to decide last minute when to go (no need to pre-register)
  • People who like to grab a beer with strangers and talk about their industry (if their industry is tech)
  • If you are looking for a job and you’d like to meet a human being in person who works at that company.
  • If you are looking for a local job.

Note: The events do cost money (about $10 each) but there is no group membership fee.

Another note: If you are looking for a new job, you’d likely want to plan to attend a few of the events since they can be highly variable.