{"id":254,"date":"2022-03-17T23:01:00","date_gmt":"2022-03-17T13:01:00","guid":{"rendered":"https:\/\/sysmit.com\/cf22\/?p=254"},"modified":"2023-12-13T15:28:02","modified_gmt":"2023-12-13T05:28:02","slug":"site-reliability-engineering-culture-visual-summary-walkthrough","status":"publish","type":"post","link":"https:\/\/sysmit.com\/cf22\/site-reliability-engineering-culture-visual-summary-walkthrough\/","title":{"rendered":"Site Reliability Engineering Culture Patterns"},"content":{"rendered":"\n<div class=\"wp-block-columns has-base-2-background-color has-background is-layout-flex wp-container-core-columns-layout-1 wp-block-columns-is-layout-flex\">\n<div class=\"wp-block-column is-vertically-aligned-center is-layout-flow wp-block-column-is-layout-flow\" style=\"flex-basis:60%\"><h2 class=\"wp-block-heading\" style=\"font-size:clamp(16.834px, 1.052rem + ((1vw - 3.2px) * 0.975), 26px);font-style:normal;font-weight:600\" id=\"who-should-read-this\">Who should read this:<\/h2>\n\n\n<div class=\"wp-block-stackable-icon-list stk-block-icon-list stk-block stk-3c03516\" data-block-id=\"3c03516\"><style>.stk-3c03516 ul li,.stk-3c03516 ol li{font-size:20px !important}.stk-3c03516 li{padding-inline-start:6px !important;margin-bottom:6px !important}.stk-3c03516 ul,.stk-3c03516 ol{padding-left:40px !important}.stk-3c03516 ul li{list-style-image:url('data:image\/svg+xml;base64,PHN2ZyBhcmlhLWhpZGRlbj0idHJ1ZSIgZm9jdXNhYmxlPSJmYWxzZSIgZGF0YS1wcmVmaXg9ImZhcyIgZGF0YS1pY29uPSJ1c2VyLWNpcmNsZSIgY2xhc3M9InN2Zy1pbmxpbmUtLWZhIGZhLXVzZXItY2lyY2xlIGZhLXctMTYiIHJvbGU9ImltZyIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIiB2aWV3Qm94PSIwIDAgNDk2IDUxMiIgc3R5bGU9ImZpbGw6ICMwMDAgIWltcG9ydGFudDsgY29sb3I6ICMwMDAgIWltcG9ydGFudDt0cmFuc2Zvcm06IHJvdGF0ZShkZWcpICFpbXBvcnRhbnQ7Ij48cGF0aCBmaWxsPSIjMDAwIiBkPSJNMjQ4IDhDMTExIDggMCAxMTkgMCAyNTZzMTExIDI0OCAyNDggMjQ4IDI0OC0xMTEgMjQ4LTI0OFMzODUgOCAyNDggOHptMCA5NmM0OC42IDAgODggMzkuNCA4OCA4OHMtMzkuNCA4OC04OCA4OC04OC0zOS40LTg4LTg4IDM5LjQtODggODgtODh6bTAgMzQ0Yy01OC43IDAtMTExLjMtMjYuNi0xNDYuNS02OC4yIDE4LjgtMzUuNCA1NS42LTU5LjggOTguNS01OS44IDIuNCAwIDQuOC40IDcuMSAxLjEgMTMgNC4yIDI2LjYgNi45IDQwLjkgNi45IDE0LjMgMCAyOC0yLjcgNDAuOS02LjkgMi4zLS43IDQuNy0xLjEgNy4xLTEuMSA0Mi45IDAgNzkuNyAyNC40IDk4LjUgNTkuOEMzNTkuMyA0MjEuNCAzMDYuNyA0NDggMjQ4IDQ0OHoiIHN0cm9rZT0iIzAwMCIgc3R5bGU9ImZpbGw6IHJnYigwLCAwLCAwKTsgc3Ryb2tlOiByZ2IoMCwgMCwgMCk7Ii8+PC9zdmc+') !important}.stk-3c03516 li::marker{font-size:2em !important}@media screen and (max-width:1023px){.stk-3c03516 ul li,.stk-3c03516 ol li{font-size:20px !important}}<\/style><ul><li>Developers new to Site Reliability Engineering (SRE) who want to understand the culture <\/li><li>Current SREs who are seeking to guide others like management on key aspects of SRE culture<\/li><li>Technical leaders who want to create an ideal culture for effective software reliability practices<\/li><\/ul><\/div>\n<\/div>\n\n\n\n<div class=\"wp-block-column is-vertically-aligned-center is-layout-flow wp-block-column-is-layout-flow\" style=\"flex-basis:40%\">\n<figure class=\"wp-block-image alignright size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"823\" src=\"https:\/\/sysmit.com\/cf22\/wp-content\/uploads\/2022\/04\/sre-culture-patterns-cruform-1024x823.jpg\" alt=\"Thumbnail of SRE culture pattern visual summary\" class=\"wp-image-228\" srcset=\"https:\/\/sysmit.com\/cf22\/wp-content\/uploads\/2022\/04\/sre-culture-patterns-cruform-1024x823.jpg 1024w, https:\/\/sysmit.com\/cf22\/wp-content\/uploads\/2022\/04\/sre-culture-patterns-cruform-300x241.jpg 300w, https:\/\/sysmit.com\/cf22\/wp-content\/uploads\/2022\/04\/sre-culture-patterns-cruform-768x617.jpg 768w, https:\/\/sysmit.com\/cf22\/wp-content\/uploads\/2022\/04\/sre-culture-patterns-cruform-1536x1234.jpg 1536w, https:\/\/sysmit.com\/cf22\/wp-content\/uploads\/2022\/04\/sre-culture-patterns-cruform.jpg 1810w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><figcaption class=\"wp-element-caption\">We will explore elements of the above visual summary in this post (larger size below)<\/figcaption><\/figure>\n<\/div>\n<\/div>\n\n\n<h2 class=\"gb-headline gb-headline-4af31bd7 gb-headline-text\" id=\"introduction\">Introduction<\/h2>\n\n\n<p>Despite its now antiquated sounding name, Site Reliability Engineering (SRE) as a discipline has strong future promise to proactively improve software reliability in production.<\/p>\n\n\n\n<p><strong><em>As software complexity continues to increase, so will the need for better and better practice of SRE.<\/em><\/strong><\/p>\n\n\n\n<p>It is undoubtedly an exciting but enigmatic field, with the <em>how<\/em> of SRE being widely open to interpretation.<\/p>\n\n\n\n<blockquote class=\"wp-block-quote\">\n<p>The first episode of the&nbsp;<a href=\"https:\/\/www.linkedin.com\/company\/on-call-me-maybe-podcast\/\">On Call Me Maybe Podcast<\/a>&nbsp;[reminded] me that most SREs are beginners, we&#8217;re in unchartered territory. <\/p>\n<cite>Stephen Townshend, Developer Advocate at SquaredUp<\/cite><\/blockquote>\n\n\n\n<p>The above quote is interesting because Stephen Townshend is far from a beginner, but his sentiment makes sense. SRE is a constantly moving target.<\/p>\n\n\n\n<p>My experience has shown that many teams operate very differently from what was outlined in Google&#8217;s 2016 book dubbed <em>Site Reliability Engineering<\/em>.<\/p>\n\n\n\n<p>And they should. Because they are not Google. <\/p>\n\n\n\n<p>Instead, I propose a few culture patterns that I&#8217;ve picked out from reading 100s of posts on the topic and talking with 10s of engineers in the discipline.<\/p>\n\n\n\n<p>This post will <strong>explore 7 of the cultural patterns<\/strong> I&#8217;ve identified to form Site Reliability Engineering culture. <\/p>\n\n\n\n<p>It&#8217;s not perfect, but it will give a clearer picture to those refining their SRE culture.<\/p>\n\n\n\n<p>The complete visual summary is below, which I will break down and walk through in the rest of the post.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1810\" height=\"1454\" src=\"https:\/\/sysmit.com\/cf22\/wp-content\/uploads\/2022\/04\/sre-culture-patterns-cruform.jpg\" alt=\"Site Reliability Engineering culture patterns described in a visual summary by Ash Patel of SREpath.com\" class=\"wp-image-228\" srcset=\"https:\/\/sysmit.com\/cf22\/wp-content\/uploads\/2022\/04\/sre-culture-patterns-cruform.jpg 1810w, https:\/\/sysmit.com\/cf22\/wp-content\/uploads\/2022\/04\/sre-culture-patterns-cruform-300x241.jpg 300w, https:\/\/sysmit.com\/cf22\/wp-content\/uploads\/2022\/04\/sre-culture-patterns-cruform-1024x823.jpg 1024w, https:\/\/sysmit.com\/cf22\/wp-content\/uploads\/2022\/04\/sre-culture-patterns-cruform-768x617.jpg 768w, https:\/\/sysmit.com\/cf22\/wp-content\/uploads\/2022\/04\/sre-culture-patterns-cruform-1536x1234.jpg 1536w\" sizes=\"(max-width: 1810px) 100vw, 1810px\" \/><figcaption class=\"wp-element-caption\">A visual summary of Site Reliability Engineering culture patterns by SREpath.com<\/figcaption><\/figure>\n\n\n<h2 class=\"gb-headline gb-headline-bc3e523f gb-headline-text\" id=\"sre-culture-pattern-1-balance-competing-priorities\">SRE Culture Pattern #1: Balance competing priorities<\/h2>\n\n\n<figure class=\"wp-block-image aligncenter size-full is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/sysmit.com\/cf22\/wp-content\/uploads\/2022\/04\/sre-balances-conflicting-priorities.png\" alt=\"Site Reliability Engineering balances the need to deploy often with the need for reliable software\" class=\"wp-image-220\" width=\"383\" height=\"366\" srcset=\"https:\/\/sysmit.com\/cf22\/wp-content\/uploads\/2022\/04\/sre-balances-conflicting-priorities.png 511w, https:\/\/sysmit.com\/cf22\/wp-content\/uploads\/2022\/04\/sre-balances-conflicting-priorities-300x286.png 300w\" sizes=\"(max-width: 383px) 100vw, 383px\" \/><\/figure>\n\n\n\n<p>SRE can be considered a balancing act between two counteracting forces within the software organization.<\/p>\n\n\n\n<p>On one end, there&#8217;s the desire to deploy releases fast and often. On the other end is the need for reliable software with acceptable uptime.<\/p>\n\n\n\n<p>In traditional software contexts, neither end sees the true benefit of the other.<\/p>\n\n\n\n<p><strong>Feature developers want to release features often <\/strong>to keep the product competitive, which increases the risk of software breaking in production.<\/p>\n\n\n\n<p><strong>Operations want reliable software. <\/strong>Their true north is <em>stability, stability, stability<\/em>. This is antagonistic to how product developers see the software&#8217;s success.<\/p>\n\n\n\n<p>There&#8217;s a common quip that goes like this:<\/p>\n\n\n\n<blockquote class=\"wp-block-quote\">\n<p>If it were up to sysadmins and operations, the software would get released once and that&#8217;s it.&nbsp;<\/p>\n<\/blockquote>\n\n\n\n<p id=\"block-ae4fc1a6-5856-43d7-97f2-34648315c0de\">Examples of change include new code deployments and updates to infrastructure configs.<\/p>\n\n\n\n<p id=\"block-ad03de96-59f4-41ab-9ed7-02976d8f69c9\">Operations people have traditionally hated change because change compromises what is perfectly running software in their minds.<\/p>\n\n\n\n<p id=\"block-7070986b-8ff5-41fb-bb4e-8b71d72e4ed4\">The thinking goes that <strong>more change \u2192 more potential for errors \u2192 more pager alerts<\/strong>.<\/p>\n\n\n\n<p id=\"block-708abe56-309c-4dba-9fe5-1a79cecbd88c\">What could be more stable (and peaceful) than not making any changes to the system?<\/p>\n\n\n\n<p id=\"block-a15c0ce9-91d3-444b-8957-71c76de6cfc0\">The risk from this thinking is that deployments can become over-engineered and an onerous compliance process in some situations.<\/p>\n\n\n\n<p id=\"block-5f5e3ca5-cef1-4601-8e77-5786c61807fd\">We will explore this later on when I explain the need to <em>Empower developers<\/em>.<\/p>\n\n\n\n<p id=\"block-26fbdcbc-5469-48bd-b16e-aa7b4dae3dbd\">In conclusion, both ends have their merits. If you don&#8217;t deploy often, your software can go stale and lose its competitiveness in the market.<\/p>\n\n\n\n<p id=\"block-42708e1c-0a3a-465f-8683-e1101cc2d598\">At the same time, if you release without considering production stability, customers can open the page to 500 errors en masse.<\/p>\n\n\n\n<p id=\"block-65f694e7-8e23-4e5f-8da2-759e8ac6d97c\"><strong>SRE brings about a mindset that respects both developer and operations camps.<\/strong><\/p>\n\n\n\n<p id=\"block-aaf52a4c-e1ba-4285-a63e-5aaa09cb9347\">It supports the need of developers to ship fast and often <em>and <\/em>does the work to ensure the software will work reliably. <\/p>\n\n\n\n<p id=\"block-aaf52a4c-e1ba-4285-a63e-5aaa09cb9347\">It&#8217;s a balanced approach to deploying features into production fast but without falling apart or taking down adjacent services. The subsequent patterns allow SREs to achieve this goal with grace. <\/p>\n\n\n\n<p id=\"block-a63cb8ef-1f1f-4d4a-a4f5-288e79f43fdf\">Keep reading.<\/p>\n\n\n<h2 class=\"gb-headline gb-headline-0ac02825 gb-headline-text\" id=\"sre-culture-pattern-2-eliminate-toil\">SRE Culture Pattern #2: Eliminate toil<\/h2>\n\n\n<figure class=\"wp-block-image aligncenter size-full is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/sysmit.com\/cf22\/wp-content\/uploads\/2022\/04\/sre-culture-eliminates-toil.png\" alt=\"Site Reliability Engineers try to eliminate toil (manual work) wherever possible\" class=\"wp-image-221\" width=\"652\" height=\"383\" srcset=\"https:\/\/sysmit.com\/cf22\/wp-content\/uploads\/2022\/04\/sre-culture-eliminates-toil.png 652w, https:\/\/sysmit.com\/cf22\/wp-content\/uploads\/2022\/04\/sre-culture-eliminates-toil-300x176.png 300w\" sizes=\"(max-width: 652px) 100vw, 652px\" \/><\/figure>\n\n\n\n<p><strong>What does toil mean?<\/strong> <\/p>\n\n\n\n<p>It means doing manual, repetitive work. Let&#8217;s enhance this definition further&#8230;<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-style-large has-medium-font-size\">\n<p>Toil is boring or repetitive work that does not gain you a permanent improvement.<\/p>\n<cite>&#8211; Jennifer Mace (Macey), Google SRE<\/cite><\/blockquote>\n\n\n\n<p>Responding to incidents is <strong>not toil until it becomes a pattern<\/strong> of the same incident being responded to for the same system.&nbsp;<\/p>\n\n\n\n<p>An example of eliminating toil is automating a release process versus continuing to review releases manually.<\/p>\n\n\n\n<p>If you are <a href=\"https:\/\/www.srepath.com\/runbooks-for-better-incident-response\/\">following a runbook<\/a> and notice that you are <strong>following Step 1,2,3,4 regularly, work out how to automate it<\/strong>. Write a chron job or script.<\/p>\n\n\n\n<p>A key aspect of eliminating toil is to find out where it exists. That means getting feedback from stakeholders like developers, operations, and everyone involved in the SDLC.&nbsp;<\/p>\n\n\n\n<p>There\u2019s a saying that <strong>SREs should automate themselves out of a job<\/strong>. Why are they not concerned about this? Because there will ALWAYS be a new challenge.&nbsp;<\/p>\n\n\n\n<p>So what does automation mean in the SRE sense? <\/p>\n\n\n\n<p>It\u2019s not about AI and all that funky business. It can be<strong> as simple as not clicking around or manually typing<\/strong> in commands every time something needs to happen.&nbsp;<\/p>\n\n\n\n<p><strong>How would it look to reduce toil? <\/strong><\/p>\n\n\n\n<p>It can be as simple as running CronJobs for scheduled tasks going all the way up to AIOps.&nbsp;<\/p>\n\n\n\n<p>The key is to start small and scale up your efforts as you see improvements come to fruition.<\/p>\n\n\n<h2 class=\"gb-headline gb-headline-cb2cde69 gb-headline-text\" id=\"sre-pattern-3-empower-developers\">SRE Pattern #3: Empower developers<\/h2>\n\n\n<figure class=\"wp-block-image aligncenter size-full is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/sysmit.com\/cf22\/wp-content\/uploads\/2022\/04\/sre-culture-empower-developers.png\" alt=\"Site Reliability Engineers empower their developer peers by giving them tools to deploy fast and setup guardrails to prevent mishaps rather than slowing down deploys with pre-deployment checks\" class=\"wp-image-223\" width=\"479\" height=\"521\" srcset=\"https:\/\/sysmit.com\/cf22\/wp-content\/uploads\/2022\/04\/sre-culture-empower-developers.png 479w, https:\/\/sysmit.com\/cf22\/wp-content\/uploads\/2022\/04\/sre-culture-empower-developers-276x300.png 276w\" sizes=\"(max-width: 479px) 100vw, 479px\" \/><\/figure>\n\n\n\n<p>I previously mentioned that operations can put out a lot of stops in the name of stability.<\/p>\n\n\n\n<p>In some situations, deployments turn into an onerous compliance process.<\/p>\n\n\n\n<p>The process may employ methods to slow down or prevent changes altogether. <\/p>\n\n\n\n<p>This can include strict release policies, lengthy compliance checklists, and<strong> approval workflows involving non-technical management<\/strong>.<\/p>\n\n\n\n<p>The last issue is known in many organizations as a CAB or Change Approval Board. Word to those running CABs: developers hate them and they greatly slow software velocity.<\/p>\n\n\n\n<p>Being cautious is a virtue, but being overcautious can be disastrous. Onerous checks and balances that slow down the deployment to production can hurt in many ways. It can:<\/p>\n\n\n\n<ul>\n<li>hurt developer morale<\/li>\n\n\n\n<li>create rifts between feature and operations teams<\/li>\n\n\n\n<li>slow down deployments of critical new features<\/li>\n\n\n\n<li>reduce business competitiveness due to slowed releases<\/li>\n<\/ul>\n\n\n\n<p>Consider this as an SRE&#8217;s path to achieving reliable systems without creating animosity with developers:<\/p>\n\n\n\n<div class=\"wp-block-stackable-icon-list stk-block-icon-list stk-block stk-d0a659a\" data-block-id=\"d0a659a\"><style>.stk-d0a659a ul li,.stk-d0a659a ol li{font-size:20px !important}.stk-d0a659a ul li{list-style-image:url('data:image\/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9IjAgMCAxOTAgMTkwIiBzdHlsZT0iZmlsbDogIzAzNjQwNSAhaW1wb3J0YW50OyBjb2xvcjogIzAzNjQwNSAhaW1wb3J0YW50O3RyYW5zZm9ybTogcm90YXRlKDBkZWcpICFpbXBvcnRhbnQ7b3BhY2l0eTogMSAhaW1wb3J0YW50OyI+PHBvbHlnb24gcG9pbnRzPSIxNzMuOCwyOC40IDYwLjQsMTQxLjggMTUuNyw5Ny4yIDUuMSwxMDcuOCA2MC40LDE2MyAxODQuNCwzOSAxNzMuOCwyOC40IiBmaWxsPSIjMDM2NDA1IiBzdHJva2U9IiMwMzY0MDUiIHN0eWxlPSJmaWxsOiByZ2IoMywgMTAwLCA1KTsgc3Ryb2tlOiByZ2IoMywgMTAwLCA1KTsiLz48L3N2Zz4=') !important}.stk-d0a659a li::marker{color:#036405 !important;font-size:2em !important}@media screen and (max-width:1023px){.stk-d0a659a ul li,.stk-d0a659a ol li{font-size:20px !important}}<\/style><ul><li>Offer an easy-to-follow pre-launch checklist for developers<\/li><li><a href=\"https:\/\/www.srepath.com\/reduce-software-outage-risk-with-passive-guardrails\/\">Create passive guardrails e.g. 2-person authentication<\/a><\/li><li>Offer <a href=\"https:\/\/engineering.atspotify.com\/2020\/08\/how-we-use-golden-paths-to-solve-fragmentation-in-our-software-ecosystem\/\">&#8220;Golden Paths&#8221; (like Spotify does)<\/a> of opinionated and supported ways to package code for production <\/li><\/ul><\/div>\n\n\n\n\n<h2 class=\"gb-headline gb-headline-2b97512e gb-headline-text\" id=\"sre-culture-pattern-4-work-to-improve-the-future-state\">SRE Culture Pattern #4: Work to improve the future state<\/h2>\n\n\n<figure class=\"wp-block-image aligncenter size-full is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/sysmit.com\/cf22\/wp-content\/uploads\/2022\/04\/sre-culture-improve-future-state.png\" alt=\"SRE means working to improve the future state - engineers spend more time on proactive futureproofing work than reactive operations work\" class=\"wp-image-225\" width=\"578\" height=\"432\" srcset=\"https:\/\/sysmit.com\/cf22\/wp-content\/uploads\/2022\/04\/sre-culture-improve-future-state.png 578w, https:\/\/sysmit.com\/cf22\/wp-content\/uploads\/2022\/04\/sre-culture-improve-future-state-300x224.png 300w\" sizes=\"(max-width: 578px) 100vw, 578px\" \/><\/figure>\n\n\n\n<p>One of Google&#8217;s original SRE rules was that Site Reliability Engineers should keep operational (toil) work below 50%.<\/p>\n\n\n\n<p>This is one of those adages that can translate well outside of Google&#8217;s sphere of influence.<\/p>\n\n\n\n<blockquote class=\"wp-block-quote\" style=\"font-style:normal;font-weight:400\">\n<p><em>&#8220;At least 50% of each SRE&#8217;s time should be spent on engineering project work that will either reduce future toil or add service features.&#8221;<\/em><\/p>\n<cite>&#8211; <a href=\"https:\/\/sre.google\/sre-book\/eliminating-toil\/#:~:text=Our%20SRE%20organization%20has%20an,toil%20or%20add%20service%20features.\">Eliminating toil<\/a>, Site Reliability Engineering (2016)<\/cite><\/blockquote>\n\n\n\n<p>I&#8217;ve noticed a regular trend on r\/sre of &#8220;I want to quit SRE&#8221; posts, but also a common thread among the people posting these.<\/p>\n\n\n\n<p>These <strong>SREs are burnt out and\/or want to quit their current role<\/strong> to move back to developer work.<\/p>\n\n\n\n<p>Many join with the (correct) assumption that they will be doing proactive, future-changing work.<\/p>\n\n\n\n<p>But they spend way more than 50%, sometimes almost 100% of their time on reactive operational work like being on-call or inspecting pre-launch configurations.<\/p>\n\n\n\n<p>Where an SRE stands out from their operational counterparts like sysadmins is that they can create tooling to <strong>make the system more resilient to future failures<\/strong>.<\/p>\n\n\n\n<p>All this without having to physically <em>be there<\/em>.<\/p>\n\n\n\n<p>So please, let your SREs do as much proactive work as they can handle.<\/p>\n\n\n<h2 class=\"gb-headline gb-headline-41f949f9 gb-headline-text\" id=\"sre-culture-pattern-5-embrace-risk\">SRE Culture Pattern #5: Embrace risk<\/h2>\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"590\" height=\"423\" src=\"https:\/\/sysmit.com\/cf22\/wp-content\/uploads\/2022\/04\/sre-culture-embraces-risk.png\" alt=\"Site Reliability Engineering means embracing an acceptable threshold of risk (also known as an error budget)\" class=\"wp-image-222\" srcset=\"https:\/\/sysmit.com\/cf22\/wp-content\/uploads\/2022\/04\/sre-culture-embraces-risk.png 590w, https:\/\/sysmit.com\/cf22\/wp-content\/uploads\/2022\/04\/sre-culture-embraces-risk-300x215.png 300w\" sizes=\"(max-width: 590px) 100vw, 590px\" \/><\/figure>\n\n\n\n<p>Let&#8217;s wait for a moment while the IT traditionalists clear the room. Okay, now we can talk about risk&#8230; but we will aim to embrace it rather than &#8220;mitigate it&#8221;.<\/p>\n\n\n\n<p><strong>Risk mitigation like change approval workflows often plays out to be the good guy<\/strong> for fast-moving and complex situations, but my experience has shown me that it often isn&#8217;t.&nbsp;<\/p>\n\n\n\n<p>It can bog down progress.<\/p>\n\n\n\n<p>Site Reliability Engineering allows for handling risk with an alternative view. It does this through <strong>the lens of<\/strong> <strong>error budgets.<\/strong><\/p>\n\n\n\n<p>An error budget is the margin of downtime allowed in any given period. If your reliability requirement is 99.9%, you have an error budget of 0.1% or 43.83 minutes per month.<\/p>\n\n\n\n<p>With error budgets, <strong>teams can take risks that can potentially cause downtime<\/strong>. But the total value of the downtime should fall below the error budget.&nbsp;<\/p>\n\n\n\n<p>By the way, the scary autoscaler in the image is an inside joke for those who use Kubernetes (K8s). Many K8s users are aware of or may have experienced bizarre issues with the autoscaler. It can bring down a production system and chew up your error budget in one go.&nbsp;<\/p>\n\n\n<h2 class=\"gb-headline gb-headline-95204c6e gb-headline-text\" id=\"sre-pattern-6-have-an-experimenters-mindset\">SRE Pattern #6: Have an experimenter&#8217;s mindset<\/h2>\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"501\" height=\"498\" src=\"https:\/\/sysmit.com\/cf22\/wp-content\/uploads\/2022\/04\/sre-culture-experimentation-mindset.png\" alt=\"Site Reliablity Engineers constantly experiment as they measure, make hypotheses, develop tools and repeat the process\" class=\"wp-image-224\" srcset=\"https:\/\/sysmit.com\/cf22\/wp-content\/uploads\/2022\/04\/sre-culture-experimentation-mindset.png 501w, https:\/\/sysmit.com\/cf22\/wp-content\/uploads\/2022\/04\/sre-culture-experimentation-mindset-300x298.png 300w, https:\/\/sysmit.com\/cf22\/wp-content\/uploads\/2022\/04\/sre-culture-experimentation-mindset-150x150.png 150w\" sizes=\"(max-width: 501px) 100vw, 501px\" \/><\/figure>\n\n\n\n<p>Remember that SREs should spend at least 50% of their time on non-reactive, non-operational work.&nbsp;This <strong>time should be partly dedicated to experiments<\/strong>. <\/p>\n\n\n\n<p>That\u2019s when the error budget can get burned down.&nbsp;<\/p>\n\n\n\n<p>It can involve experiments like chaos engineering to test resilience in parts of the system.<\/p>\n\n\n\n<p>To gather experiment results, Site Reliability Engineers should <strong>keep a good observability setup at a <\/strong><strong>minimum<\/strong>. That means adequate logging, tracing, and monitoring.&nbsp;<\/p>\n\n\n\n<p>In experimenter mode, SREs can create automation tools, APM tools, and more from the data they gather.&nbsp;<\/p>\n\n\n\n<p>The final step in creating these tools is to test them in a controlled way and evaluate the observability results.<\/p>\n\n\n<h2 class=\"gb-headline gb-headline-1c42fafa gb-headline-text\" id=\"sre-culture-pattern-7-understand-the-wider-system-of-work\">SRE Culture Pattern #7: Understand the wider system of work<\/h2>\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"485\" height=\"514\" src=\"https:\/\/sysmit.com\/cf22\/wp-content\/uploads\/2022\/04\/sre-culture-understand-wider-system.png\" alt=\"Site Reliability Engineers understand the wider system including a multitude of tools, microservices and even sociotechnical systems\" class=\"wp-image-226\" srcset=\"https:\/\/sysmit.com\/cf22\/wp-content\/uploads\/2022\/04\/sre-culture-understand-wider-system.png 485w, https:\/\/sysmit.com\/cf22\/wp-content\/uploads\/2022\/04\/sre-culture-understand-wider-system-283x300.png 283w\" sizes=\"(max-width: 485px) 100vw, 485px\" \/><\/figure>\n\n\n\n<p>When a system has become unstable or failed, several things across many facets of the system may have gone wrong.<\/p>\n\n\n\n<p>That\u2019s why SREs need to have their finger on the pulse of many things and <strong>do much more than incident response and running post-mortems.<\/strong> (JC van Winkel 2017)<\/p>\n\n\n\n<p>As SREs increase in seniority, they get involved in architecture reviews. This calls for a broader understanding of how the underlying system has been designed.<\/p>\n\n\n\n<p>It&#8217;s <strong>also<\/strong> <strong>helpful to understand topics that are not technical <\/strong>but can help your approach to the work.<\/p>\n\n\n\n<p>Examples of these include:<\/p>\n\n\n\n<div class=\"wp-block-stackable-icon-list stk-block-icon-list stk-block stk-b023ff7\" data-block-id=\"b023ff7\"><style>.stk-b023ff7 ul li,.stk-b023ff7 ol li{font-size:24px !important}.stk-b023ff7 li{padding-inline-start:6px !important;margin-bottom:6px !important}.stk-b023ff7{column-gap:0px !important}.stk-b023ff7 ul,.stk-b023ff7 ol{padding-left:34px !important}.stk-b023ff7 ul li{list-style-image:url('data:image\/svg+xml;base64,PHN2ZyBhcmlhLWhpZGRlbj0idHJ1ZSIgZm9jdXNhYmxlPSJmYWxzZSIgZGF0YS1wcmVmaXg9ImZhcyIgZGF0YS1pY29uPSJpbmZvIiBjbGFzcz0ic3ZnLWlubGluZS0tZmEgZmEtaW5mbyBmYS13LTYiIHJvbGU9ImltZyIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIiB2aWV3Qm94PSIwIDAgMTkyIDUxMiIgc3R5bGU9ImZpbGw6ICMwMDAgIWltcG9ydGFudDsgY29sb3I6ICMwMDAgIWltcG9ydGFudDt0cmFuc2Zvcm06IHJvdGF0ZShkZWcpICFpbXBvcnRhbnQ7Ij48cGF0aCBmaWxsPSIjMDAwIiBkPSJNMjAgNDI0LjIyOWgyMFYyNzkuNzcxSDIwYy0xMS4wNDYgMC0yMC04Ljk1NC0yMC0yMFYyMTJjMC0xMS4wNDYgOC45NTQtMjAgMjAtMjBoMTEyYzExLjA0NiAwIDIwIDguOTU0IDIwIDIwdjIxMi4yMjloMjBjMTEuMDQ2IDAgMjAgOC45NTQgMjAgMjBWNDkyYzAgMTEuMDQ2LTguOTU0IDIwLTIwIDIwSDIwYy0xMS4wNDYgMC0yMC04Ljk1NC0yMC0yMHYtNDcuNzcxYzAtMTEuMDQ2IDguOTU0LTIwIDIwLTIwek05NiAwQzU2LjIzNSAwIDI0IDMyLjIzNSAyNCA3MnMzMi4yMzUgNzIgNzIgNzIgNzItMzIuMjM1IDcyLTcyUzEzNS43NjQgMCA5NiAweiIgc3Ryb2tlPSIjMDAwIiBzdHlsZT0iZmlsbDogcmdiKDAsIDAsIDApOyBzdHJva2U6IHJnYigwLCAwLCAwKTsiLz48L3N2Zz4=') !important}.stk-b023ff7 li::marker{font-size:1.5em !important}@media screen and (max-width:1023px){.stk-b023ff7 ul li,.stk-b023ff7 ol li{font-size:24px !important}}<\/style><ul><li>Complexity theory, in particular, the work of Sonja Blignaut and Dave Snowden (Cynefin framework)<\/li><li>Team Topologies by Matthew Skelton and Manuel Pais<\/li><li>Capability maturity model for software development, as developed by the Carnegie Mellon University <\/li><li>Sociotechnical systems, exploring the relationship between human social dynamics and technical systems <\/li><li>New design patterns in underlying systems as they evolve e.g. container patterns in Kubernetes <\/li><\/ul><\/div>\n\n\n<h2 class=\"gb-headline gb-headline-70a26d5c gb-headline-text\" id=\"parting-words\">Parting words<\/h2>\n\n\n<p>Site Reliability Engineering is a proactive approach to the growing software complexity problem. Reliability is at the heart of the function.<\/p>\n\n\n\n<p>Site Reliability Engineers are production-level engineers focused on the performance of software once it enters the real world. <\/p>\n\n\n\n<p>An SRE&#8217;s north star was originally <em>software reliability<\/em>. <\/p>\n\n\n\n<blockquote class=\"wp-block-quote\">\n<p>\u201cWe want to keep our site up, always.\u201d \u2014 JC Van Winkle, Site Reliability Engineer at Google Zurich.<\/p>\n<\/blockquote>\n\n\n\n<p>This north star is evolving as SREs become interested in being involved in reliable, secure, and performant software. It&#8217;s an interesting future for Site Reliability Engineering.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Who should read this: Introduction Despite its now antiquated sounding name, Site Reliability Engineering (SRE) as a discipline has strong future promise to proactively improve software reliability in production. As software complexity continues to increase, so will the need for better and better practice of SRE. It is undoubtedly an exciting but enigmatic field, with [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[60,1,29,4],"tags":[],"_links":{"self":[{"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/posts\/254"}],"collection":[{"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/comments?post=254"}],"version-history":[{"count":67,"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/posts\/254\/revisions"}],"predecessor-version":[{"id":5768,"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/posts\/254\/revisions\/5768"}],"wp:attachment":[{"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/media?parent=254"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/categories?post=254"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/tags?post=254"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}