{"id":5872,"date":"2023-10-31T12:41:20","date_gmt":"2023-10-31T02:41:20","guid":{"rendered":"https:\/\/sysmit.com\/cf22\/?p=5872"},"modified":"2024-03-28T16:14:05","modified_gmt":"2024-03-28T06:14:05","slug":"making-sense-opentelemetry-observability-adriana-villela","status":"publish","type":"post","link":"https:\/\/sysmit.com\/cf22\/making-sense-opentelemetry-observability-adriana-villela\/","title":{"rendered":"#13 Making Sense of OpenTelemetry and Observability (with Adriana Villela)"},"content":{"rendered":"\n<p><strong><span class=\"stk-highlight\" style=\"color: var(--base-3);background-color: var(--contrast-3)\">Episode 13 [SREpath Podcast]<\/span><\/strong><\/p>\n\n\n<div class=\"gb-grid-wrapper gb-grid-wrapper-8615df8f\">\n<div class=\"gb-grid-column gb-grid-column-dceeb69e\"><div class=\"gb-container gb-container-dceeb69e\">\n\n<iframe loading=\"lazy\" src=\"https:\/\/podcasters.spotify.com\/pod\/show\/srepath\/embed\/episodes\/Making-Sense-of-OpenTelemetry-and-Observability-Landscape-with-Adriana-Villela-e2aknqg\/a-aag07rp\" height=\"142px\" width=\"600px\" frameborder=\"0\" scrolling=\"no\"><\/iframe>\n\n<\/div><\/div>\n\n<div class=\"gb-grid-column gb-grid-column-a40a75d0\"><div class=\"gb-container gb-container-a40a75d0\"><\/div><\/div>\n<\/div>\n\n\n<p>Ash Patel interviews Adriana Villela who is a CNCF ambassador, OpenTelemetry contributor, and senior developer advocate at Lightstep.  <\/p>\n\n\n\n<p>Adriana talks about her experiences discovering observability, life as a team leader, and the promise of OpenTelemetry. <\/p>\n\n\n\n<p>She sheds light on the importance of observability practices and the role of OpenTelemetry in standardizing instrumentation collection, enabling organizations to achieve comprehensive observability in complex systems.<\/p>\n\n\n<h2 class=\"wp-block-heading\" id=\"episode-transcript\">Episode Transcript<\/h2>\n\n\n<p><strong>Ash Patel:<\/strong>&nbsp;Adriana, great to have you. I&#8217;m looking forward to learning what you have to teach us about all things SRE, or at least a few interesting things you&#8217;ve been working on recently.<\/p>\n\n\n\n<p>And I know you&#8217;ve been talking about OpenTelemetry quite a bit. So, looking forward to talking about that. But let&#8217;s do a quick introduction of who you are and what you do.<\/p>\n\n\n\n<p>What do you do SRE-wise in terms of you and your company?<\/p>\n\n\n\n<p><strong>Adriana Villela:<\/strong>&nbsp;I guess I&#8217;m in the SRE adjacent space if you will. I guess it is part of SRE.<\/p>\n\n\n\n<p>So I&#8217;m a Senior developer advocate at LightStep, which now is part of ServiceNow. So it&#8217;s actually called ServiceNow Cloud Observability. And Lights or ServiceNow Cloud Observability, I&#8217;m, I&#8217;m still getting used to the new name is an observability tool. And observability I would say is one of the important aspects of SRE.<\/p>\n\n\n\n<p>What I do as part of my job is to educate folks on observability practices with a focus on OpenTelemetry, which is a great open-source CNCF project that aims to standardize how we collect instrumentation from our systems so that we can achieve those beautiful observability goals.<\/p>\n\n\n\n<p><strong>Ash Patel:<\/strong>&nbsp;OpenTelemetry has got to be one of the hottest projects out there on CNCF right now.<\/p>\n\n\n\n<p><strong>Adriana Villela:<\/strong>&nbsp;Mm hmm. I think it&#8217;s got the second-highest contributions behind Kubernetes, which is pretty wild when you think about it.<\/p>\n\n\n\n<p><strong>Ash Patel:<\/strong>&nbsp;I was gonna say. I know Kubernetes is number one for obvious reasons.<\/p>\n\n\n\n<p>Actually, not obvious reasons, but you know what I mean. Yeah. Yeah, open telemetry is such an important area.<\/p>\n\n\n\n<p>I say to a lot of people that observability is the foundation practice that any cloud operation needs to have to be effective.<\/p>\n\n\n\n<p><strong>Adriana Villela:<\/strong>&nbsp;Yes, yes, absolutely.<\/p>\n\n\n\n<p>It is such a foundational practice. I think one of my pet peeves, and I had to edit myself when I was introducing myself and talk about SRE because I think a lot of the industry tends to treat observability as a separate thing when in fact it&#8217;s part of the overall practice.<\/p>\n\n\n\n<p>When you consider the fact that we are in this microservices world, right? I think a lot of organizations have moved away from monoliths and have embraced the microservices model, which, you know, has its ups and downs.<\/p>\n\n\n\n<p>But one of its downs is you have so many different services. And depending on the size of your organization, you can have like ton of microservices interacting with each other in ways that you would never even predict. Right? And so observability helps to untangle those interactions.<\/p>\n\n\n\n<p>All of a sudden, You&#8217;re like, oh, I see how data flows from point A to point B of my application.<\/p>\n\n\n\n<p><strong>Ash Patel:<\/strong>&nbsp;You mentioned monoliths. What are your thoughts on a lot of companies talking about how they&#8217;re switching back to monoliths, but I don&#8217;t know why they would do that. Is it to save money or is it to cut down on their spending on doing things like observability? What are your thoughts on monoliths?<\/p>\n\n\n\n<p><strong>Adriana Villela:<\/strong>&nbsp;I&#8217;m kind of happy to hear that because I do feel like a lot of organizations went on this microservices craze because it was a fad.<\/p>\n\n\n\n<p>And so I feel like some C suite executive attended some seminar or whatever where they were talking about microservices and then all of a sudden it&#8217;s like, Oh, we must do this at, our organization, it&#8217;s going to solve all of our problems and feel like a lot of companies rushed onto the microservice bandwagon without really thinking about does this actually do what I need it to do?<\/p>\n\n\n\n<p>So I&#8217;m kind of happy that some companies are taking a step back to the monolith.<\/p>\n\n\n\n<p>I think what people conflate with monoliths and microservices is that there are still efficiencies to be made when it comes to monoliths where I think if you structure your project properly, you don&#8217;t have to have some of those huge long build times that you would normally see with your monoliths in the past. And I think that&#8217;s one of the things that I think folks try to alleviate with the microservices model. Cause it&#8217;s like, you just change one little thing. You don&#8217;t have to build the entire frigging project.<\/p>\n\n\n\n<p>But I think if you structure your monolithic app properly, it doesn&#8217;t have to be painful either.<\/p>\n\n\n\n<p><strong>Ash Patel:<\/strong>&nbsp;Exactly. You don&#8217;t actually have to do a monolith as a waterfall project either. Mm. Mm. Which I think people construe a few things together. They say, hey, microservices is agile, monoliths are waterfall.<\/p>\n\n\n\n<p>And I&#8217;m saying, no. But anyway, we could talk about architectural decisions all night long.<\/p>\n\n\n\n<p>How did you get to working with people in observability in the first place? How did you get into the space in terms of your work history, your personal history with it?<\/p>\n\n\n\n<p><strong>Adriana Villela:<\/strong>&nbsp;So I&#8217;ll say I got into the observability space.<\/p>\n\n\n\n<p>I caught wind of Charity Major&#8217;s tweets when I was doing some research for something else. I had like a little side hustle with, with a friend of mine and I don&#8217;t know, we were just doing some late-night research and then I happened upon one of her tweets, which I don&#8217;t think even had anything to do with observability.<\/p>\n\n\n\n<p>And then my friend, he started obsessing and reading all her tweets. And then he&#8217;s like, Hey, you should check out this observability thing. It&#8217;s really cool. And I&#8217;m like, okay, but I don&#8217;t get what it is. And he tried to explain it to me a bunch and I&#8217;m like, I sorta get it, but I don&#8217;t really get it.<\/p>\n\n\n\n<p>And then like the company where I,<\/p>\n\n\n\n<p>I was out of the time.<\/p>\n\n\n\n<p>I was at the time they were exploring some observability products. They had some microservices and they were like looking into getting into Kubernetes. They, they were like hung up on this one product.<\/p>\n\n\n\n<p>I&#8217;m like, you know, we should check out these other products around observability. Me saying this with like a vague notion of what observability was about. And then a couple of months later I found myself working at Tucows where I was managing two teams. It was a platform engineering team where, instead of using Kubernetes, they were using Nomad.<\/p>\n\n\n\n<p>So I had to learn that. But I was also managing an observability team, which was kind of funny because I had a vague notion of observability from my, you know, like, my chats with, with my friend doing, during our side hustle. But also, I was trying to wrap my head around and when I got this job I started thinking I&#8217;m like, I&#8217;m like leading a team of people.<\/p>\n\n\n\n<p>I better know what observability is because otherwise how am I supposed to like lead an entire organization on this journey, right? So I set out to do what I do best, which is learn the thing that I don&#8217;t know so I can do my job. And as part of that, I started blogging about it.<\/p>\n\n\n\n<p>And I already had a blog at the time on Medium which was focused on, like, more the Kubernetes, Argo CD side of things. And so I started exploring observability. Blogging about it.<\/p>\n\n\n\n<p>As I learned more, then I was able to start setting a direction for my team. So this team that I inherited It was an observability team.<\/p>\n\n\n\n<p>I&#8217;ll say this in air quotes. And really what the team did was manage a set of, of open source tools in house. But at the same time, the organization was sending its observability data to a SaaS vendor. So I&#8217;m sitting here thinking, okay, we&#8217;re already paying somebody to receive our data. Why the hell is this team managing a set of tools in house?<\/p>\n\n\n\n<p>At this point, I had a better idea of what observability was all about, and I said, let&#8217;s pivot this team completely. I want to focus on observability practices and not on observability tooling. Leave the tooling to the experts. So then I sought to educate my team on like, what are proper observability practices?<\/p>\n\n\n\n<p>And that&#8217;s around the time that I got wind of open telemetry. Started looking at it a little bit closer. So the company before I was there was like that, I think that was the first time I&#8217;d heard of open telemetry, but it&#8217;s. It seemed like, I don&#8217;t know, it, it, it seemed like a very small thing. And then when I, when I went to my job where I was managing this observability team It was something that I started looking at more closely as I started reading blog posts about it and whatnot and educating myself and I&#8217;m like, huh, I feel like this is the future of observability.<\/p>\n\n\n\n<p>So I started pushing really hard for the organization to adopt OpenTelemetry, which was very difficult because I found myself in a position, this was 2021, where OpenTelemetry, like, the tracing specification was not yet in general availability. And this was like summer, fall of 2021. And I&#8217;m sitting there thinking, no, no, this is going to be the next big thing.<\/p>\n\n\n\n<p>I assure you it&#8217;s going to be the next big thing. But how can I convince these folks that OpenTelemetry is the next big thing when tracing isn&#8217;t even in general availability?<\/p>\n\n\n\n<p>So I thought, I, I need to like, calm these folks concerns. So what I did, which is what I do best, I think, in my which I&#8217;ve been able to leverage in my capacity of as a developer advocate now, which is like make connections with people.<\/p>\n\n\n\n<p>And so I had made some friends in the open telemetry space.<\/p>\n\n\n\n<p>I&#8217;d talk to a couple of folks from two different vendors that we were also considering at the same time, because we were, you know, we had the SaaS vendor, but I&#8217;m like, well, is, is this the right fit for us? We should explore other vendors.<\/p>\n\n\n\n<p>I reached out to these folks who were, who worked for other vendors, but they put on their OpenTelemetry hats for this, right? They said, please come speak with us on a unified front under the OpenTelemetry umbrella to answer questions and concerns for folks in the organization who are hesitant about adopting OpenTelemetry.<\/p>\n\n\n\n<p>And to my surprise, they went along with my idea, I, I honestly thought, this is a total long shot, this is not happening. But they went with it and I remember telling my team, we need some seed questions in case this thing is going to be crickets, otherwise I&#8217;m going to look like total ass.<\/p>\n\n\n\n<p>So we came up with some seed questions, but guess what?<\/p>\n\n\n\n<p>For the whole hour, the folks from the dev teams, they had one question after another, and we didn&#8217;t even have to ask any of our seed questions, which I was very relieved to hear. To me, it showed that there was an interest in what was being done.<\/p>\n\n\n\n<p>There was a genuine interest in OpenTelemetry, and I think at the end of the day, mission accomplished. We managed to quell those fears as a result. And so the company was a lot more comfortable in terms of adopting OpenTelemetry. So that was my original foray into it. And then I ended up getting hired by one of the observability vendors as a developer advocate because of my blog posts around observability and OpenTelemetry, my explorations.<\/p>\n\n\n\n<p>Which for me, honestly, is, is something that I&#8217;ve dreamed of.<\/p>\n\n\n\n<p>I love educating folks through my blogging and I love digging into technical topics. So I feel like it was the best use of my, of my skills. And it&#8217;s been great because also like this job has been my first opportunity to contribute to open source, which I&#8217;d never done before.<\/p>\n\n\n\n<p>And it&#8217;s scary if you&#8217;ve never contributed to open source cause you&#8217;re being vulnerable. You&#8217;re putting yourself out there. It is terrifying. Like what if someone says your thing is crap, but I&#8217;ve been lucky cause in the OpenTelemetry community, no one goes and says your PR is crap.<\/p>\n\n\n\n<p>They always have very thoughtful suggestions always useful suggestions. So, even though I&#8217;m scared every single time, my fears are always put at ease because the folks are just genuinely nice about it.<\/p>\n\n\n\n<p>That&#8217;s been my journey into OpenTelemetry and observability.<\/p>\n\n\n\n<p><strong>Ash Patel:<\/strong>&nbsp;That&#8217;s the great thing about open source communities. It&#8217;s psychologically safe in that everyone&#8217;s trying to learn and trying to improve what they&#8217;re working on and they&#8217;ll share ideas, but in a constructive way.<\/p>\n\n\n\n<p><strong>Adriana Villela:<\/strong>&nbsp;Yes, absolutely.<\/p>\n\n\n\n<p><strong>Ash Patel:<\/strong>&nbsp;And I said that because I&#8217;m calling out to people to try and do the same as Adriana and contribute to open source if you can, and if you have the time to do it.<\/p>\n\n\n\n<p><strong>Adriana Villela:<\/strong>&nbsp;I do want to mention one thing actually on that same vein, like when, when I was at Tucows, I myself wasn&#8217;t practicing what I was preaching, but I did encourage my team as a whole, like my observability team, like when they were running into issues with, Oh, I&#8217;m waiting on this feature for OpenTelemetry and blah, blah, blah.<\/p>\n\n\n\n<p>I&#8217;m like, dude, just try to work on it if you can, if you feel like you, you can like put in a fix, do it. It took me a little bit longer and, you know, switching roles to like muster the courage to do it myself. But I, I totally agree with you.<\/p>\n\n\n\n<p>I think it&#8217;s so important for us to contribute to open source.<\/p>\n\n\n\n<p>And for our organizations to encourage contributing to open source, especially if we are consumers of that open source software.<\/p>\n\n\n\n<p><strong>Ash Patel:<\/strong>&nbsp;I have a question about your thoughts on how organizations that are non tech native feel about open source. We&#8217;ll get to that in a second, but I want to actually go back one step and talk about how OpenTelemetry fits into the whole observability area of software.<\/p>\n\n\n\n<p><strong>Adriana Villela:<\/strong>&nbsp;I think the important distinction to make is that open telemetry is not equal to observability. Open telemetry is one of those tools that enables observability. And honestly, like, you know, before open telemetry, there were things like open tracing and open census. OpenTelemetry is at the forefront now because it is the one that all of the big vendors have rallied behind and, and for a little bit of history, open source projects like OpenCensus and OpenTracing, they actually merged to form OpenTelemetry.<\/p>\n\n\n\n<p>But I wouldn&#8217;t be surprised if, like, somebody for funsies decided to, like, create their own tracing library or whatever. But at the end of the day, like, observability is about, you have that information. What do you do with the information about your system so that you can follow the breadcrumbs to answer the question, why is this happening?<\/p>\n\n\n\n<p>OpenTelemetry enables that to happen, right? It&#8217;s the thing that supplies that information.<\/p>\n\n\n\n<p>I would encourage folks to use OpenTelemetry as the standard for that sort of thing because if we get to the point where, like, we&#8217;re all using different tools, we end up losing out on the standardization and the work that has gone into OpenTelemetry, if that makes any sense.<\/p>\n\n\n\n<p>But a tool like this is only as good as the user base. And if we don&#8217;t have that standardization, if we don&#8217;t rally as a community to standardize around how we get our observability how we emit the observability data then we end up with all these different factions.<\/p>\n\n\n\n<p>I don&#8217;t want to be super prescriptive about it, but I think it ends up losing its worth if, if we go all like everyone on their own, do whatever they want for these sorts of things. Right?<\/p>\n\n\n\n<p><strong>Ash Patel:<\/strong>&nbsp;Yeah, I would be totally against it myself.<\/p>\n\n\n\n<p>If I were managing a team or multiple teams, which like I have in the past, I always used to tell my teams, don&#8217;t reinvent the wheel. Look at what&#8217;s available out there. And try and use that first. And then if we have to make a solution from scratch, then we&#8217;ll look at it then. But start learning from what everyone else has done, and then we can work with that.<\/p>\n\n\n\n<p>And this is how OpenTelemetry is doing. It&#8217;s giving you all this capability from the get go.<\/p>\n\n\n\n<p><strong>Adriana Villela:<\/strong>&nbsp;Yeah, totally. And the other thing that I want to emphasize is that because, you know, all the major observability vendors are behind it. It means that these vendors are not competing on the data that is ingested. They&#8217;re just competing on how they&#8217;re rendering the data.<\/p>\n\n\n\n<p>We all speak the same language. The data that we send to whatever vendor, vendor X and vendor Y, it&#8217;s the same data. Vendor X will probably render it differently from vendor Y, and so as a consumer. It&#8217;s up to you to decide which one is more useful for me for my troubleshooting purposes.<\/p>\n\n\n\n<p><strong>Ash Patel:<\/strong>&nbsp;You actually have a bit of experience with running teams. So, I&#8217;d be interested to know in terms of your observability team, how you managed them, how you hired them, how you managed the performance. .<\/p>\n\n\n\n<p><strong>Adriana Villela:<\/strong>&nbsp;When I joined Tucows, I was lucky enough that there were some folks on both of my teams already, and it was, like I said, it was managing two teams, and they were similar but different.<\/p>\n\n\n\n<p>And so there were already folks on those teams, but I needed to also hire a headcount. At that point, I already knew what I wanted out of my teams. And in particular, I&#8217;ll talk about the observability team.<\/p>\n\n\n\n<p>I wanted to hire somebody who either appreciated what observability had to offer either as an expert in observability or somebody who maybe they didn&#8217;t come from that observability background, maybe they came from a more traditional monitoring background, but saw the importance of observability and were willing to do a paradigm shift.<\/p>\n\n\n\n<p>It was a weird role because it was a technical role that wasn&#8217;t terribly hands on. So my team our mandate was to be able to educate folks at the organization on observability practices.<\/p>\n\n\n\n<p>We taught them the overarching best practices, but also we assisted them as far as unpacking some of the gnarly bits of OpenTelemetry so that they could instrument their code.<\/p>\n\n\n\n<p>Now that was very different from actually instrumenting the code themselves, because the slippery slope that you can get into by having an observability team, and it wasn&#8217;t just my team you can end up getting asked to instrument your developer&#8217;s code. Which is a very bizarre ask, because I know nothing about your code.<\/p>\n\n\n\n<p>I don&#8217;t know what&#8217;s important to you for debugging purposes. I don&#8217;t know what the code does in general. I might have a vague idea from a high level. But realistically speaking, it is not my code. I don&#8217;t have that kind of vested interest. I will help you in terms of, like, Hey, these might be the things that you might want to instrument.<\/p>\n\n\n\n<p>And I can provide some assistance on how to instrument in whatever language. But as far as instrumenting your own code, that was something that was on purpose outside of the realm of my team, because we wanted to encourage developers to instrument their own code.<\/p>\n\n\n\n<p>So hiring for the team was&#8230; was tricky in that sense, because I had a lot of candidates applying for the position who came from like an infrastructure as code background and were super gung ho on managing tools and I&#8217;m like that&#8217;s not exactly what we do.<\/p>\n\n\n\n<p>I wanted somebody who had an appreciation from an infrastructure as code background, somebody who had an appreciation from like an SRE type of background, you know, someone who felt the pain so that they could then go on and advocate for those observability practices.<\/p>\n\n\n\n<p>In a way, I guess I was hiring for an internal team of advocates in the organization. So it makes it a very tricky type of role to hire for because it&#8217;s a more squishy role that&#8217;s not completely hands on, but I need you to be an engineer. I need you to understand the pain around this sort of thing.<\/p>\n\n\n\n<p>That was on the hiring side, but then for the folks who I had inherited was a little bit tricky too, because all of a sudden I come in, I&#8217;m saying we&#8217;re pivoting the team&#8217;s mandate. That went well for some folks and for other folks they&#8217;re like, but I want to just do infrastructure as code.<\/p>\n\n\n\n<p>That is my thing. I enjoy it. So I had to have some hard conversations with some of my team members saying, look, I understand that this is your preference and that I have gone ahead and shifted what our team&#8217;s priorities are.<\/p>\n\n\n\n<p>And I am okay. If we don&#8217;t align, if you want to go look elsewhere in the organization, I will support you in that search, to ensure that you find something that&#8217;s more in line with your career goals in the organization.<\/p>\n\n\n\n<p>It&#8217;s kind of a tough conversation to have because it means that person admitting, yeah, I don&#8217;t really want to do this and me saying I&#8217;m okay letting you go because I think at the end of the day, when you&#8217;re managing a team you kind of have to all toe the party line together.<\/p>\n\n\n\n<p>You have to be aligned as to what the overarching team goals are. It&#8217;s okay to question certain things. I think it&#8217;s always important to question direction. Hey, maybe you should be doing X instead of Y. That&#8217;s totally fine, but to have folks who are constantly questioning what you&#8217;re doing as if you&#8217;re butting heads because you don&#8217;t see eye to eye on the philosophies and where the team is going, I think it ends up being very unhealthy for the team, and so you&#8217;re better off parting ways because you&#8217;re better off parting ways.<\/p>\n\n\n\n<p>All it takes is one person to start sowing dissent on the team to ruin the team cohesion. And then that becomes a very toxic work environment for everybody, right? Because then you start creating factions and people aren&#8217;t working together, they&#8217;re working against each other, and that&#8217;s not something that you can have.<\/p>\n\n\n\n<p>So, I mean, at the end of the day, being aligned on what our mandate was was very important, whether it was somebody already existing on the team or somebody being hired into the team.<\/p>\n\n\n\n<p><strong>Ash Patel:<\/strong>&nbsp;That was very insightful about a lot of things that you can apply beyond what your team issue was in terms of there&#8217;s so many teams that are facing this exact thing. There are people already there, there are people that you need, and you need to all align on whatever mandate you as a manager have to create to align with goals.<\/p>\n\n\n\n<p>Coming back to now, what you&#8217;re doing right now, you&#8217;d be having some interesting conversations with a lot of people every day about what their challenges are. So what kind of person typically reaches out to you or you end up having conversations with around observability and OpenTelemetry?<\/p>\n\n\n\n<p><strong>Adriana Villela:<\/strong>&nbsp;So we talked to, I&#8217;d say, a lot of people who are new to OpenTelemetry, who are just trying to figure this thing out. I had a conversation recently with somebody who was just starting to wrap their head around it. They knew that observability is important. They knew that OpenTelemetry is important, but they weren&#8217;t sure how to get started.<\/p>\n\n\n\n<p>So one of the things that I like to do is, I&#8217;ll give them a little overview and I&#8217;m happy to, anyone who wants to talk about observability, OpenTelemetry, like reach out to me. Let&#8217;s have that conversation. You know, I work for a vendor, but I&#8217;m not here to sell you a vendor product.<\/p>\n\n\n\n<p>I&#8217;m here to sell you on observability first and foremost. I&#8217;m here to sell you on observability, OpenTelemetry, the power of that.<\/p>\n\n\n\n<p>And I think that&#8217;s the most important thing we, we need to do right by, by our systems and make sure that we have the tools necessary to be able to troubleshoot in production.<\/p>\n\n\n\n<p>So having those conversations with folks to understand what their needs are, what their difficulties are. I can come because I have the experience of having been on the other side of looking at a couple of different vendors and really just focusing on observability practices.<\/p>\n\n\n\n<p>I can have that conversation with folks around what do you need to do in order to be successful in terms of bringing observability and open telemetry to your organization? Just, just like wrapping your mind around observability. I think with a lot of things in tech it&#8217;s very easy to get into various anti patterns.<\/p>\n\n\n\n<p>We saw that with the DevOps movement. I think one of my biggest pet peeves is still the idea of, like, DevOps as a job rather than as a practice because I think we, we ended up moving away from what DevOps was trying to promise.<\/p>\n\n\n\n<p>And I feel like, you know, it, it can be a similar slippery slope with observability where I have used OpenTelemetry, therefore my system is observable, and I&#8217;ve encountered situations where you can instrument till you&#8217;re blue in the face, but if you&#8217;re instrumenting the wrong thing that&#8217;s useless.<\/p>\n\n\n\n<p>Recently I&#8217;m one of the co leads of the Open Telemetry End User Working Group, and we had one person come in and present to us, Hazel Weekly, and she was talking about a story where one of the organizations where she was in.<\/p>\n\n\n\n<p>They instrumented their code. They over instrumented their code to the point where like, great, we&#8217;ve got open telemetry. It is useless. We have no idea what is going on here. It&#8217;s not giving us any meaningful information. So she actually had to spend some time unraveling that big mess and trying to get it to a state where the instrumentation was more meaningful.<\/p>\n\n\n\n<p>We need to be aware of the initial lack of knowledge around both observability and open telemetry &#8211; both from a I don&#8217;t know where to get started, to, Oh, I&#8217;ve come in guns blazing and I&#8217;ve over instrumented and it&#8217;s still not helping me.<\/p>\n\n\n\n<p>We really need to take a step back and really, really focus on the practices.<\/p>\n\n\n\n<p><strong>Ash Patel:<\/strong>&nbsp;Okay Adriana. Where do you see the category of observability heading in the next two, three, five years?<\/p>\n\n\n\n<p><strong>Adriana Villela:<\/strong>&nbsp;This is my, my hope and I, I, I hope that it, it follows my hope. So, in terms of observability, I hope to see more integrated with the SRE practice and not so much of this separate adjacent practice. We need to start talking about it from a more unified standpoint. The other thing that I wanted to mention is that I really want us to look at observability as a team sport.<\/p>\n\n\n\n<p>And not as just the thing that is the burden of the SRE. Because when you think about it, yes, the SRE is probably going to be your first line of defense when there&#8217;s an issue happening in production. But, let&#8217;s think back to who instrumented that code in the first place, the developer.<\/p>\n\n\n\n<p>So the developer, you would expect, would want to have a vested interest in making sure that the instrument, the code properly so that our poor friend, the SRE has an easier time of troubleshooting issues in production. But I think we can take a step further and say, the developers should take advantage of these observability platforms to even help troubleshoot their own code while they&#8217;re debugging, while they&#8217;re unit testing.<\/p>\n\n\n\n<p>But let&#8217;s take it a step further than that, which is before it goes into production, we send it over for testing, right, to our friends, the QAs and this idea of: &#8220;why don&#8217;t we empower QAs to be able to use observability data to make sense of what is happening in the application when they&#8217;re testing the code so that when there is an error, then the QA can go to the developer and say, &#8220;Hey, there&#8217;s bug and I can tell you where it is&#8221;.<\/p>\n\n\n\n<p>Or if there&#8217;s a bug and the QA can&#8217;t tell the developer where it is, that&#8217;s fine too because they can say to the developer, &#8221; Hey, your system&#8217;s not emitting enough information. I don&#8217;t know what the problem is. I think you&#8217;re missing some instrumentation. Go back. And then the other piece of that puzzle is leveraging those traces that you already have in place and creating trace based tests.<\/p>\n\n\n\n<p>Which is basically, as the name implies, you use traces to create automated integration tests, and there are some tools out there that already do that. The most talked about one in the space is TraceTest. It&#8217;s an open source tool that facilitates trace based testing.<\/p>\n\n\n\n<p>There&#8217;s another one called Maliby: it&#8217;s more of a JavaScript based trace based testing tool, and I don&#8217;t think it&#8217;s been maintained for the last little while. And then Helios I believe they were looking at trace based testing definitely last year. I don&#8217;t know where things are at now, but these are three different types of tools that you might want to look at if you want to explore trace based testing.<\/p>\n\n\n\n<p>I think it&#8217;s a really, really cool concept and that&#8217;s something like I want to see more. I want to see us talking more about trace based testing in the future because I think trace based testing can give us extra superpowers in terms of achieving more reliable systems especially because now you&#8217;re not having to worry about wrangling different testing frameworks to be able to do end to end tests of your systems, especially if you know, you&#8217;re dealing with a system where the microservice is written in different languages.<\/p>\n\n\n\n<p>Now trace based testing is like your single unified language for writing your your integration tests, which is awesome because the unified language itself is the trace.<\/p>\n\n\n\n<p>So again, we need to make sure that we have good quality traces so that we can leverage those traces to really make them work for us.<\/p>\n\n\n\n<p><strong>Ash Patel:<\/strong>&nbsp;It&#8217;s interesting you&#8217;re talking about tracing because I was crossing University Avenue in Toronto and I overheard two developers talking about tracing and testing. It was a little haphazard.<\/p>\n\n\n\n<p>They were talking about trace testing, but they were very confused about what they had just heard in a meeting prior. So there&#8217;s a lot of, it was surprising to me because then I stopped them. I said, Hey, what do you guys know about observability? And they said. Not much. And I asked them, do you need to learn more?<\/p>\n\n\n\n<p>They said, yes, please. Well, they didn&#8217;t say it like two boys in a Charles Dickens novel, but you get the idea that they were pretty keen on figuring out what is this thing that&#8217;s now suddenly become part of our responsibility in this increasing you build it, you run it model.<\/p>\n\n\n\n<p>It&#8217;s interesting that these kinds of things are being brought out. Trace testing was just mentioned. I&#8217;m gonna say one thing, and I&#8217;m not sure whether you&#8217;re gonna agree with me. It&#8217;s that all this needs to be brought together cohesively, so that when it&#8217;s shared with people, they can see this is where this fits into the observability practice.<\/p>\n\n\n\n<p>Where do you see SRE heading in the same way in the future? Where do you see it heading in the next few years?<\/p>\n\n\n\n<p><strong>Adriana Villela:<\/strong>&nbsp;I think for SRE, I&#8217;d like to see more conversations happening around things like self service provisioning.<\/p>\n\n\n\n<p>Actually Anna and I have a talk coming up at KubeCon where that is our subject matter. The idea that there&#8217;s the rise of platform engineering, which this may or may not be a popular opinion, but in, in the definition that we&#8217;re using, we see platform engineering as an extension of SRE.<\/p>\n\n\n\n<p>Where you&#8217;re not only concerned about the external customer, but the internal customer, i. e. the developer. And this idea that in platform engineering, as a developer, it&#8217;s so annoying when you&#8217;re trying to get your job done, and you need certain tools to get your job done, and you gotta wait on this platform team to provision stuff for you.<\/p>\n\n\n\n<p>And so you&#8217;re sitting there waiting, waiting, waiting, thinking, Frigman, why can&#8217;t I just use, like my own Kubernetes cluster? I don&#8217;t want to use this stupid Kubernetes cluster that&#8217;s, you know, in the cloud and blah, blah, blah. And, like, this is taking forever. I need to get my job done. And by having self service tooling available, it means that developers can then provision the resources that they need to get their jobs done on demand.<\/p>\n\n\n\n<p>These resources are provided by the platform team. However, the platform team automates the provisioning of those resources, right? They&#8217;re not having to like, you know, execute on these requests manually but on top of that, this whole idea of being packaged, so that it&#8217;s compliant within the organization.<\/p>\n\n\n\n<p>So security concerns are addressed. It&#8217;s using standard configurations. So you&#8217;re not deviating from what the standards within the organization are. So the thing that&#8217;s delivered to you that you&#8217;re able to use as part of your tooling is wrapped in a nice little neat bow.<\/p>\n\n\n\n<p>It&#8217;s got all the configurations you need and you don&#8217;t have to wait days and days and days to get the thing that you need to be able to do your job. So having the self service provisioning.<\/p>\n\n\n\n<p>And then the other thing that is important that I think follows in with that is whenever we go from one organization to another, I can&#8217;t tell you how many times you figure out this automation, then you go to the next organization.<\/p>\n\n\n\n<p>Oh crap. I got to do this exact same automation. Like, come on. We keep reinventing the same automations over and over and over again. Wouldn&#8217;t it be nice if we just had this marketplace of standardized automations where we can just pick and choose, right?<\/p>\n\n\n\n<p>And then make some customizations here and there, but we&#8217;re not having to like reinvent these friggin automations every single time for our SRE practice.<\/p>\n\n\n\n<p>The last one is policy as a service where I feel like we&#8217;re always at odds with InfoSec. InfoSec has a very tough job because they&#8217;re protecting us from all these attacks in our systems making sure that everything&#8217;s safe.<\/p>\n\n\n\n<p>And I feel like there&#8217;s&#8230; And they can&#8217;t even talk about their triumphs, right, because if you talk about, like, oh, I, I blocked this attack from happening, I blocked that attack from happening, that already, like, that&#8217;s not secure by nature. So, you, you can&#8217;t even, like, you can&#8217;t even gloat about your accomplishments but to be able to work together with Infosec so that we have certain policies in place that are codified, and again, so we don&#8217;t have to go through these long drawn out processes of, Oh, I need access to XYZ and have like all these approvals.<\/p>\n\n\n\n<p>Well, yes, we&#8217;ve already had the pre approvals because we already have all of these policies, like pre packaged, pre approved. Everything&#8217;s compliant. And so we&#8217;re not mad at our InfoSec friends. They&#8217;re just trying to do their jobs and we&#8217;re getting these compliant things. These things that are properly secured. Our Kubernetes clusters that are properly secured and are cloud resources that have the right permissions without having to like wait forever and ever and do this song and dance with InfoSec.<\/p>\n\n\n\n<p>Those are the areas that I, I would like to see like SRE going towards.<\/p>\n\n\n\n<p><strong>Ash Patel:<\/strong>&nbsp;We can only wait and see.<\/p>\n\n\n\n<p><strong>Adriana Villela:<\/strong>&nbsp;I know, right? Fingers crossed.<\/p>\n\n\n\n<p><strong>Ash Patel:<\/strong>&nbsp;What advice would you give to SREs who are dealing with a lot of challenges right now to help them in their working lives?<\/p>\n\n\n\n<p><strong>Adriana Villela:<\/strong>&nbsp;I would say never stop being curious. Whether or not you&#8217;re an SRE, I think that will take you very far in your career. Because by being curious, by always looking to do things a better way. It helps us grow in our practice. I think it allows us to innovate. I think we risk stagnation when we stop caring about innovating in our technologies, in our jobs.<\/p>\n\n\n\n<p>So let&#8217;s continue to be curious.<\/p>\n\n\n\n<p><strong>Ash Patel:<\/strong>&nbsp;There we have it. Never stop being curious. Thank you so much for joining me, Adriana.<\/p>\n\n\n\n<p><strong>Adriana Villela:<\/strong>&nbsp;Thanks for having me, Ash.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Episode 13 [SREpath Podcast] Ash Patel interviews Adriana Villela who is a CNCF ambassador, OpenTelemetry contributor, and senior developer advocate at Lightstep. Adriana talks about her experiences discovering observability, life as a team leader, and the promise of OpenTelemetry. She sheds light on the importance of observability practices and the role of OpenTelemetry in standardizing [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[72,48],"tags":[31],"_links":{"self":[{"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/posts\/5872"}],"collection":[{"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/comments?post=5872"}],"version-history":[{"count":7,"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/posts\/5872\/revisions"}],"predecessor-version":[{"id":6056,"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/posts\/5872\/revisions\/6056"}],"wp:attachment":[{"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/media?parent=5872"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/categories?post=5872"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/tags?post=5872"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}