Eileen Uchitelle - The Myth of the Modular Monolith - Rails World 2024

[Music] hi everyone uh I'd ask if you can hear me but I can hear myself pretty well so I think the sound is good who got woken up by the fire alarm at 2:30

a.m. wo who got stuck in an elevator and

a.m. wo who got stuck in an elevator and had a panic attack uh thank you to the strangers and my friends who were in the elevator with me who uh talked me off a ledge and then

figured out how to get me up the stairs so I didn't have to take the elevator again it was working fine this morning though so we're good um I'm Eileen usel

I'm honored to be your day two keynote it's uh really exciting to be here and I'm so happy that you all can make it to rails world thank you to the rails Foundation Amanda and everyone who

worked hard uh to put this conference on today it's really lovely to see all of you here I'm a senior staff software engineer at Shopify on the Ruby and

rails infrastructure team Toronto is one of our hubs and we love rails uh please come by our booth to chat or learn more about what roles we're currently hiring

for I've been a member of the Rails core team since 2017 the rails core team is the driving force behind the framework we make decisions about the direction and evolution of rails as well as

collaborate with contributors and the community being on the core team has been the highlight of my career it's enabled me to have a deep and Lasting effect on the

framework I've been building rails applications for about 14 years now throughout my career I've seen a lot of different types of rails applications I've worked at companies

that had less than 10 employees and companies with over 11,000 I've worked on Rails applications that were brand new and others that have been around since the dawn of the framework

I've seen applications by developers just learning rails and I've seen ones written by dhh himself throughout my career I've spent an extensive time working on two of the earliest rails applications in existence

base camp classic and Shopify while base camp classic is still running in production it only gets security updates and it's actually still running on Rails 3 to this makes shopify's core monolith the oldest

continuously developed production application on the planet it was built on a version of rails that wasn't yet released the public and now runs off rails main while the application has changed a lot in the last 20 years it's

still fun to look back and see Toby's first commit uh and see just how excited he was to be using rails whoops in between 37 signals and Shopify

I worked at GitHub for 5 years and spent the first two and a half years there upgrading the main monolith from rails 32 to running off rails Edge I think I'm the only person who's worked at all

three of these companies and seen these three early rails applications one thing that stood out over the years is that eventually rails applications get to a state where the

framework stops bringing developers joy and happiness as an organizations grow uh applications tend to become a ball of mud the code lacks organization

and structure onboarding is difficult and painful CI is slow and Flaky tests um tests are flaky simple simple small

and simple changes seem to hit endless amounts of friction in development at this point many companies ask themselves what now they feel that

rails is no longer meeting their needs from an architectural perspective often organizations will start looking into microservices in order to get the experience of building a Greenfield application and move away

from the monolith there was a point at GitHub where all anyone could talk about was microservices saving us from ourselves about 6 years ago Shopify began

exploring modularizing our core monolith it seemed like The Best of Both Worlds the Hope was that we could feel like we're working on a smaller application without the network latency and organizational politics of

microservices it could solve all of our engineering problems than we get to keep writing rails what could be better than that right before I left in 2012 GitHub started modularizing their monolith as

well realizing that turning the entire monolith into microservices wasn't a tenable goal Gusto zendesk doximity and other companies with large monoliths have also adopted this

architecture and while I hear a lot of voices pushing for this new pattern in rails applications it's proven to not be the Silver Bullet that we'd hoped for after all these years if we look back at the problems that we were trying to

solve we'll see that they are still ever present and new challenges have Arisen from the from our effort the myth of the modular monolith is that it promises A Greener pasture better structure less coupling but what

we get instead is a new set of challenges and unmet goals on the surface our problems appear to be technical but if we look deeper we'll see that they are actually human and cultural challenges and you can't solve

human and culture Problems by changing your architecture today we'll examine the difficulties that companies face as their applications scale we'll explore why modular monoliths seem like an ideal

solution and uncover the underlying causes of issues we're trying to address we'll discuss how to tackle human problems um while avoiding the pitfalls of

chasing false promises and silver bullets first let's talk about the very real pain points that led companies like Shopify GitHub Gusto and others to modularize their rails

applications many companies that are expanding hiring and growing will face these same challenges over time I've categorized our common problems into three types architectural operational and

organizational architectural issues refer to challenges related to overall design and structure of a softare Ware system they affect maintainability and scalability of an application

architectural problems creep in over time um as more and more features are added to the an application and refactoring isn't prioritized a common architectural problem that occurs in

growing applications is lack of organization and structure rails is an MVC framework so all the models go in app models and controllers and app controllers and Views and app views and

so on as the number of those models and controllers and Views uh increases rail's default directory structure can become unwieldy if you're not careful the lack of organization and structure

means that developers rarely consider where new code should go or when existing code should be refactored to live elsewhere everything just goes into the default folders making it difficult to discern what concepts belong together

this is often what developers mean when they talk about the ball of mud as if it's Ra's fault and not our own doing that code didn't follow proper design in addition to lack of organization and structure another that

growing applications have uh is tight coupling and lack of boundaries it's easy to see how this happens in a Ra's application Ruby is a language where code is globally accessible so without good judgment code can become tightly

coupled changes that seem simple unintentionally impact other parts of the application in ways that are hard to predict and control when this happens you either have to fix all the collar or refactor the code base to separate the

functionality these kinds of side effects uh caused by typ tight coupling and lack of boundaries harm developer product AC ity in addition to architectural problems applications face

operational issues these types of problems are referred to the Practical aspects of running and maintaining applications they affect the overall happiness of your engineering organization an example of an

operational problem is flaky tests and slow CI while the any size application can have flaky tests the larger your monolith is the more likely flakes are to occur and uh the harder it is to

narrow down where they started flaky tests block deploy or makes CI runs take longer due to rerunning failed tests a flaky build adds a lot of friction and frustration to

development often Engineers working on large modol lists will complain that about their CI Suite taking too long while the amount of tests certainly can contribute to longer C CI runs it's

often not the only cause CI may be slower due to a test that creates too many records queries that are ridiculously slow or network calls that were unintentionally added there are a

lot of things that contribute to a slow test Suite but they become more problematic as a monolith an organization grows faster than problems are addressed in addition to slow C when a

monolith gets to a certain size I it's common to hear complaints that it's difficult to scale deployments effectively the larger the application the longer it will take to check out uh the code and restart the servers as an

application gets more popular you need more infrastructure to handle customer traffic because rails a rails is a monolith it's not possible to deploy more servers for one resource intensive

part of the code base lastly organizational issues are at the company leadership uh and structure level they are related to how work is managed how quickly problems get solved and how well an engineering organization is

functioning organizational problems can happen regardless of the size of your monolith or or company however they're exasperated by growth and scaling needs an example of an organizational problem

is difficulty assigning and finding code owners uh in a large application at small companies like especially if you've got only five Engineers everybody's responsible for the entire application however that doesn't scale

as your company gets bigger and you have thousands of Engineers on an all all and call rotation for code they didn't write or and don't understand when an application gets to a certain size you need to split up the responsibility so

the bugs get fixed and you know who to page during an incident another organizational issue is the length of time onboarding new hires takes the argument is that it's hard for new hires to get started shipping

because they can't find their way around a model that a Cod base if that monolith was modular they could in theory just focus on code that belongs to their team coming into a giant monolith especially if you haven't written rails

before can be quite daunting it makes sense that a smaller code base would feel easier to reason about for new hires lastly one argument I often hear for why large monoliths are problematic

is people always say I can't hold the whole application in my head and when I hear this the only thing thing I can think is why are you even trying to do that what I can hold in my head is

different from what you can can hold in your head and what dhh can hold in his head uh you can thank Aaron for this one and Matthew for the little dhh head

um I don't I don't agree this is a real problem nor do I think it's a useful metric for whether an application is a ball of mud realistically as our companies and our applications are more

successful our monoliths are going to grow over time it's important that we focus more on having well-designed properly structured applications that follow Real's conventions than it is to

pick an arbitrary number of lines of code that we can hold full context on having monoliths small enough to hold in your head isn't a tenable or reasonable goal at

scale the other problems we've discussed though are very real I've experienced them at multiple companies I've worked at and I've heard these problems from others that I've talked to in order to solve this set of problems companies

that often reach for microservices and try to carve up their monolith instead Shopify Gusto GitHub doximity zendesk and others are exploring the modular monolith a modular monolith is a

monolith that is organized into modules inside the codebase by grouping logical uh a grouping domains into logical directories the promise of a modular monolith is that in theory it provides

the Best of Both Worlds the isolation of boundaries of microsurfaces was with the ease of deployment and development of a monolith there are a lot lot of reasons to choose a modular monolith over

microservices with a modular monolith you get to use the same language often if an organization is moving towards microservices you are all now writing go when Ruby is likely what you were hired

to write this is not a knock on go it has its place but as rubyist I want to write Ruby so a modular monol lift benefits me and other rubyists in that we get to keep writing the language that we

love in addition to using the same language modular monolith mean you don't have to build out new infrastructure deploys in CI just work the same way as they did before which means uh the overhead for migrating to a modular

monolith is a lot lower and quite minimal at least to start deployments and testing for microservices can be more complex if they're interdependent whereas a modular monolith is deployed and tested as a single

unit another advantage of a modular monolith is that it provides a path towards package isolation isolation means creating boundaries uh between components and attempting to remove dependencies and

coupling in order to Define an boundaries between your modules you need to use a tool like packwork and then have to do the work to remove dependency violations if you want

to modularize your rails application and your goal is to be able to run separate CI jobs for your package or deploy your packages to separate servers uh isolation is going to be required true isolation is incredibly difficult to

achieve and later we'll look take a closer look at why this might not be a path that you want to pursue currently I'm not actually aware of any applications that are deploying fully isolated modules of their monoliths to

to production at Shopify we have a single isolated component that can run CI and isolation but it doesn't contain any product functionality it's essentially our active support of our application

and therefore it's not useful to deploy to its own infrastructure um most of the companies using modular monoliths have implemented them the same way generally speaking in the rails World a modular monolith uses

rails engines to organize code and packwork to Define and enforce boundaries between their packages rails engines are native to the to the framework and packwork is a gem written by Shopify uh to understand how this work

let's say we have an application called fur and foliage that sells both plants and pets uh in a standard rails application the directory structure might look like this

uh if you have an application with four Concepts dog cat tree and flower you have corresponding models in the app models directory controllers and app controllers and so

on if we were to modularize our rails uh with rails engines our application might be organized like this here we have grouped dog and cat into a Pet's engine while tree and

flower are grouped into a PL plants engine can't say plants uh we we call these packages uh I've purposely oversimplified an example how modular

monoliths work at Shopify we have top level components and many nested packages inside and other uh applications are doing that too but I didn't want to like spend all my time

making a fake application about plants and pets for you uh there's also there's many blog posts and talks available online that talk about the decision-making process

of how to uh figure out where stuff goes and how to modularize your rails application uh so I'm not going to go deeper on that uh this talk is meant to be a higher level view of the problems

that we're trying to solve um but a a monolith that's only modularized with rails engines doesn't inherently reduce dependency or create

isolation because all the code is still accessible between packages we can use packwork to identify and enforce isolation between plants and pets as mentioned earlier packwork is written at

Shopify and is what most applications are using to enforce boundaries uh this talk is not an endorsement or criticism of packwork I'm using it for examples because it's the only tool that I'm

aware of that actually does this uh packwork allows you to Define dependencies between packages and en Force boundaries by not allowing Undeclared dependencies so with packwork

dependencies are defined in a yaml file every package has its own package. yl

that contains the metadata uh for a package and uh the packages that it depends on so in this example plants depends on pets this is considered an allowed

dependency um this says it's okay for the pets package to rely on constant in the plants package there's an En uh Force dependency setting which prevents adding any new dependencies when it's set to

strict in addition to the package. yaml

file there is a package too yaml file which declares dependency violations that you want to work on removing if you want your package to be isolated from all other code in the monolith you would need to fix all the to-dos and remove

the allowed dependencies working through all these to-dos can be quite difficult in a large application at Shopify we have a little over 40 level top 40 top level constants to find and most of

those contain nested packages in our core application we have over 90 package to- do files and there's no concerted effort to burn those down we're not even tracking whether we're actively reducing dependencies because it's not a useful

metric for us we've found that enforcing strict dependencies causes a lot of friction for developers packwork is a tool that's useful for knowing what most dependencies between packages but it

can't tell you how to write your code with less coupling or how to refactor existing violations to create boundaries rather than think of packwork as a to-do list think of it more as data for helping you understand your

dependency graph and where changes may be needed because modularization and isolation tools can't tell you how to write better code it can cause new

problems to crop up in our applications it's not packwork Fault by any means attempting to fully isolate and remove dependencies in a large scale monolith is very difficult and can result in Des

undesirable design patterns one problem that I've seen in large monoliths trying to prevent new or reduce existing violations is primitive Obsession an example of Prim of

obsession is passing IDs around rather than active record objects in order to avoid calling constants between packages often developers will load all the rec load the records from the database get

the IDS pass those to another package and then load the records again using those IDs because packwork doesn't see that as a violation to remove this dependency we need to avoid calling cat directly from

the flower model we can do that by making an explicit public interface called something like cat getter and pass the ID to that primitive Obsession makes the code more difficult to follow and can lead to

problems with the database due to unnecessarily complex queries less eff efficient queries and data duplication if one package already loaded cat just to get the ID to pass it

to flower it means that the database is queried for the same data twice while using Primitives might circumvent a dependency violation and reduce coupling the downside is database database performance issues that affect your

customers and patterns that don't utilize the efficiency of active record if you prevent using abstract that Engineers are used to like active record for example you end up with far worse patterns and issues in your application

design and structure one of the benefits of a mon modular monolith is there's no network latency between packages like there would be with microservices however primitive Obsession leads to Performance issues that isn't exactly

better we have to be careful that when we put up guard rails we don't end up encouraging far worse anti-patterns in addition to primitive Obsession a challenge I I've observed is

ownership Obsession this refers to the mindset that code within a package is solely the domain of the owning team leading to resistance against input or oversight from others while establishing

clear boundaries between packages can promote modularity it can also create silos that result in an us ver stem mentality when the codebase should be viewed as a shared responsibility ownership Obsession

results in a selfish mindset as well teams will refuse to fix code in a package they don't own because they don't see it as their problem often the right change is the harder change but they'll avoid touching someone else's

code at all costs often to the detriment of code quality and simplicity drawing strict boundaries within an application can have a negative effect on collaboration the

idea that a package functioning like a small the idea of a package functioning like a smaller rails application is appealing in theory however when this leads teams to prevent others from accessing their code or to defend poor

design choices it becomes a negative consequence of modularity and this undermines collaboration and overall engineering culture another problem that's crept up is

developers being obsessed with putting everything in a new package every Concept in its own package teams often want to create yet another package because they feel like it doesn't fit into the existing ones if you're not

careful about preventing everything from being a new package you'll end up with a code base that's modeled after your org chart it's important to constantly scrutinize whether a new package is

really necessary as humans we love to categorize things and we want everything to fit neatly into little boxes but going as far as to make everything its own category leads to a Fractured code base where related functionality isn't

grouped together it's important to think critically about what concepts you turn into a package you don't want to end up feeling like your monolith is a bunch of microservices in a modularized monolith

that enforces boundaries code duplication often becomes a big problem no one wants to violate or create new dependencies so it's common to see code copied from one package to another when

this happens it becomes more difficult to m maintain duplicated code if one version changes and the other doesn't now you have bugs or maybe you're going to make upgrading rails more difficult the code base becomes bloated over time

and if you're concerned with application design and structure duplicating code is a massive design smell copying code just to avoid a dependency violation should be treated

as an opportunity to rethink what code is being used and why if it truly is shared it should be moved to a package that's meant to have shared code or maybe move it to a gem but definitely

don't copy it to another package that is almost never the correct answer another challenge that I that happens when trying to module Ariz and isolate your rails application is

figuring out how to deal with circular dependencies untangling circular dependencies in a large scale rails application is incredibly daunting and it it's also difficult to prevent new ones from being added if the pets

package depends on plants and plants depends on pets it becomes much more difficult to isolate one of those packages from another to avoid the circular dependency many developers would use Primitives or rest apis or

graphql calls introducing performance regressions along the way to correctly isolate these packages often major refactoring needs to be done to pull that share code and untangle the

mess having explored these problems caused by modularization and isolation let's revisit the original problem set to see how our architecture falls short uh in addressing them the problems we

talked about earlier are ones that many companies site when reaching for modularization and yet if we look at the state of the applications using this architecture today we'll see that none of these problems that we set out to

solve can can be solved by changing our architecture this is because they're human and cultural problems not technical ones at the root no amount of ring moving code around into different directories or migrating to a different

architecture can fix these architectural problems are human problems because our tools can't tell us how to refactor our code to have better organization and structure they can only tell us that there's potentially a

problem so can modularization uh and isolation automatically improve structure and organization of an application no when an application is modularized code no longer lives in the

top level directory but that doesn't mean the packages themselves are well organized and properly structured we still need to put a lot of effort into first figure out where the code should live and then prevent new packages from becoming

disorganized humans are pretty good at categorizing things but we're not always great at choosing the right category simply looking at the name of something doesn't tell us who uses it or where it should live while modularization and

isolation are a way to improve organization and structure it still requires an understanding of how to design software which is something we don't often teach can a modular monolith make our

code automatically less tightly coupled no well while modularization and isolation can help you identify dependencies it doesn't actually reduce

coupling or introduce boundaries unless you refactor the existing code adding dependency violations to the to-do list and setting up allowed dependencies doesn't fix the design of your code it

just tells you where there's uh possibly an undesirable design you still have to understand how to rewrite and redesign the existing code while not deviating too far from the rails way this is a human problem because it requires

educating teams on how to design software for rails while avoiding implementing worse anti-patterns like primitive Obsession operational problems are human problems because they need to be fixed

at the source uh they are caused by ignoring technical debt rather than the architecture being used so while mod modularizing a monolith speed up CI and make test less

flaky no there is nothing about modularization and isolation that inherently improves CI testing flaky tests aren't necessarily caused by having a monolith

they are probably called by uh caused by Network calls creating too many records race race conditions resource contention or leaked State between tests because because flaky tests have many causes

that go beyond architecture modularizing and isolating rails applications won't automatically fix fix your flaky test Suite the same goes for speeding up CI if the reason it's slow is because you

have slow queries or a really large test Suite it's not going to be made Faster by moving our code into smaller directories will modularization allow us

to deploy an isolation not yet uh the existing modularized monoliths in the rails ecosystem are all still deployed as a single unit as if they were a traditional ra monolith as

far as I'm aware no one is isolated a part of their monolith to be deployed separately while remaining in the same code base this sounds like an architecture problem but it's a human problem because untangling dependencies in order to scale deployments and

isolation requires redesigning major parts of an application organizational problems are human problems because they're deeply rooted in our organizations leadership and are quite literally directly related

to what we Define as company culture when you modular Riz your monolith can finding and assigning code owners be easier somewhat but not really it is theoretically easier to assign a

team to an entire folder instead of individual files however just because you know who owns something doesn't mean you can actually get them to do the work if exceptions and deprecations

aren't prioritized or incentivized the owning team will almost always choose to ship features over maintenance work additionally reorgs team renam or pivoting parts of your product can

result in sections of the code base being essentially unowned as your organization grows it's important to be able to Define uh and have ownership however changing your architecture

doesn't actually fix that all it does is contain the code that needs an owner ownership is a culture problem not an architecture one in a modular monolith can or uh

onboarding new hires be easier this is really hard to measure but I'd argue no a modularized monolith is still a rails monolith that runs all the CI tests in one build and is still deployed as a

single Unit A change in one package can still affect another because many packages are tightly coupled so I don't think it's accurate or fair to say that modularization allows new hires to only

consider the domains they own in addition to modular modularized monolith deviates from rails conventions and therefore onboarding someone to your application requires teaching them how to use these tools and how they change

how code is written for rails the tools we use uh to maintain structure and style create friction into development and may not actually put new hires in a position to ship faster setting boundaries in an application doesn't

teach anyone how to write idiomatic rails code it just tells them when they have a violation looking back at the problems that we set out to solve that are still present and all the new problems that we

have you may think that I'm against modularization but where code lives is not my biggest concern what concerns me is that isolation is actually very hard especially in an application uh

especially in a language and application that is designed to be Global it's also hard because these monoliths are 15 to 20 years old are made of millions of lines of code and worked on by thousands

of Engineers of varying experience the reason these problems aren't solved isn't because we didn't try hard enough or because modularization is bad or because we don't have the right tools the reasons we haven't solved

architectural organizational and operational issues is because you cannot solve human problems with modularity the problems we're trying to solve are cultural indicative of dysfunctional engineering organizations

we would have these problems if we instead had chosen to stay in a monolith or migrated to microservices no amount of architecture can save you from an organizational structure that doesn't prioritize code quality fixing technical debt and

allowing Engineers to Pivot when a path clearly isn't working organizations promote and incentivize silver bullets instead of rewarding maintenance and foundational work in order to address

these problems that we looked at today we need to understand their causes how does an application get to the point where it feels like a ball of mud if you've worked at a startup or a Greenfield application you know it

doesn't start out this way if rails created a so-called ball of mud from day one none of us would be here using rails the truth is that an application goes to slowly over time commit by

commit until it feels like development and productivity have come to a screeching halt there's no single cause of problems we often fail to see them until it's too late when there's no time

or ability to fix the problems that we see the correct way we blame our tools we blame rails and we blame each other pressure to ship is one of the many reasons that applications turn to a ball

of mud it's much easier to keep an application Loosely coupled with good structural organization when there's just a few developers adding features and no rush to ship but then you get funding and your stakeholders want to

see a return on their investment or you go viral and your customers uh are mad because you can't handle the amount of traffic and so the only way to keep going is to ship ship ship any maintenance work or technical

debt incurred is ignored because fixing that isn't appro it's not even on leadership's radar over time code becomes tightly coupled because bolting onto existing functionality is easier than pausing to do the work to properly

refactor what's already too entangled as the pressure to ship mounts technical debt grows this is both incurred in the form of ignoring maintenance tasks like rails upgrades and and Gem upgrades as

well as building changes into existing functionality when the priority is shipping over quality and fixing things the right way technical debt has a way of slowly growing silently without being noticed

before you know it you're years behind on your rails and Ruby versions there's monkey patches everywhere and seemingly innocuous changes take weeks to ship fixing exceptions feels near impossible now that there's thousands

hundreds of thousands a day and with all that noise what's one more no one wants to take responsibility for maintenance tasks when the only way to get promoted is to ship features the pressure to ship and the

mounting technical debt incre increases the pressure to hire you need more developers to ship more and the ones you have aren't working fast enough because the company is prioritizing feature growth over code quality new

hires never get onboarded properly no one teaches them rails or how an application should be designed they're thrown into the deep end with no support they feel overwhelmed and like they aren't productive so they end up blaming

the framework new hirers are complaining that Ruby is slow and bad and we should rewrite in another new popular language that's faster like go all they see is technical debt no structure tight

coupling and slow CI shipping too fast and increased hiring of both symptoms of a larger problem that growing organizations have growing problems and those problems are

caused by misaligned incentives as organizations get larger they need to add more layers to leadership to make sure everyone's doing their job properly okrs and kpis become the metric for whether things are going well and the

obvious culture problems that led us to to a state where our application feels fragile and fractur continue to be ignored in order to meet their okrs and kpis managers and VPS need to know who

to assign work to and who's accountable for outages the thinking is if we just knew who was in charge of this code we can measure who isn't isn't doing their job while you do need to know who owns code in a large application it's often

taken too far rather than working together and collaborating as an organization misaligned incentives breed ble breed a bleed breed a BL ba blame blame based culture I should not

have put too many bees in one sentence um that results in siloed teams who want to protect their code from the rest of the organization at all costs the desire to modularize and isolate part partially

comes from wanting to feel like you don't have to think about the rest of the code base however it quickly turns into this code is mind to protect mentality I actually watched this happen at GitHub while I was there uh as the

application and organization grew it felt like it became more important to figure out who to blame than it did to work together on fixing technical debt blame based culture leads to teams wanting to protect their code and

database queries from the rest of the organization so they can prove they weren't the cause of a site outage within the or keeping teams away from your code becomes far more important than collaboration because collaboration

means it's not clear who to blame this blame based culture results in the loss of teamwork and empathy the problems that we looked at today aren't caused by organization structure and aren't solved by

organization structure and isolation because they aren't problems caused by technology or lack thereof they are human problems caused by culture within within an organization and there is no Silver Bullet technology that will fix

that it takes good leaders and a concerted effort to address the underlying challenges and allow applications that allow applications to turn into a ball of mud over

time not all hope is lost though just because we can't solve these problems with architecture changes doesn't mean they're not fixable or avoidable it is not inevit evitable that your application turns to a ball of mud and

your only recourse is to spend years moving files around into different folders in order to address the human problems that we facing we need to improve our developer education we cannot keep hiring developers and

failing to train them on how to write rails the rails way engineering onboarding at most companies is a week that's not enough time for a new hire that's not proficient in rails to really

actually learn rails if we don't train new hires why we use rails how to write rails how to organize code how to write tests how to use the features of the framework how to avoid sharp knives and

how to follow rails conventions we're doing ourselves and them a disservice we cannot expect dependency violations to teach anyone how to design software for rails applications companies who are hiring developers to

write rails need to do more education and part of that education comes in the form of making sure every team has at least one person who knows rails well having whole teams of developers from other languages who were never trained

on Rails can't possibly produce uh welld designed idiomatic rails code at scale it's not their fault it's ours because we failed to train them in addition to education we need indoctrination this is

different from education because it goes beyond the technical basics of the of uh how to write rails we need to evangelize new developers coming into the framework I remember back in the early days we

used to all give like talks about how rails is the way it is and how it develop how it promotes developer happiness and we kind of stopped doing that uh as developers have more and more choices about what languages they can

write we need to show them the joy that rails brings you can be so productive if you're following conventions and you know how to use the framework but when your only experience with rails is an application that's deviated from

conventions and is littered with tech technical debt you might not be able to see the beauty of the framework but without indoctrination without getting our co-workers to fall in love with rails like we did all they see is

another tool another language reals is more than simply a ruby framework and I want others to feel the same Joy I do when writing rails applications leaders need to re

prioritize quality it's easy to fall into the ship faster trap but waiting until shipping is painful to improve quality isn't good for the application or team morale when we ignore technical debt for

a long time it has a tendency to build up to uh to a point where Everything feels impossible I've seen so many organizations wait until it's too late and when that happens we start searching for a silver bullet we invest years in a

path that won't fix the problems we have because it's easier easier than addressing the real problem that we have an engineering culture that values shipping over quality features over bug fixes magic Cur all solutions over

fixing individual problems at their Source part of the quality issues come from incentives that value Shi features or silver bullets over targeted methodical improvements if improving performance cleaning up technical debt

or upgrading rails isn't rewarded the same as shipping features why would anyone in your engineering organization prioritize the grueling on shiny work it's up to leaders to highlight that

work and reward it with promotions and raises we cannot uh possibly expect any Improvement to code quality if refactoring isn't rewarded just because someone doesn't say look at me I made

this better doesn't mean they aren't the ones keeping the lights on leaders need to both highlight and reward good work especially when it's not visible if you decide to modularize your

Ra's application in the future there are some technical challenges to keep in mind as we've discussed it won't fix human andri cultur problems but that doesn't mean that a modular monolith won't help you with some of the problems

in your application and organization first if you modularize start with the least number of packages possible there is no reason to decide every domain boundary upfront because you're going to learn a lot about your

application and product during this process it's also a lot easier to undo modularization if you have just a handful of packages and you decide it doesn't work than if you've got 50 plus packages especially if they're all

nested uh by making less packages UPF front you you also get to avoid package Obsession and ownership Obsession you also don't want to end up with an application design and architecture that

models your org chart uh otherwise you'll spend more time moving files around then you will fixing technical debt when isolating a monolith the focus should be on functional isolation rather

than domain isolation this means that instead of creating strict boundaries between every single concept the focus is on whether where the application seams are so for example if we look if we have our fur and foliage application

we may look in there and find that there's staff admin functionality built into the monolith for running data transitions or looking up customer accounts this is a great example of something that can be functionally

isolated from the rest of your monolith it's not part of the core product and it doesn't affect customers but you still need it to share code or share information by isolating on the

functional level instead of domain level we can be more cognizant of avoiding uh indirection and permi of obsession when modularizing your rails applications be sure to not do it prematurely uh it's hard to say what the

right time is but it's certainly not with just a few thousand uh few hundred lines of code or few few hundred thousand lines of code uh instead of modularizing too early spend time

identifying and addressing technical debt otherwise modular ization can end up hiding poor design decisions for years and introduce worse patterns remember that modularization

does not fix organizational structure nor does it improve existing technical debt it's important to fix problems at their Source uh this means that even if you modularize your rails application you still need to figure out why CI is

slow and Flaky you need to refactor tightly coupled code reorganize poorly structured code teach Engineers how to write and design admatic rails and avoid a blame based Culture by ensuring

maintenance uh work on on technical debt is as valued and incentivized as future work the real heroes in your organization are those who painstakingly work on improving bugs performance issues and ensure that upgrades are done

in a timely manner and remember don't fall for the sunk cost fallacy you should never continue down a path that isn't working just because it feels too hard to turn back it's important to

re-evaluate architectural and Technical Solutions over time what was right a few years ago might not work well today it's okay for your goals to change at Shopify when we started modularizing our monolith our goal was to isolate

packages and run them as separate CI builds deploy them separately have each package owner do their rails upgrade work over time this is proven to not only be really difficult to do it's not really right for our organization we

refocus our efforts on functional isolation and production improving the developer experience and removing um checks that cause more friction than benefits in development it's important

to have a healthy engineering culture that can critically look at what is working and isn't and without feeling like pivoting is indicative of failure so I've been writing rails applications for 14 years and I've seen

a lot of different applications at varying stages in their evolution I've spent a significant amount of time in codebases of some of the very first rails applications ever built and what we we've seen over time

is that engineering organizations scale and applications grow eventually they get to a state where the framework stops bringing joy to uh developers joy and happiness we find ourselves was missing

the productivity we had when we first started building our product as development slows and joy is replaced with friction and frustration we look for someone or something to blame when I

hear developers talk about how Ra's applications turn to a ball of mud they make it sound like it's inevitable it's very common for an application that's millions of lines of code and worked on by thousands of Engineers to feel like a

big mess they say something like rails doesn't provide patterns or tooling for managing the increasing complexity of a monolith but I don't think the challenges that we looked at today are the responsibility of the Rails

framework to solve the truth is the ball of mud is actually caused by an engineering culture that incentivizes shipping over code quality hiring fast over education

and Silver Bullets Over rewarding those working on technical debt why should rails provide tooling for modularization when the complexity lack of structure and friction fill development environments aren't caused by the

framework itself often it seems that we when we reach for modularization we're trying to engineer ourselves out of a large organization we keep trying to find ways

to make a 20-year-old monolith feel like a Greenfield application but it can't because it's not and it shouldn't be it makes sense that as developers we want to reach for technology uh to solve our

problems because that's where we have control but rails can't engineer engineer us out of our problems and neither can modularization or engines or packwork or microservices this isn't

because modularity and isolation is bad it's because is we're trying to solve human and cultural Problems by changing our architecture at Shopify we've been trying to engineer ourselves out of

these problems for 6 years Gusto GitHub doximity zenes are trying as well and the truth is this is kind of Uncharted Territory we're still trying to figure out how to make working on a monolith of this size bring joy to

developers if we look at the current state of our applications we'll see that many of our problems that we set out to solve are ever present and we have new issues that have cropped up we want to blame our tools our Engineers our

framework work but ultimately blame won't help we need to shift our engineering culture to Value what's really important in order to solve architectural organizational and operational issues leaders in our

companies and in our community need to do more engineering culture comes from the top we have to start educating new hires not just on how to use rails but on software design and rails philosophy

as well we need to prioritizing fixing technical debt and invest in code quality over rewarding only those who ship features we need to start talking about why we love rails and indoctrinate

newcomers so they not only keep writing rails but they fall in love with the with it the way we did we need more collaboration in our engineering organizations rather than focusing on blame based

development we need to be curious and Discerning don't fall for thinking there's a magic Cur all solutions to these problems because there isn't the myth of the modular monolith is that architecture cannot fix human and

culture problems but fixing human and culture problems can improve our architectural oper ational and organizational challenges and while AI technology is trying to eat our jobs and livelihoods these human problems are

only going to become more obvious and more important to solve refocusing on education indoctrination and collaboration will ultimately make us more successful at solving these hard problems let's invest in our engineering

culture embrace our monoliths and ReDiscover joy in programming rails at a large scale thank you [Music]