Technology
How rendering partials can slow your Rails app to a crawl
Partials are great. They help with code reusability and keep you sane on those large projects with many moving parts. But they are also a curse. A small one, but one that can get big quickly.
Slower than a turtle
It was a cold November morning, and I was getting ready for another day of remote work. I pressed the spacebar on my laptop keyboard to wake my 13" AirBook, entered the password and looked at the list of my tasks for the day.
The list, or rather the TODO section, had about 15 items in it, all arranged in the order of importance. The item at the top was what I needed to work on next. I clicked on the free space between the title and the meta info and started dragging it into the DOING section. Immediately, I noticed that something was off. Dragging the item wasn't smooth, and when I dropped it, it took about 5 seconds to update the status.
By then, I knew that this behaviour wasn't a once-in-a-lifetime occurrence. It had become a regular annoyance of my everyday route. That cold morning day in November was the final straw. I had to do something about it.
At first, I thought the issue was the javascript. I had described in several other articles how doing javascript incorrectly could lead to all sorts of memory leaks and cause your app to slow down. I fixed those javascript issues, and it improved things a bit.
A Bullet For The Database
But in December, I was still experiencing the same speed problems. Slow to load, slow to update parts of the app. It was becoming unmanageable, especially since several beta users had started to voice their complaints.
I had to fix the issue. The next step, of course, would be to take a look at the queries. Bullet, here we go.
I loaded up the Bullet gem, turned on all the right settings and started fixing queries.
Note: To those who aren't aware of what Bullet Gem is, check out their git repo. Bullet gem looks at your queries and figures out which ones have issues, and suggests what you can do about fixing them.
And boy, did I fix some queries? Tons of N+1 Queries were causing slowdowns. The most simple fix for them was either using .includes(:association_name) or adding counter caches.
What are N+1 Queries? Here's an example. Let's say you have a task partial that shows which project it is part of. Like this:
Project: <%= @task.project.name %>
This causes Rails to go and fetch the project every time it loads a task. Not a big deal if you have one task if you are on the Tasks Details page, for example. But a huge deal if you have 20 tasks all calling that same function.
Task Load (1.0ms) SELECT * FROM "tasks"
Project Load (0.4ms) SELECT * FROM "project" WHERE ("projects".id = 1)
Project Load (0.4ms) SELECT * FROM "project" WHERE ("projects".id = 1)
Project Load (0.4ms) SELECT * FROM "project" WHERE ("projects".id = 1)
Project Load (0.4ms) SELECT * FROM "project" WHERE ("projects".id = 1)
Project Load (0.4ms) SELECT * FROM "project" WHERE ("projects".id = 1)
Project Load (0.4ms) SELECT * FROM "project" WHERE ("projects".id = 2)
Project Load (0.4ms) SELECT * FROM "project" WHERE ("projects".id = 2)
Project Load (0.4ms) SELECT * FROM "project" WHERE ("projects".id = 3)
Project Load (0.4ms) SELECT * FROM "project" WHERE ("projects".id = 3)
Project Load (0.4ms) SELECT * FROM "project" WHERE ("projects".id = 3)
Project Load (0.4ms) SELECT * FROM "project" WHERE ("projects".id = 3)As you can see from this example, Tasks are loaded once, and when you iterate over each one, it also loads a project instance from the database. This is a typical N+1 Query problem.
I had several of these queries. And usually, the fix was simple. Use the includes option on the queries.
Task.active.includes(:project).all
This simple addition reduces the mess from the above query to this:
Task Load (1.0ms) SELECT * FROM "tasks"
Project Load (1.5ms) SELECT "projects".* FROM "projects" WHERE ("projects".id IN (1,2,3))This was a significant improvement from 44ms to 1.5ms.
So things got better for a bit.
By January, however, "slow" became the byword of the day. No matter what we did, loading tasks in our boards was getting slower and slower.
The Profile of Realization
When things get slow, call the profiler.
I knew I had to figure this out. Otherwise, TeamHQ wouldn't be of use to anyone. We had a few beta users, and even though they were complaining a little bit, I knew more complaints were coming.
The next step, get some profiling done. I installed the rack-mini-profiler gem. And the problem was there for all to see. View load times were the issue.
But I already knew that. A quick check of the log output would be enough to realize that the application spent most of its time rendering views.
Here was a typical conclusion of a view render on our active sprint view page.
Completed 200 OK in 3449ms (Views: 3015.3ms | ActiveRecord: 33.7ms | Allocations: 1930660)
It took almost 3.5 seconds to load that page. The application spent only 33 milliseconds getting the data, and it spent over 3 seconds showing it. Something was not right. Notice the allocations count. It's in the millions.
Was I just a terrible Rails programmer? Perhaps, the jury is still out on that. But I had to do something. I had to figure out what was causing this problem.
Closer inspection of the logs and data from rack-mini-profiler revealed the real issue. The truth was not in my inability to program. The problem was in my misunderstanding of how Rails partials work. I had no understanding of how they are loaded, compiled and processed on each request.
My lack of knowledge was the main reason for this slowdown.
I was thinking of partials as simple files that you use to organize your code. Like if you were to break down common methods into modules and reuse them in different classes. I thought of Rails partials as classes.
And this was the wrong way to think about it. Because partials are not like classes. Partials are like methods.
Huh? Yes, partials are like methods.
But first, let me put forth a simple (a very simple) explanation of what a class is and how it works.
A class is a collection of instructions that take in data, do something with it, and provide an output from that data.
On Rails boot, the app looks for all class definitions, compiles the code and prepares it to be used to service browser requests. When a request comes in, Rails instantiates the class from memory and makes it do what it is supposed to do. This makes the majority of Rails quick and responsive.
Views, on the other hand, work a little differently.
An in-depth view at a View
My initial understanding of views was minimal. I actually never thought about how Rails processes and renders views. Everything just worked, and I was happy to leave that area of the framework unexplored.
Necessity, as they say, is the mother of invention, but in my case, necessity was the mother of learning.
You can do a deep dive into how Rails renders views, but what you really need to understand is the following. When a request is finished, Rails figures out which template to show, goes to the disk to find that template, loads it from a file, compiles it and runs the template. In development mode it happens on every request, but in production the loading and compilation part are usually done once.
Sounds pretty simple, right? And on the surface, it is. But in reality, there is a lot of code to make sure everything works as it should. (This part of the post was inspired by the talk Joel Hawksley gave at RailsConf last year)
Probably the most important part of this process is the compiling part. Here's the code from Rails that handles this part of the process:
def render(view, locals, buffer = ActionView::OutputBuffer.new, add_to_stack: true, &block)
instrument_render_template do
compile!(view)
view._run(method_name, self, locals, buffer, add_to_stack: add_to_stack, &block)
end
rescue => e
handle_render_error(view, e)
endThe two interesting parts about this code are the compile! and run the methods.
The compile! method loads the code from your partials and turns it into ruby. So this ERB + HTML mix:
<div>Task: <%= @task.name %></div>
Becomes:
@output_buffer.safe_append='<div>Task:'.freeze; @output_buffer.safe_append=( @task.name ); @output_buffer.safe_append='</div>'.freeze; @output_buffer.to_s
What is this stuff you say? Well, this is what becomes of your partial code. It gets turned into a ruby string.
This string of code then gets constructed into a method. Something similar to this:
def #{method_name}(local_assigns, output_buffer)
# code from above
endThe method name is generated from the rubified path to the partial or template with a unique identifier. It becomes something like this:
def _app_views_tasks__show_html_erb__33939483943493_4393943(local_assigns, output_buffer) ## code from above end
Now all of this is just a bunch of strings that were generated by Rails compile! method. After that is done, Rails calls a module_eval method to process these strings and turn them into bonafide ruby methods.
It then runs these methods passing along the right context, variables and receiving the HTML output to be sent back to the browser.
This is a very brief dive into how Rails partials work. Rails does many more things to make sure every render works.
The main takeaway here is that your partials are turned into ruby methods and executed by the Rails render routine. This means that essentially every template and partial becomes a compiled method that is part of ActionView::Base.
In most cases, this works out well and usually, and there is no problem. But when your application grows in complexity, using many partials can become a significant drawback.
2 Many Partials
And as TeamHQ grew in complexity, I did exactly that - created too many partials. I was excited about organizing my code properly and ended up splitting up my partials for Tasks, Projects, and Boards into as many sub-partials as possible.
I started with the normal rails approach. Something similar to this:
You have a model Task. In your views folder, you define a partial to display that model.
app/views/tasks/_task.html.erb
You do the same thing for Projects, Boards and other models. And I think many people would stop at that. But I did not.
Because my Task partial was getting too big with many if statements to handle all the different conditions and other intricacies, I wanted to split my partials even more. So it turned into something like this.
app/views/tasks/_task.html.erb > app/views/tasks/shared/_meta.html.erb > app/views/tasks/shared/_info.html.erb > app/views/tasks/fields/_name.html.erb > app/views/tasks/fields/_description.html.erb > app/views/tasks/fields/_status.html.erb > app/views/tasks/fields/_priority.html.erb > app/views/tasks/fields/_impact.html.erb > app/views/tasks/fields/_subtasks.html.erb > app/views/tasks/fields/_comments.html.erb > app/views/tasks/fields/_hours.html.erb > app/views/tasks/fields/_due_by.html.erb > app/views/tasks/fields/due_by/_display_editable.html.erb > app/views/tasks/fields/due_by/_display_nonediable.html.erb > app/views/tasks/fields/_start_on.html.erb > app/views/tasks/fields/start_on/_display_editable.html.erb > app/views/tasks/fields/start_on/_display_noneditable.html.erb > app/views/tasks/fields/_assigned_to.html.erb > app/views/tasks/menu/_active_menu.html.erb > app/views/tasks/menu/_completed_menu.html.erb
And this is not the full list. I had partials loading all manners of things. And not just for tasks. Boards had 5 to 6 different partials that loaded additional details and functionality. Boards loaded Sprint partials, Sprint partials loaded bucket partials, bucket partials loaded list partials, and list partials loaded Task partials, which in themselves loaded other smaller partials.
That was hard to write and read. Basically, I had a lot of partials being loaded.
Each partial took between 3ms to 8ms to load, and if you added all of them together, you'd get page load times measured in seconds.
Realization dawned on me that I needed a better way to deal with this complexity.
One of the solutions was to reduce the number of partials - combine partials into bigger files. Another solution was to use caching and some of the Rails' tools, like partial collection options.
And I've done all of them. It sped things up a bit. But still, the complexity of the application was becoming difficult to manage.
Combining partial files into one giant partial may reduce the number of partials, but it still wouldn't solve the problem of speed. This is because of the issue of memory allocation.
Allocation Trap
Allocations are a measure of how many times an object is created using Object.new method. If you look at your code, you'd wonder where are all these allocations are coming from. There are few, if any, .new method calls. But that's not the only way you are creating objects. Many actions will create objects, including strings and arrays.
When you create a string using quotes, it creates a new object. Partials are ready-made for string creation allocations hell. Every <%= %> and every bit between <%= %> is turned into a string and creates a new String object.
Such a simple template like this:
<h1 id="<%= dom_id(@task)%>" class="border <%= 'border-top' if @task.active? %>"><%= @task.name %></h1>
This simple code can create up to 7 new objects. And the more of these <%= %> you have, the more objects you create. If your partials have lots of if statements or print out a lot of data, your allocation counts will be high. And the more partials you have, the more 'partial' methods Rails will create. This in itself requires more memory and results in more allocations.
This means that combining partials into bigger files doesn't really solve the problem.
I was faced with these realizations, and there were few things I could do with Rails default tools to solve the problem. (Caching always has to be the last resort).
There had to be a better way. And then I found the ViewComponent gem.
To View or Not To View, That's Not The Question
The solution turned out to be a brilliant one: ViewComponents.
Rather than creating partials to reuse bits of your views, the best way forward is to understand what components are and how they solve the problem of speed, reusability and testing—all in one glorious, efficient package.
From the ViewComponnet website:
ViewComponent is a framework for building reusable, testable & encapsulated view components in Ruby on Rails.
And that's exactly what I needed to solve the problem.
First, what are components?
A component is a piece of code that takes data in, processes it somehow and provides an output, just like a class or a view for that matter. A component is a class for all intents and purposes.
A ViewComponent is a class that takes in data, processes it and generates a view that you can then show to the user.
One thing about components is that they are pretty much stand-alone. They don't depend on anything other than the data that you give them and the code that you write to process that data.
In the ViewComponent framework, components live in their own directory app/components, and you can use them to render pieces of your views in the same way that you use partials.
# Partial render <%= render "tasks/name", task: @task %> # ViewComponent render <%= render Task::NameComponent.new task: @task %>
Both bits of code take some data in and produce some HTML output. But the difference lies in how they work.
While views are complied into ruby code and executed at runtime, ViewComponents are classes that are loaded into memory at Rails boot time and are then called when needed in the view templates.
ViewComponents can use HTML files to render your views. These files are just like partials. In many aspects, components function similarly to partials.
But there is one difference. ViewComponents are loaded at boot time, processed, optimized (unused memory allocations are cleared up) and are ready to be used during runtime. Partials go through the same steps at runtime, all the time.
This makes ViewComponents much faster than partials, up to 10x faster.
Since ViewComponents are standalone, meaning they are not dependent on other parts of the application, you test them similarly to Rails models.
A ViewComponent takes in some data and produces an HTML string. This HTML string can then be verified using standard assert methods.
Switching to ViewComponents
I spent a better part of the last month moving partials into components. The result has been an astounding success.
On pages that used to load 20 partials x the number of tasks, speed improvement has been amazing. From average response time of 1.5 sec in production to under 400ms was a huge improvement. Simpler pages now load in 120-140ms average time.
All of these improvements have been made without caching or increasing the server speed or RAM.
Conclusion
Rails partials are awesome, but how you use them can become a tremendous burden on your application. The best solution is to use ViewComponents to encapsulate parts of your views into quick, reusable and testable bits.
Learn more about ViewComponent here.
Read my first article, Introducing ViewComponent.