Preheat your oven to 325°F and grease your baking pan(s)—either one 9x5 inch loaf pan for a large loaf or two smaller pans for mini loaves. While the oven preheats, prepare the crumble topping by combining the sugar, cinnamon, and butter in a small bowl, stirring until the mixture resembles wet sand with small clumps. Place this mixture in the freezer so it stays cold and maintains its crumbly texture until you're ready to top the batter.
Toast the chopped walnuts in a dry skillet over medium heat for 3-4 minutes, stirring frequently, until fragrant and lightly golden—this deepens their flavor and adds a subtle nuttiness to the bread. While the walnuts cool slightly, whisk together the flour, salt, baking soda, and nutmeg in a separate bowl to evenly distribute the leavening agents and spices throughout.
In a large mixing bowl, whisk together the brown sugar, vegetable oil, room-temperature eggs, buttermilk, and vanilla until well combined and smooth. The room-temperature eggs will incorporate more easily and create a better emulsion with the oil, resulting in a more tender crumb. This takes about 1-2 minutes of steady whisking.
Pour the dry ingredient mixture from Step 2 into the wet ingredients from Step 3 and gently fold together with a rubber spatula until just combined—do not overmix, as this can develop gluten and make the bread tough. Once the flour is mostly incorporated, fold in the diced rhubarb and the toasted walnuts from Step 2, distributing them evenly throughout the batter. I like to fold the rhubarb in gently to avoid breaking the pieces and keep them suspended throughout the loaf rather than settling to the bottom.
Pour the batter into your prepared pan(s), filling them about two-thirds full to allow room for the bread to rise without overflowing. Remove the crumble topping from the freezer and evenly distribute it over the surface of the batter, pressing it down gently so it adheres but doesn't sink into the batter. The cold topping will help create a defined crumbly layer as it bakes.
Bake at 325°F for 60 minutes if making a large loaf, or 35-45 minutes for mini loaves, until a toothpick inserted into the center comes out clean or with just a few moist crumbs. The bread is done when the crumble topping is golden brown and the sides have started to pull away slightly from the pan. Let the bread cool in the pan for 10-15 minutes before turning out onto a wire rack to cool completely—this resting time allows the structure to set so it won't fall apart when you remove it from the pan.