By "stuff insulation in the crack" I assume you used a scrap of fiberglass batt insulation. That's not a super tight seal. Air filters are made out of that stuff; dust collects on the fibers, but the air flows through it easily enough. By really packing enough of that material into a crack, the leakage of air through the crack is reduced but not totally eliminated. It might be useful to apply a strip of blue painter's tape over the crack, on top of the FG stuffed in there. That will provide a better air leakage barrier. In the warmer weather you can do a proper fix.
Air leaks, especially in the upper level of the house, are worse in bitter cold weather, due to "stack effect." Air at zero F is about 15% more dense than inside air at 70 F. That's a relatively huge difference. The taller the column of warm interior air "immersed" (in a sense) in cold outside air, the greater the pressure difference. Its much like diving down into the lake, where the pressure increases steadily due to the weight of the water above any point. Think, too, about the need to have sufficient chimney height above a wood stove to get adequate draft.
Most homes could benefit from a thorough air sealing effort. Typical leakage places are up high, through wiring holes in top plates of the upper story (caulk those with orange can foam), can lights in upper story ceilings (replace those with low profile LED fixtures, sealed to the ceiling), and under the sill at the top of the foundation and where there are any openings to the outside for electrical or plumbing. Also, a chimney often can have a huge open space around it up in the attic.\
There is a wealth of information online. A quick search on "air sealing existing home" turns up many. Here is but one:
https://www.greenbuildingadvisor.com...g-homes-part-1
Reducing the leakiness of a house makes a huge difference not just in heating cost but also in comfort. Air leakage, at its worst in bitter cold and windy weather, flushes out interior humidity produced by human occupancy. Excessive leakage results in exceedingly dry air, so uncomfortable that many resort to active humidification. Needing that almost always is a dead giveaway that the house is far too leaky. Humidification is not the answer; the first step should be tightening up the house. Deliberately adding humidity to a house runs the risk of condensation on cold exterior sheathing and roof decks, with associated mold and rot.