We have studied the effects of methotrexate (MTX-Glu1) and the polyglutamate derivatives of methotrexate (MTXPGs) with 2, 3, 4, and 5 glutamyl residues on the catalytic activity of thymidylate synthase purified from MCF-7 human breast cancer cells and on the kinetics of the ternary complex formation by 5-fluoro-2'-deoxyuridine 5'-monophosphate, folate cofactor, and thymidylate synthase. MTX-Glu1 exhibited uncompetitive inhibition of thymidylate synthase when reaction kinetics were analyzed by either double reciprocal plots or a computerized mathematical model based on nonlinear least-squares curve fitting. The Ki for MTX-Glu1 inhibition was 13 microM and the I50 was 22 microM, irrespective of the degree of polyglutamation of the folate. In contrast, the polyglutamated derivatives of MTX all acted as noncompetitive inhibitors. The MTXPGs had 75-300-fold greater potency than MTX-Glu1 as inhibitors of thymidylate synthase catalytic activity, with Ki values from 0.17 to 0.047 microM for MTX-Glu2 to MTX-Glu5, respectively. Neither MTX-Glu1 nor MTXPGs promoted the formation of a charcoal-stable ternary complex with thymidylate synthase and 5-fluoro-2'-deoxyuridine 5'-monophosphate. CH2-H4PteGlu5 (where PteGlu represents pteroylglutamic acid) was found to be 40-fold more potent than CH2-H4PteGlu1 in participating in the formation of a ternary complex, and 10 microM MTX-Glu5 significantly inhibited the formation of a ternary complex containing this folate as cofactor. The inhibition was determined to be due to a reduction in the kon. The potency of this inhibition was markedly greater in the presence of CH2-H4PteGlu1 as compared to CH2-H4PteGlu5. This finding suggests that the degree of interference with complex formation in intact cells would depend on the state of polyglutamation of available folate cofactor. Ternary complex formation with H2PteGlu5 as the folate cofactor was also investigated, and a 50% reduction in complex formation was found in the presence of a 2 microM concentration of MTX-Glu5. These findings have significant implications regarding the mechanism of action of MTX-Glu1 and contribute to an understanding of the complex interactions of MTX-Glu1 and 5-fluorouracil.