TONY CONZA had dropped out of college and was working as an order clerk for E.F. Hutton when he went to a party in Jersey City. It was 1964. Among the guests were two high school friends - Peter DeCarlo and Angelo Bandassare - whom he hadn't seen in a while. After a few drinks, they began to bat around the idea of starting a business.

But what business? One of them mentioned a sandwich shop in Point Pleasant called Mike's Submarines. Whenever they visited the Jersey shore, the shop seemed packed. Why not open a sandwich place and make subs as good as Mike's?

Great idea, they agreed.

There was only one problem: Even though they had often passed by Mike's, they had never gone in. How good were Mike's sandwiches, really? One Saturday they drove to Point Pleasant and gorged themselves. They liked what they ate.

Next, they borrowed $2,000 from a friend and $500 from the man who ran the jukeboxes in Jersey City and opened the first Blimpie in Hoboken, N.J. They served the exact sandwich that Mike did. He didn't mind; he even gave some advice.

For the past 23 years, Blimpie has been assembling submarine sandwiches in the same, simple way - it now puts together 8.5 million a year - without doing anything spectacular like driving McDonald's into the tombstone business and without serving any hot food other than microwave sandwiches, soup and coffee.

But they've been 23 very hard years. At times, it looked like Blimpie would be no more. Even now, in Manhattan, home to about 60 of Blimpie's 300 stores, the chain is battling a reputation for unsanitary facilities and freewheeling franchisees.

For years, Blimpie management has been admittedly lax in deciding who to sell franchises to and how it supervised the stores. In a business that thrives on uniformity, it has found itself with a number of renegade owners who, rather than make sure employees wore Blimpie uniforms, let them walk around in T-shirts and jeans. Instead of abiding by Blimpie's limited sandwich menu, they have put in pizza ovens or have sold Chinese food.

''Imagine the fit a customer would throw if he walked into a McDonald's and discovered Lower Slobovian food?'' a Blimpie franchisee said. ''But stuff like that happens to Blimpie customers.''

Blimpie is trying to put such problems behind it and struggling to grow after a few years when it tried to keep a profile as low as a slice of cheddar.

International Blimpie Company headquarters is a warren of small, untidy rooms squeezed into the sixth floor of a building on the corner of Broadway and Astor Place. The international bit is speculative, since there aren't yet any Blimpie's outside the United States. (Technically, Blimpie is the main division of the Astor Restaurant Group.) Mr. Conza, who eats about two Blimpie sandwiches a week, is chairman and president. Since the Blimpie is billed as ''America's Best Dressed Sandwich,'' Mr. Conza feels obliged to dress well, too. The other day, he had on a well-pressed blue shirt and a smart-looking paisley tie.

''Even though we've been in business 23 years, I think our product is more timely than ever,'' Mr. Conza said. ''People are more conscious of what they eat. We say to the consumer, 'You're probably tired of eating greasy hamburgers and chicken, so come to Blimpie.' ''

HE is hopeful that Blimpie is once again on a sustained growth track. For most of the 1980's, the chain was stuck at about 200 Blimpie restaurants. Last year, however, 40 opened, and another 60 have appeared this year, bringing the total to nearly 300. Still another 100 are scheduled to join the chain in 1988.

The Blimpie name, incidentally, came about somewhat haphazardly. Mr. Conza didn't want to call his sandwiches subs because he thought that name implied a greasy spoon. He was perfectly content to call them hoagies, the name Philadelphia used for subs, but no one in Hoboken knew what a hoagie was. So Mr. Conza pulled out a dictionary and thumbed through it word by word. When he got to ''blimp,'' he stopped. ''Blimp,'' he thought, sounded like a sandwich (what's more, if he took it he didn't have to page through thousands of additional words). He tacked on an ''ie'' and had his name.

Blimpie never had trouble selling its sandwiches. The original store moved a lot of sandwiches right from the start, Mr. Conza said. Patrons asked about starting franchises. One was sold the first year for $600 to a friend in West New York.

But there were a lot of ups and downs and detours. In 1965, Mr. Bandassare left to start his own food supply company which is now one of Blimpie's major New York suppliers. Meanwhile, Mr. Conza and Mr. DeCarlo planned a Blimpie invasion of New York, and began it with a store on 55th Street near Carnegie Hall.

By 1967, when there were 10 Blimpie's (four owned by Mr. Conza and Mr. DeCarlo) everything began to veer out of control. Mr. Conza and Mr. DeCarlo admitted they weren't skilled businessmen, and they had been incautious about the cost of goods and employee salaries. The sandwich shops were generating ample sales but disappearing profits.