brickbat (n.)

"a stick or staff used in beating, a war-club, staff used to strike the ball in certain games," c. 1200, from rare Old English batt "cudgel," a western England word at first, probably from Welsh or another Celtic source (compare Irish and Gaelic bat, bata "staff, cudgel"), later reinforced and influenced by Old French batte "pestle," from Late Latin battre "to beat;" all from PIE root *bhat- "to strike." As a kind of wooden paddle used to play cricket (later baseball), it is attested from 1706.

A Middle English sense of "a lump, piece, chunk" (mid-14c.) was used in reference to bread, clay, wool, and survives in brickbat and batting (n.1). 

The phrase right off the bat (1866), also hot from the bat (1870), probably represent a baseball metaphor, but cricket or some other use of a bat might as easily be the source—there is an early citation from Australia (in an article about slang): "Well, it is a vice you'd better get rid of then. Refined conversation is a mark of culture. Let me hear that kid use slang again, and I'll give it to him right off the bat. I'll wipe up the floor with him. I'll ---" ["The Australian Journal," November 1888].

"rectangular block of artificial stone (usually clay burned in a kiln) used as a building material," early 15c., from Old French briche "brick," which is probably from a Germanic source akin to Middle Dutch bricke "a tile," etymologically "a bit, a fragment, a piece broken off," from the verbal root of break (v.).

Of a brick-shaped loaf by 1735. The meaning "a good, honest fellow" is from 1840, probably on notion of squareness (as in fair and square), though in English brick and square when applied to persons generally are not compliments.

Brick wall in the figurative sense of "impenetrable barrier" is from 1886. Brick-and-mortar (adj.) as figurative of "physically real" is from 1865. To do something like a ton of bricks "vigorously" is from 1929 (earlier thousand of bricks, 1836), probably from the notion of how hard such a weight of them would fall or hit.