Buggies are to blame:

[Senate Historian Don Ritchie] says lawmakers reasoned that Monday was out because people would have to travel to the polls in their buggies on Sunday, the Sabbath (this is where the buggies come in). And in a mostly farming society, Wednesday wouldn't work because that was often market day. So, Tuesday was the day, and that seemed to work great for 19th century voters.