Two rollers inside the mouse touch the ball. One of the rollers is oriented
so that it detects motion in the X direction, and the other is oriented 90 degrees
to the first roller so it detects motion in the Y direction. When the ball rolls,
one or both of these rollers roll as well. The following image shows the two
white rollers on this mouse:

The rollers that touch the ball and detect X and Y motion
- The rollers each connect to a shaft, and the shaft spins a disk with holes
in it. When a roller rolls, its shaft and disk spin. The following image
shows the disk:

A typical optical encoding disk. This disk
has 36 holes around its outer edge.
- On either side of the disk there is an infrared LED and an infrared sensor.
The holes in the disk break the beam of light coming from the LED, so that
the infrared sensor sees pulses of light. The rate of the pulsing is directly
related to the speed of the mouse and the distance it travels.

A close-up of one of the optical encoders that track
mouse motion. There is an infrared LED (clear) on one side of the
disk and an infrared sensor (red) on the other.
- An on-board processor chip reads the pulses from the infrared sensors
and turns them into binary data that the computer can understand. The chip
sends the binary data to the computer through the mouse's cord.

The logic section of a mouse is dominated by an encoder chip,
a small processor that reads the pulses coming from the
infrared sensors and turns them into bytes sent to the computer.
You can also see the two buttons that detect clicks
(on either side of the wire connector).
Almost all mice used on personal computers use this optomechanical arrangement.
The disk moves mechanically, and an optical system counts pulses of light. On
this mouse, the ball is 21 mm in diameter. The roller is 7 mm in diameter. The
encoding disk has 36 holes. So if the mouse moves 25.4 mm (1 inch), the encoder
chip detects 41 pulses of light.
You might have noticed that each encoder disk has 2 infrared LEDs and 2 infrared
sensors, one on each side of the disk (so there are four LED/sensor pairs
inside a mouse). This arrangement allows the processor to detect the disk's
direction of rotation. There is a piece of plastic with a small, precisely
located hole that sits between the encoder disk and each infrared sensor.
It is visible in this photo:

A close-up of one of the optical encoders that track mouse
motion.
Note the piece of plastic between the infrared sensor (red) and the encoding
disk
This piece of plastic provides a window through which the infrared sensor
can "see". The window on one side of the disk is located slightly higher than
it is on the other -- one half the height of one of the holes in the encoder
disk, to be exact. That difference causes the two infrared sensors to see
pulses of light at slightly different times. There are times when one of the
sensors will see a pulse of light when the other does not, and vice versa.