The operation also included airstrikes, which local health officials said killed and wounded dozens in the southern Gaza city.
The Israeli military identified the freed hostages as Fernando Simon Marman and Louis Hare.
The two men were kidnapped by Hamas on Oct. 7 along with around 250 others, according to Israeli tallies.
Israel has responded with a military assault on the Gaza Strip that has killed more than 28,000 Palestinians, according to the Hamas-run health ministry.
Residents of Rafah contacted by Reuters said Monday's attacks caused widespread panic.
They added that planes, tanks and ships took part in the strikes, and that two mosques and several houses were hit.
Some feared a ground offensive into the city had begun.
The Israeli military said on Monday it had conducted a "series of strikes" on southern Gaza that had now "concluded," without providing further details.
A military spokesman said Marman and Hare were extracted from the second story of a building by Israeli forces under the cover of an airstrike.
The military said they were in good condition and had been taken to a hospital near Tel Aviv.
Israeli Prime Minister Benjamin Netanyahu said in an interview aired on Sunday that "enough" of the 132 remaining Israeli hostages held in Gaza were alive to justify Israel's war in the region.
His office has said that it had ordered the military to develop a plan to evacuate Rafah and destroy four Hamas battalions it says are deployed there.
Hamas-run television on Sunday quoted a senior Hamas leader as saying any ground offensive in Rafah would "blow up" the hostage-exchange negotiations.
The same day, Egypt warned of "dire consequences" of a potential military assault on Rafah, which lies near its border.
Aid agencies say an assault on Rafah would be catastrophic as it is the last relatively safe place in an enclave devastated by Israel's military offensive.
