The attackers arrived on motorbikes and started shooting in to the air, causing people to flee, witnesses said.
They targeted the Government Science Secondary School - where more than 800 students are said to reside - in Katsina state on Friday evening.
More than 200 students have been rescued, while the army and air force have joined the search for the missing.
Residents living near the all-boys boarding school in the Kankara area told the BBC they heard gunfire at about 11pm on Friday (local time), and that the attack lasted for more than an hour.
Security personnel at the school managed to repel some of the attackers before police reinforcements arrived, officials said.
In a statement on Saturday, police said that during an exchange of fire, some of the gunmen were forced to retreat. Students were able to scale the fence of the school and run to safety, they said.
However, witnesses said they saw a number of students being carried away.
One police officer was taken to hospital after being shot and wounded, police said.
Several local residents on Saturday said they had joined the police in searching for the students who remained missing, while many parents said they had withdrawn their children from the school.
"The school is deserted, all the students have vacated," one witness, Nura Abdullahi, told AFP news agency.
"Some of the students who escaped returned to the town this morning, but others took a bus home," he added.
Katsina is the home state of Nigerian President Muhammadu Buhari, who is currently there for a week-long private visit.
The attack on Friday came two days after the kidnapping of a village leader and 20 others in another part of the state.
In 2014, more than 270 girls were kidnapped by the militant Islamist group Boko Haram from a school in the north-eastern Nigerian town of Chibok.
No group has yet said it carried out the raid on the school in Katsina, which is far from Boko Haram's usual area of operation in the north-east.