Labor leader Anthony Albanese says the Prime Minister must weigh up whether it is appropriate for the Minister at the centre of a historical rape allegation to remain in Cabinet until the matter is investigated.