Because the Dursleys were told by Dumbledore that they had to adopt the child. And if we don't adopt him, their family, as Harry's relatives, will be implicated and in danger. For Harry's future, if Harry is not at the Dursleys' house, he will be in danger. Dumbledore forced them to do so. They are Harry's only relatives, and they are sensible. Even in Muggle society, they must adopt him! I hope my answer will satisfy you!