All is revealed when Silas takes a bath and tells Ned how years earlier, when he was a teenager, he was swimming nude in the river and a girl stole his clothes. He had to bare everything in order to chase her and regain his trousers and says that, to this day, he has no idea who she was. Mrs. Betts does though.