April is a good time to go, hopefully before it warms up enough for the rattlers to emerge. I've backpacked there like 5 times, but in different locations; there is so much space in all directions to camp and get a private spot; you don't need to be right at Ancient or Dusty Lake and the crowds.