Cream butter, peanut butter, and both sugars in the bowl of an electric mixer fitted with the paddle attachment. Add egg, vanilla, milk, and salt and baking soda. Mix until combined. Slowly add flour and mix until dough comes together.
Scoop 1 tablespoon sized balls of dough and place them on prepared cookie sheets, 2 inches apart. Gently press each ball with the palm of your hand to flatten. Using a spoon, make marks to resemble a basketball, if desired. (You can also make the criss cross marks with a fork instead.)
Assemble the s’mores: turn cookies upside down on a cookie sheet. Place large marshmallow halves on half the cookies and a peanut butter cup on the other half. Broil until the marshmallow is toasty and peanut butter cup is melty, under a minute. Make sandwiches and serve warm.