Understanding Energy going upstairs and downstairs

So I've been doing my physics report and stumbled on a question that made me lose my mind a bit.

https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhjoWA37QMQLc6bl4HQXUJugYA-syF6aiprL5O65gub9iwuU70Q3-0FeFpvqP0mZrJvo4dxbuC9Khg-n9aIs6mYRaGkbPI2dygAzV22STkYIuEa5LvV9Kak3-HT2EBpd13XHQRB3EOxzvn8/s320/confused-man.jpg

Why is it more difficult to go upstairs than downstairs?

It seemed easy at first but when you realized that work going downwards is negative... it's a bit of a headache. Negative work, or energy, means that you are GAINING energy. Sounds simple enough. In practice, wouldn't that mean that going down the stairs would replenish our energy? (We won't get tired, instead get energized). I still don't understand this and ended up with the conclusion (Just for the sake of finishing the report:

W = mass x gravity x distance. Since going up means that we are opposing gravity, it means that gravity is applying 9.8m/s^2 acceleration to us. If mass = 10kg and the distance is 10m then that means that Work = 980J. Going down means that we are working with the help from gravity, which means that gravity is -9.8m/s^2. This would mean that we are doing -980J of work. By that understanding, in theory, going upstairs is infinitely harder than going down since going up we spend energy and going down we gain energy.

 If anyone could explain how work upstairs and downstairs REALLY work, please don't hesitate to comment.

Reference:
https://www.slackerastronomy.org/wordpress/2006/10/energy/