We calculate SaverLife balances using net gain/loss, not the balance in your linked bank account. You earn chances to play and win when your deposits outweigh the withdrawals.
A net gain is an overall improvement of your SaverLife account balance. The SUM of all deposits must be greater than the SUM of all your withdrawals. Any withdrawals or bank fees will count against the needed $5.