I don't bother spelling magic with a k. Some people do; in an attempt to designate sleight of hand from real magic. To me, that is an artificial distinction. Part of what the sleight of hand section of magic does, is to create an altered state of consciousness in which abilities which we might not normally or easily access have an easier time manifesting. To me, that entire process, from the beginning of the opening of ourselves to connect with the universe to the final changes we make, is magic.
Magic does not work against the laws of the physical world, it is part of them. I think that on the surface it most closely resembles quantum physics; and that can be an intimidating thing to study and work with. Intimidating; not impossible. It can also be very simple. When you smile, people smile back. You have affected the world. There are millions of examples; the magnitude changes, the principles are the same.
One of the basic Gardnerian beliefs is that magic works. This does not mean that you need to work magic. I know folk so arch-conservative that they would possibly argue that smiling at someone is an attempt to thwart their will (the logic of this would be: if they hadn't intended to smile, and you deliberately sent them energy and it affected them that sending constitutes an uninvited interaction of energy; therefore it is an attempt to thwart their attitude; and because you haven't been invited, it is an attack, and therefore unethical).
I don't go that far; in that I do not see smiling at someone as an attack. However I will agree that affecting someone against their will is not a bright idea. I have two reasons; Firstly to do something to anyone who has not asked for it is rude, and an invasion of privacy at best. It also may not be wanted; there are plenty of folks who bemoan their problems and do not solve them. Part of them doesn't want that problem solved. If you attempt to do so, they may work to thwart you. Ultimately the easiest way to avoid that is to give them information, and let them determine their actions.
Which leads to my second reason. I'm lazy. It takes energy to affect things magically, just as it does physically. Sometimes more than it does physically. Add to that the normal resistance of a person (everything has inertia, and this is true both on the physical and on the spiritual plans). Getting permission allows you to bypass all that natural resistance. Getting a 'no' lets you show respect, and save your energy for where it will be needed. Trust me; you are needed. It's just a matter of finding where.
To go back to the core thing; to be Gardnerian, you don't even have to do magic. You just have to realize and accept that it exists. To know that you could.