Banks and ATM's can be found just about anywhere in New York. Most banks are open Monday-Friday from 9am to 5pm with some offering limited hours on Saturday. ATM's are easily found wherever you go, charges for using other bank's machines an fees for international accounts if you are still using your home countries bank may apply. Different banks offer different benefits and monthly charges, so shop around to find the bank that best meets your needs.

Typically, there are Savings and Checking accounts. A savings account usually offers a higher percentage of interest per month, while the checking account is where money is automatically taken out unless otherwise specified.
Debit cards are frequently used and accepted almost anywhere. When making a purchase with your debit card, it is immediately deducted from your account.
Credit cards are also frequently used, and work a bit differently. When a purchase is made, it is in the name of that account and the money is not automatically taken out. At the end of the month it is expected that you pay a lump sum for all the purchases made on that card for the month.
Checks are another method of payment which are used less frequently and may not be accepted at some stores and restaurants. When purchasing by check, the store gets the check and you keep the receipt of the check. The store turns the check into their bank and several days later the money is withdrawn from your account.

Several trust-worthy banks are :

Bank of America
261 Broadway
New York, NY 10007
(800) 432-1000

Citibank
50 Avenue A
New York, NY 10009
(212) 627-3999

The Bank of New York Mellon
One Wall Street
New York, NY 10286
(212) 495-1784

For a list of banks:
http://www.newyorkcitymetro.com/banks.htm