When you first install a WordPress website you will be presented with a Getting Started splash screen. This screen has several links that you need to visit to get the basic configuration done and have a clean site to start with.
The first part is all about Basic Settings.
Choose your privacy setting
So if you go to the first link, make sure you select that your website can be indexed by the search engines like Google. I have only encountered one site owner who did not want to appear in Google as it was a very personal blog.
Select your tagline and time zone
The Tagline is the second field of your general settings and should be changed from Just another WordPress site to something that directly reflects the purpose of your site. On this site, the Tagline is How To Set Up Your WordPress Blog & Website.
You can always change it again via your Dashboard -> Settings -> General Settings.
Now the time zone is important for publishing your articles / posts. If you set the publishing date and time you want it to be going live in the target time zone of your audience. Normally that would be your own time zone, but if you want to target the USA then changing to the US CDT zone might be a better idea.
Turn comments on or off
This is very personal, You might want to have visitors comment on your articles, especially if you have a Blog. If you just want to use WordPress as a CMS for a static website you can turn this off.
You can always allow comments back on or give the possibility to comments on a separate post or page.
Fill in your profile
Your profile is sometimes shown on the site by your theme, so you need to give it some attention.
The first thing you need to change is the Display name publicly as you don't want to show the username here. Change it to something better suited. There are already way too many Admin users who leave this to the standard.
On this site you will see the meta info below the post as written By WordPress Site Basics, that text comes from the profile setting.
If you have a Gravatar account your Gravatar is connected to the email address used in the profile settings.
Adding a few lines to your Bio can potentially be a good introduction to your visitors although not all WordPress Themes will support showing it.
Change the Permalinks setting
This is not on the standard list of WordPress, but should be done right away with the other settings. The permalinks option will change the URLl of your articles from ?p=49 to a URL based on our selection like /post-title-name/ which is easier for users as well as search engines.
Go to your Dashboard -> Settings -> Permalinks and select Post name, this will put in /%postname%/ in the custom field but don't worry about that. Using Post name is the new option for permalinks for WordPress 3.3.2 and up. The new option has the best performance in database retrieval and this better for your site loading speed.
I like to change the last to fields as well, normally WordPress will show them as /category/ and /tag/.