Bookmakers are in the business of managing risk. Like actuaries (who would be horrified to be described as bookmakers but effectively do the same job), the bookie assesses the probabilities of an event happening and then adds a profit margin to his odds to give him a cushion against mistakes. That profit margin is where the theoretical profit lies.
Until perhaps 15 years ago most bookmakers created their odds in-house, perhaps taking outside advice on more obscure sports such as snooker or darts and on lower league football, where specialist knowledge was cheaper to obtain externally than employ. Statistics were used to help shape prices but there was a lot of 'feel' pricing, where the odds compilers at each bookie would discuss with each other what they thought was the right price and what the general public's likely wagering activity would be on the event.
Each bookie would come up with their own prices, usually without much reference to each other, and there were fairly regularly enough of a diversion of beliefs that a sharp punter could arbitrage between firms, locking in a profit regardless of the outcome of the game. Really sharp punters could arbitrage between bookies in different countries - for example, in an Ashes cricket series betting England with the Aussie bookies and backing Australia in the UK. On some occasions you could even arbitrage the same bookie - I remember Coral used to run separate trading books between their betting shops, on the racecourse and in their phone betting operation; I remember the joy of finding that you could bet with three different Coral operations and lock in a guaranteed profit on a football match without them realising that they were essentially competing against themselves (it took about six years and substantial losses before they decided that a single trading book across the firm was the only way to run the business, something that their rivals figured out rather earlier).
These days the bookies' trading floor is a very different environment. The 'quants' (quantative analysts) are now the dominant influence, although there is room for some of the old trading expertise. With the vast amount of data now available to analyse most sporting events (thanks to companies such as Opta and Enetpulse), it is much easier to build pricing models to analyse what the odds should be for any game. Crucially, these models also allow the bookie to take the odds of the teams at the start of a match and then to derive an in-running price at all points of the game. These algorithmic trading models allow bookies to have one trader handle perhaps 30 in-running games at once with most bets being accepted automatically below pre-set loss limits and the trader then deciding whether they want to accept or limit a client's action.
The algorithmic models are not perfect by any means and they do struggle to correctly price markets when there are exceptional circumstances - a strong wind blowing in one direction in a soccer match would be a good example - but the amount they save in staffing costs means bookies can afford for them to make the odd mis-pricing mistake while offering customers a vast array of markets to bet on. For example, Bet 365 will now have something like 60 different markets on the average soccer match with markets such as time of next goal, under/over on total corners, first goalscorer in play and so on, all of which are automatically derived from the win price and total goals market, which in turn is derived from the 'market' price set on Betfair and by the Asian handicap markets.
I would guess that the average bookmaker trading department now has about one-tenth of the number of traditional odds compilers that they did 20 years ago and that instead they have been replaced by PhD physicists building trading models or a feed of market prices from some of the third party companies such as Sporting Solutions and Bet Genius that build the trading models on behalf of the bookies. Odds compilers are now effectively traders, keeping an eye on a Betfair screen and making sure the trading models aren't spitting out complete nonsense but also monitoring client activity. Bookmakers now effectively manage their trading risk a great deal by profiling customers, restricting people who look as though they even have a faint clue about what the correct odds should be while letting the complete mugs have as much as they like on. It is a sad fact that for many bookies sports betting is a way of getting customers through the door so they can then cross-sell them online casino and poker, which is basically a 'no lose' proposition for the average bookie.
However it is pointless as a punter complaining about these changes. It is how the market has evolved and while it makes life a bit more boring, it does create some opportunities, as long as you can spot when there is a 'basis change' to an event that a trading model will not have been able to pick up. Getting a shrewd bet on may be harder than ever but it's not yet impossible!