By Todd Buell

FRANKFURT -- Italy's constitutional referendum Sunday looms as a potential stress-test for Europe's new financial-regulation regime.

Following the U.S. market turmoil of 2008 and the ensuing euro crisis, European Union officials expanded oversight of the bloc's finance sector. The alphabet soup of agencies has spent the past several years staffing up, implementing regulations and grilling banks, insurance companies and other industry players on their liquidity and capital health.

The goal has been to prevent financial institutions from collapsing and ensuring taxpayers don't have to foot the bill to save private financiers. Now Italian voters could show if it all worksas planned, in the event any post-vote market turmoil shakes the nation's vulnerable banks.

Italy's vote on Sunday will decide on a constitutional change designed to facilitate reforms and limit the power of the Senate. Recent surveys favor the "no" camp. Italian Prime Minister Matteo Renzi has said he would quit if the referendum loses, which could lead to political instability. Mr. Renzi's resignation could lead to a caretaker government that would ferry the country to elections as soon as next year. Such a prospect is spooking investors, given the recent surge in Italy's large antiestablishment 5 Star Movement.

Analysts warn that a "no" vote could bring political and financial instability, as market uncertainty and risk-averse investors could make funding conditions tougher for the country's ailing lenders.

That could trigger a showdown between Rome and EU officials over how to apply new rules and result in losses for smallItalian investors. Financial pain for ordinary Italians could turn them further against mainstream politicians.

Italian banks have tested the patience of markets and regulators all year with their struggle to recapitalize. Now, a "no" vote would be a real shock to the system, say some economists. It wasn't supposed to play out like this.

The European Central Bank in 2014 became supervisor of the 19-country eurozone's largest banks, with the aim of harmonizing standards. The next year the eurozone imposed a unified system for winding down failing banks, the Single Resolution Board. This year, Europe started implementing new rules on bank rescues, known as the Bank Resolution and Recovery Directive.

The new institutions follow the opening in 2011 of the European Banking Authority, which sets rules for banks across the whole 28-country EU. The EU also in 2011 established a market regulator, the European Securities and Markets Authority, and an insurance regulator, the European Insurance and Occupational Pensions Authority.

Europe's new system was designed to make banks less reliant on governments if they get in trouble and force market discipline into decision-making. The new regime is working so far, "but it has not been violated openly," said Jan Pieter Krahnen, a finance professor at Goethe University in Frankfurt. "I think of Italy as the first country facing this issue."

A "no" vote entails "significant risk that depositors start to pull their money out" of banks, said Jack Allen, an economist at Capital Economics.

No bank faces more risk than Banca Monte dei Paschi di Siena, the world's oldest bank and Italy's third-largest. The only bank to effectively fail European stress tests last summer, it needs to raise 5 billion euros (about $5.3 billion) in new capital by year-end. Political and market instability would significantly complicate the task.

New EU rules would let Rome help the bank only if investors also take a hit, for example by converting subordinate bonds issued by Monte dei Paschi for a lower value of its battered shares to fill the EUR5 billion hole in its balance sheet. Italian mom-and-pop investors hold EUR2 billion in such bonds.

Forcing losses on voters is "not going to be very popular," said Mr. Allen.

Italian efforts to protect those small investors by circumventing new EU rules risk sparking a clash between Rome and Brussels. EU officials could then face a choice between enforcing their get-tough standards and worsening an Italian political crisis.

Mr. Allen predicted EU officials would reach a compromise rather than aggravate the situation following a "no" vote.

Other observers said markets might not react extremely if the referendum fails.

"There's an exaggeration of the disruptive nature of the 'no' vote," said Nicolas Véron, ofBrussels think tank Bruegel.

Even a "yes" vote wouldn't solve the problems in Italy's crowded banking sector. The country has "more bank branches than restaurants," said investor Alberto Gallo, manager of the Algebris Macro Credit Fund. Intense competition among Italian banks has hurt profits that were already weakened by years of recession and weak economic growth.

Analysts said that whatever the outcome of Sunday's vote, Italy will continue to present challenges for EU financial regulators.

--Giovanni Legorano contributed to this article.

This article was available on Pro Financial Regulation first, before being published to wsj.com. Separate subscription required for Pro Financial Regulation; more information here.

Write to Todd Buell at todd.buell@wsj.com

(END) Dow Jones Newswires

December 01, 2016 16:37 ET (21:37 GMT)

Copyright (c) 2016 Dow Jones & Company, Inc.