The big centre-right and centre-left blocs in the European Parliament have lost their combined majority amid an increase in support for liberals, Greens and nationalists.
The centre-right European People's Party remains the largest bloc, and is expected to form a pro-EU coalition.
The Liberals and Greens had a good night, while nationalists were set for victory in Italy and France.
Turnout was the highest for 20 years.
Populists gained ground in some countries but fell short of the very significant gains some had predicted.
In the UK, the newly-formed Brexit Party claimed a big victory, and a strong performance by the Liberal Democrats came amid massive losses for the Conservatives and Labour.
Analysts said the EPP was likely to form a "grand coalition" with the Socialists and Democrats bloc, with support from the Liberals and Greens.
he turnout bucked a long trend of decline in voter numbers, rising to just under 51% of eligible voters across the 28 member states.
The European Parliament helps shape EU legislation and the results will play a big part in who gets the key jobs in the European Commission, the Union's executive.
What do the results mean for the EU?
Based on current estimates, the previously dominant conservative EPP and Socialists and Democrats blocs will be unable to form a "grand coalition" in the EU parliament without support.
The EPP was projected to win 179 seats, down from the 216 in 2014. The Socialists and Democrats looked set to drop to 150 seats from 191.
Pro-EU parties are still expected to hold a majority of seats however, largely due to gains made by the liberal ALDE bloc, and particularly a decision taken by the party of French President Emmanuel Macron to join the group. His Renaissance alliance was defeated by the far-right National Rally of Marine Le Pen.
Media captionItaly's Deputy Prime Minister Matteo Salvini: "Europe is changing"
"For the first time in 40 years, the two classical parties, socialists and conservatives, will no longer have a majority," said Guy Verhofstadt, the leader of the ALDE.
"It's clear this evening is a historical moment, because there will be a new balance of power in the European parliament," he said.
There were major successes for the Greens, with exit polls suggesting the group would jump from 50 to around 67 MEPs.
But gains for nationalist parties in Italy, France and elsewhere means a greater say for Eurosceptics who want to curb the EU's powers.
Matteo Salvini, who leads Italy's League party, has been working to establish an alliance of at least 12 parties, and his party set the tone winning over 30% of the vote, according to partial results.
Who were the winners and losers?
In Germany, both major centrist parties suffered. Angela Merkel's Christian Democrats dropped from 35% of the vote in 2014 to 28%, while the centre-left Social Democratic Union fell from 27% to 15.5%.
In the UK, the newly formed Brexit Party, led by Nigel Farage, took an early lead with 32% of the vote, amid gains for the Liberal Democrats and significant losses for the major Conservative and Labour parties.
The right-wing populist Alternative für Deutschland (AfD) performed worse than expected - projected in exit polls to win 10.5% - while still improving on their first results in 2014.
Amid mixed results for far-right parties across Europe, Ms Le Pen's National Rally party - formerly the National Front - was celebrating victory over Mr Macron's party, securing 24% of the vote to his 22.5%.
A presidential official described the outcome as a "disappointment" but "absolutely honourable" compared to previous results.
In Hungary, Viktor Orban, whose anti-immigration Fidesz party won 52% the vote and 13 of the country's 21 seats, was also a big winner.
"We are small but we want to change Europe," Mr Orban said. He described the elections as "the beginning of a new era against migration".
In Spain, the ruling Socialist party (PSOE) took a clear lead with 32.8% of the vote and 20 seats, while the far-right Vox party won just 6.2% and three seats - coming in fifth.
The Greek prime minister Alexis Tsipras called for an early election after the opposition conservative New Democracy party won 33.5% of the votes to 20% for his Syriza party.
Why was the turnout so high?
EU citizens turned out to vote in numbers not seen since the first EU elections in 1979, bucking years of decline.
The numbers were significantly higher than the last elections in 2014, when fewer than 43% of eligible voters took part. Turnout in Hungary and Poland more than doubled on the previous poll, and Denmark hit a record 63%.
Analysts attributed the high turnout to a range of factors including the rise of populist parties and increased climate change awareness.
What is the European Parliament?
The European Parliament is the European Union's law-making body.
It is made up of 751 members, called MEPs, who are directly elected by EU voters every five years. These MEPs - who sit in both Brussels and Strasbourg - represent the interests of citizens from the EU's 28 member states.
Image captionThe European Parliament is the EU's law-making body
One of the parliament's main legislative roles is scrutinising and passing laws proposed by the European Commission - the bureaucratic arm of the EU.
It is also responsible for electing the president of the European Commission and approving the EU budget.
The parliament is comprised of eight main groups that sit together in the chamber based on their political and ideological affiliations.
