Modern aircraft, comfortable seats and unflappably pleasant flight attendants.

These are factors many rankings consider in determining the world's best airlines.

But they overlook a crucial element, according to the passenger rights company AirHelp — how airlines treat customers when problems occur.

"We believe that an airline should be assessed not just by how they treat their passengers during business as usual, but also by the consideration they show customers when things go wrong," the report stated.

To compile its annual "AirHelp Score," the organization evaluated 83 airlines based on punctuality, customer opinions, and how efficiently they handle customers' compensation claims.

Each factor was weighted equally in the scoring, said Tomasz Pawliszyn, CEO at AirHelp.

The best and worst airlines of 2023

Based on data from Jan. 1 to Sept. 30 2023, these are the airlines that topped the 2023 AirHelp Score:

1.     Qatar Airways – 8.38

2.     Eurowings – 8.27

3.     LOT Polish Airlines – 8.11

4.     Etihad Airways and All Nippon Airways – 8.09 (tie)

6.     Austrian Airlines – 8.07

7.     American Airlines – 7.97

8.     China Airlines – 7.92

9.     Wideroe – 7.89

10.  United Airlines – 7.88

The 10 lowest-scoring airlines on AirHelp's 2023 list are:

1.     Tunisair – 4.12

2.     British Airways – 5.03

3.     Frontier Airlines and Pegasus Airlines – 5.18 (tie)

5.     Czech Airlines – 5.20

6.     Air Canada – 5.68

7.     Spirit – 5.69

8.     TAROM and Azores Airlines – 5.71 (tie)

10.  Air Austral – 5.77

Qatar Airways has topped every "AirHelp Score" ranking since 2015 — but one. In 2016, the Doha-based airline fell to No. 2, behind Singapore Airlines. In 2022, it tied for the No. 1. spot with Etihad Airways, Pawliszyn told CNBC Travel.

The 2023 full list can be found at AirHelp's website.

Best and worst airlines for passenger claims

The five airlines that scored the highest on compensation claim processing alone — which evaluates how airlines handle claims ("Do they ignore or wrongfully reject claims?") as well as response and payout times, are:

• China Airlines and Brussels Airlines – 8.2 (tie)
• United Airlines, SAS Scandinavian Airlines and airBaltic – 8.1 (tie)

Those faring the worst in terms of claims resolution are:

• British Airways – 1
• Gulf Air – 1.4
• Czech Airlines, Aircalin and Vietnam Airlines – 1.7 (tie)

British Airways may have placed last as a result of being understaffed, said Pawliszyn.

"As one of the largest and most popular airlines, they probably handle a large load of claims," he said.

In response to CNBC Travel's request for comment, British Airways stated: "We're working hard to improve our response times ... We've recruited an extra 1,500 new colleagues to help and introduced new automated technology to resolve easier claims faster, with the vast majority of cases resolved within seven weeks.

The airline added that more than half of outstanding cases relate to EU compensation claims which take time, due to checks that must be performed.

Singapore Airlines, which tops most major airline rankings, came in at No. 26 on the list, dragged down by a claim processing score of 5.4. Another highly regarded airline, Emirates, ranked No. 24 on the list for the same reason.

Claims processing is 'crucial'

AirHelp told CNBC it uses its own data to monitor how well airlines process customer claims for compensation. 

The organization "helps passengers claim compensation following delayed or cancelled flights under several international air passenger rights regulations including EC 261 in Europe and ANAC 400 in Brazil," he said.

"Examples where a passenger may be able to make a claim are flights on a European airline where the passenger arrived over three hours late at their destination, or a flight that was canceled by the airline less than two weeks before departure."

He said claims processing is critical because it directly impacts passenger satisfaction and an airline's overall operational efficiency.

"Swift resolutions over claims like lost baggage, flight disruptions, or other inconveniences could prevent passengers from selecting another airline for their future travel plans," said Pawliszyn. "It's crucial for maintaining a positive reputation and customer loyalty."