There has been a big drop in scientific plots in science fiction stories since the 1960s, the Golden Age of the Boomers. There have also been fewer philosophical stories since the death of Stanislaw Lem (except by Eliezer Yudkowsky, Scott Alexander, maybe Greg Egan and a few others online).
Most fiction (like World War Z) is bound by genre conventions. In the profitable Potter franchise, why don't the magical creatures rule the world?

Average intelligence seems to be declining while economic growth sputters ons. The most important questions can't be asked. Robotic conformity masquerades as discipline. The education-industrial complex is too self-important to be questioned.
One reason for the conformity is that the earth's population is too small for interesting networks to form. Not everyone can find a community for their interests. That would take a population with fifteen zeroes.
On this planet, we must make do with an average population of forty million per country, who speak on average ten languages, mostly obscure. Making them occupy the same space requires many suppressions.

Most human effort takes place to stop attempted progress. It's safer to stay the same. Change is almost always bad in the short term. There are few resources, and most investments fail. The more precarious the society, the less deviation is tolerated. Limited experiments create new balances after epidemics, wars, famines and the like.

People resist learning new things because humans who evolved to find learning unpleasant have left more offspring.
This conservatism also explains Third World bureaucracies. Getting government permission to do anything in Africa is as complicated as adjusting state boundaries. According to the now very unpopular 'alt-right', lower average genetic IQ rates are more than balanced by many other Q factors, like dominance and passive adaptability.
Life in 'rich' countries is about increasing credentialism. As income lagged, Western civilization became over-regulated.
Bureaucrats are everywhere. Every job requires permits. You need to produce a birth certificate to listen to the radio - or at least to SiriusXM satellite radio, which only accepts bank payments, which require a government ID, which requires a birth certificate. It's no longer possible to build new nuclear power plants or airports. Benefits and entitlements once granted can never be withdrawn, a ratcheting racket of obligations.

Complicated systems commandeer expanding support networks. The interchange of highways around a large airport can seem almost as massive as the runways or terminals, and are as expensive to permit and build.

Perhaps no human can create something genuinely new anymore. Success is rearranging existing resources. Let others spend their lives trying to evolve one good idea. The 'real' genius is exploiting that idea, and it's also more fun.

Solving big problems seems to take forever, and then the difficulties are hidden. After the false leads are eliminated, the resulting model implicitly incorporates all the things not to do. This is for experts only. The final step of simplification is just too hard. Who wants to make the added effort?

The worst example is software. Only a tiny minority of users is capable of moving the output of one program to another, or automating common tasks. Millions of frustrated punters keep performing mind-numbing functions manually, wasting thousands of mice-clicks because they can't tell the computer to complete the job in milliseconds.
Most experienced users only think they are experts, because after many years they've learned to work around the sometimes deliberate barriers. That's just how the real experts like it. As mentioned over nine times in this essay series, these people are evil.

It would take a new derivative science, almost a natural philosophy, to bridge the gap. Knowledge should be oversimplified in the best way.
Special Relativity is never properly explained because it seems uncomfortably strange to normals. But even principles first learned in kindergarten are equally misunderstood: for example the difference between kinetic energy and momentum, the ultimate potential of a lever (the weight of a flea could theoretically accelerate a galaxy to near-lightspeed in nanoseconds in a sufficiently strong gravitational field), or even the laws of motion. Scifi movies always get it wrong.
No one grasps the big picture. Barriers are lowered by making things more complex.

In the distant past, anyone who tried to measure the earth by walking around it would have been killed one hundreth of the way around. World travel wasn't possible until the sixteenth century. The world remains fractured.
The reason Tutsis can't connect to Hutus is that they inhabit different realities. At best, they appear boring to each other. But boredom is a form of stress.
Racism and culturalism are universal because they're easy. An individual merely has to focus their awareness.
In practice, one group tends to dominate the other if they occupy the same space. Eventually they may negotiate a balance, though some thrive on chaos.

Simple emotions drive history. War is mostly about retaliation. During World War 2, when Germany was being invaded from all directions, why did the Nazis keep fighting when they knew they would need to rebuild?
It would have been more painful to surrender and let the victors run wild. Better to weaken them first, so they would have less energy for creative disruption.

Large bombing raids like on Dresden didn't really help, but they felt better than bombing power plants. Instead, a portion of Germany could have been turned over to its victims after its defeat. Israel could have ended up next to Belgium. This could have been an incentive to surrender sooner.

Incentives are strange. Human behavior is controlled by pain aversion, the urge to achieve a lower profile or hide in the tall grass. That doesn't motivate anyone to work harder. Those who have already been punished feel they've earned the right to relax.

At the base of society are families. The most common favor trading involves some coercion. Unexpected pregnancy can radically simplify a sex partner's future.
There's a genetic motive for offspring to be skeptical of their parents (while closely studying how their parents treat others). Parents are biased and manipulative to maximize their own genetic potential, not their offspring's. Traditionally, they often barter one offspring in marriage so another offspring can gain a better marriage partner.
Like most limits to freedom, this problem won't be solved with laws.

The core problem of politics is that any entity with a monopoly on force is likely to use it wrong.
Could there be a better way? The USA is the only country that has been both open yet resistant to outside influences.
In some ways it's been a closed world, existing outside history.
Freedom could be achieved by doing less wrong things, especially by governments. Planned economies have about one tenth the output of owner economies.

* A political solution is to subdivide power among competing groups or individuals, so no one would have too much power.
That would require a widespread desire to be left alone. It might happen if enough people could experience steady wealth accumulation.
It would not necessarily cause a more fragmented society, if people could link up and exchange services in more ways.
The Net could be a shared reality, hiding cultural barriers and differences. In some imaginable futures, there will be only one website, but no one owns it, and it can do anything.

Humans have not evolved to live free. If freedom is forced upon them, they will try to reduce it, subsuming their identity within groups. No group can have too much power, so competing groups keep forming. The best possible human future would be like some eternal soap opera.