The Observable Universe and Beyond

Dark Energy + 2025 Update

The other problem with modern cosmology is related to the use of the Type Ia supernovae as "standard candles" to measure the distance of remote objects. The measurements imply that the cosmic expansion is accelerating as shown in Figure 02-10h, which shows that the

		supernova appears to be dimmer than expected from an uniformly expanding universe. It is proposed that there is some kind of repulsive "dark energy" to induce the acceleration. Figure 02-10i shows the proportion of the various matter-energy components in the Universe. Most of the matter-energy content is in the form of "dark energy". The composition of the Universe is listed in Table 02-02 below.
Figure 02-10h Supernova Ia	Figure 02-10i Energy-Matter in the Universe	New data in 2006 further refine the universe's contents to: 4% ordinary matter, 22% dark matter, and 74% dark energy.

According to NASA in 2012, the mass composition of the universe can be broken down into: Heavy Elements 0.03%, Neutrinos 0.3%, Stars 0.5%, Free Hydrogen and Helium 4%, Dark matter 23%, and Dark Energy 72% (where is the remaining 0.17%?).

Material	Representative Particles	Particle Mass or Energy (ev)	No. of Particles in Observed Universe	Probable Contribution to Mass of Universe	Sample Evidence
Ordinary matter	Protons, electrons	10⁶ to 10⁹	10⁷⁸	5%	Direct observation, inference from element abundances
Radiation	photons	10^-4	10⁸⁷	0.005%	Microwave telescope observations
Hot dark matter	Neutrinos	< 1	10⁸⁷	0.3%	Neutrino measurements, cosmic structure
Cold dark matter	Supersymmetric particles?	10¹¹	10⁷⁷	25%	Inference from galaxy dynamics
Dark energy	Scalar particles?	10^-33	10¹¹⁸	70%	Supernova observations of accellerated cosmic expansion

Table 02-02 Composition of the Universe

Acceleration of the cosmic expansion is placed on a firmer footing when it is observed in 2003 that the CMBR becomes slightly hotter after going through a galaxy, which forms a gravitational (potential) well. Dark energy, being gravitationally repulsive, makes a gravitational well shallower as a photon passes through, so the photon exits with slightly more energy than it had when it entered.

Another method to verify the cosmic acceleration is by detecting the bunching intervals of the clusters of galaxies. Whereas Type Ia supernovae behave like standard candles, the spacing between clusters of galaxies acts like a standard ruler. The bunching was generated by the cosmic sound wave, which compressed matter to higher density at its peaks. According to different scenarios of cosmic expansion, the amount of stretching is different as shown in Figure 02-10ja. In the primordial gas, the incoherent acoustic oscillations created peaks at intervals of 436000 light years, today the spacing should be about

Figure 02-10ja Dark Energy and Sound Wave

500 million light years depending on the kind of cosmic model (see Diagram b, Figure 02-10jb).

It was announced on March 30, 2012 that the Baryon Oscillation Spectroscopic Survey (BOSS) has completed a massive survey of 327,349 galaxies out to about 6 billion light-years away. These galaxies are used as the standard rulers inside the clusters of galaxies to check out the cosmic acceleration. The survey confirms the 500 million light years "peak separation" at the present epoch, and estimates the transition to dark energy domination at 5 to 7 billion years ago (~ 9 - 7

Figure 02-10jb Cosmic Standard Ruler [view large image]

Gyr since BB). Diagram a in Figure 02-10jb shows the various lengths of the cosmic rulers as defined by pairs of galaxies at different epoch (or redshift). Diagram b in the same figure illustrates the change in such length scale (represented by the white circle) over time (in unit of billion years ago).

Figure 02-10jc below summarizes the evidences for the presence of dark energy. Case (a) uses the type 1a supernovea as cosmic standard candle to discover the cosmic acceleration as it is dimmer than expected (for an universe with no acceleration). Case (d) uses the supercluster size as cosmic standard ruler to provide an independent check. Case (c) relies on gravitational lensing to show that matter clumping is hindered by cosmic repulsion. Case (b) happened much earlier and much faster at the beginning of the cosmic expansion, it may nor may not be related to the repulsive "dark energy" at the later epoch.

		Gradually, it dawns on some astronomers that dark energy could be responsible for turning off galaxy and star formation in the latter half of the cosmic history at redshift z ~ 0.75 (~ 7.2 billion years since the Big Bang). The central piece of evidence is the rough coincidence in timing between the end of most galaxy and cluster formation and the onset of the domination of dark energy. Both happened when the universe was about half its present age. The influence of dark energy include stopping the merger of galaxies, sorting out the types of galaxies, lowering the rate of star formation, and preventing the growth of galaxy clusters (Figure
Figure 02-10jc Dark Energy Evidences	Figure 02-10k Galaxy Formation & Dark Energy	02-10k). Such idea has been confirmed in 2008 by NASA's Chandra X-ray Observatory. The X-ray results on the hot gas in dozens of galaxy clusters some of which are relatively close and others are more than halfway across the universe reveal that accelerated expansion stifles the growth of

galaxy clusters. It also tentatively identifies the cosmic constant as the dark energy.

The nature of "dark energy" is still the subject of intense research observationally and theoretically. Some of the suggestions are listed below:

Cosmological constant - Eistein had introduced a term with the cosmological constant in the gravitational field equation to keep a static universe from collapsing. This additional repulsive force is no longer necessary when the cosmic expansion became apparent. It has become fashionable again with the new discovery of cosmic acceleration. It is very tempting to identify the cosmological constant with the vacuum energy of the various quantum fields. However, the simplest versions of quantum theory predict far too much energy - 10¹²⁰ higher than the observed value by one estimate. One explanation involves the cancellation between the boson positive contributions and the fermion negative contributions to almost zero, leaving only a residual trace corresponding to the observed dark energy (an idea from supersymmetry).

See "Vacuum Energy Density" for 2015 + 2024 updates.

Figure 02-10l Dark Energy & Fates of the Universe [view large image]

Quintessence (most perfect) - This hypothetical form of dark energy permeates all space. Like inflation, quintessence is thought to have somehow originated when the universe was just 10^-35 sec old. It is driven by a scalar field whose energy varies gradually. The difference is the energy and time scale: inflation occurred quickly at very high energies, whereas the scalar field responsible for quintessence operates at much lower energies over a much longer time frame. One of its key differences from the cosmological constant is that it can vary depending on time and place. Figure 02-10l portrays the fates of the universe according to different scenarios of dark energy. Research in 2006 found that the dark energy's influence on the universe in the current epoch is the same as 9 billion years ago. The discovery rules out quintessence models that change too rapidly.

Update 2020 - Detection of polarization of light twisted in a particular way from the 2013 Planck data has reverted the quintessence back to vogue albeit with only a weak statistical significance of 2.5 sigma - such results often fade away on further scrutiny. But could be a big deal if it is real (see "Hints of twisted light offer clues to dark energy’s nature").

Chameleon Theory - The theory is a variation of the Quintessence, which varies in time. This fifth force however depends on space in such a way that the range of the force is shorter in an environment of higher density. For example, the ratio of density from the vicinity of the Earth to the void of the Cosmos is about 1/10^-28 leading to a ratio of the force ranges of about 1/10²⁶ (Figure 02-10m). The virtue of this theory is that it can be tested by a number of observations including the modification of fine structure constant, change of the ratio between the mass of electron and proton, additional polarization of star light, variation on the age of the universe estimation, ... because of chameleon-photon oscillation (switching). Such flexibility now becomes the problem for validating the theory. Since this theory is conceived to fit observations and has yet to be derived from anything more fundamental, it is very easy to adjust its parameters to fit the available data.

Figure 02-10m Chameleon Theory [view large image]

Is Dark Energy a Chameleon

Illusion - This explanation asserts that the apparent acceleration is just an illusion. Sky survey reveals that matter is distributed unevenly on large scales with gigantic super-clusters and huge voids in between. Because we live in a gravitationally bound system (the Milkyway), our clocks run more slowly than they would in a void. In addition, space is negatively curved in the void, so the volume for a given radius is larger than in the relatively flat space. In effect, our estimate of volume is too small and the estimate of time is too slow giving the wrong impression of acceleration (see Figure 02-10n). However, slight increase of the WMAP temperature associated with intervening regions of superclusters shows that the dark energy is

Figure 02-10n Dark Energy Illusion

real. Since the CMBR photons gained a small amount of additional energy as they re-emerge from a gravitational well while the cosmic expansion is accelerating.

Void - If the Earth is located in a vast cosmic void (with less matter than the average) of the size between 300 million to 3 billion light years, then object outside the void would be further away (than envisioned from a homogeneous universe) because the void would expand faster with less gravitational retardation (see Figure 02-10o). The problem with this explanation is the requirement that the Earth has to be in the middle of the void (in violation of the Copernican Principle); otherwise it would be inconsistent with the WMAP data, which is isotropic. Measurement of the rate of cosmic expansion over time can be used to check against this hypothesis.

Figure 02-10o Earth in a Void [view large image]

The observational sensitivity required to record the tiny changes is currently beyond astronomers' capabilities. But it should become feasible with a new generation of ultra-sensitive telescopes.

Citing some seemingly inexplicable anomalies in astronomical observations, it is suggested that the "Void Theory" would remain viable even the Earth is slightly off center (up to 50 million light years). It has been shown that the CMBR is a bit lopsided - hotter in one direction than in the other. This asymmetry is usually attributed to the motion of the Solar system through space but could also be a sign of a lumpy universe. Furthermore, small fluctuations in the CMBR appear to align in the specific direction (the "axis of evil"). This alignment picks out a preferred direction in the sky, which, though hard to imagine in a Copernican universe, might be explained in terms of displacement of the Earth from the center of a void.

A preferred direction would also have other effects, such as large-scale coherent motions of galaxies and galaxy clusters. Several observations have claimed detection of such "dark flow", but it remains controversial. And then there is the argument that the "Void Theory" would not violate the Copernican principle (that the Earth is not special) if the region under consideration is very large and containing many more voids such as the inhomogeneous universe shown in Figure 02-10p, which also depicts the different explanations with dark energy and void.

Figure 02-10p Inhomogeneous Universe [view large image]

A 2011 study claims that along a direction at about 10 degree to the Milky-way's spinning axis there are more left-handed spirals (in the northern sky), while the opposite is observed in the southern sky. It is found that such axis is roughly in the same direction of the "axis of evil" mentioned above. The researchers claim that the universe was spinning at the the moment of Big Bang leaving some marks in the CMBR and the handedness of the spirals (Figure 02-10q). It is further speculated that an initially spinning universe brought on CP symmetry violation in gravity, which produced gravitational waves asymmetrically. Its interference with the inflaton field biased the production of matter over antimatter. This process left three marks behind : the axis of evil in CMBR, the inconspicuous alignment of the axes of rotating galaxies, and the all-matter universe. On top of all these speculations it is claimed lately in

Figure 02-10q Spinning Universe [view large image]

2011 that the cosmic expansion (in term of Type 1a Supernova observations) seems to go faster near the direction of the axis of evil.

Then in 2012 a different scientist using a different technique and more samples found a similar excess of left-handed galaxies, but along an axis some 85^o away (Figure 02-10r1). It is argued that since both axes have such large uncertainties that they could be aspects of the same axis. This kind of statistical error is very un-usual especially when the two directions are almost perpendicular to each others. It is equally unbelievable that the universe spins in two different directions. The observed asymmetry could be due to subtle side effects of sky scanning or data read out from the camera - none of which has been analyzed in the survey. It is suggested that future galaxy surveys imaging 10 billion or so galaxies would resolve the issue.

Figure 02-10r1 Spinning Universe 2

Long EM Wave - In classical electrodynamics, the electromagnetic wave equations emerge from the Lorenz condition contain four components. The two transversal modes are used exclusively for optics and quantum theory. The remaining two are the longitudinal and temporal components, which are combined together to give rise to the instantaneous static Coulomb interaction. However, each of them does satisfy the wave equation and can be in the form of wave. According to the latest research in 2012, these kinds of waves could be generated in the episode of violent expansion called inflation.

Figure 02-10r2 Long EM Waves [view large image]

The calculation shows that if the beginning of inflation happened at an epoch after or close to the "electroweak transition", then the energy density of the temporal waves (with wavelength longer than the size of the observable universe) would be in the same order of magnitude as the observed dark energy. The problem is on the timing, it is not in agreement with the current re-construction of the sequence for the cosmic events (see "A History of Cosmic Expansion"). It is suggested that the ESA's Planck spacecraft might resolve the contradiction or the calculation might need some numerical tweaking. Actually, the computation is rather complicated since the electromagnetic field depends on the metric tensor, which in turn has to be evaluated with the energy-momentum tensor of the electromagnetic field (among other things) in the gravitational field equation, i.e., they are coupled in a nonlinear way.

In addition, the longitudinal wave could also be generated in the presence of very strong gravitational fields such as those near the black holes or via inflation. The wavelength of such mode would be longer than the Sun-Earth distance of 10⁸ km. It could be much shorter if the inflationary epoch kicked in earlier at an even higher energy. That would bring them within reach of Earth-based technology as such the Square Kilometre Array (operational in the next decade). The detection of such kind of waves could account for the mysterious magnetic field that seems to pervade in the cosmic voids.

2021 Update on Dark Energy Survey (DES)

Weak gravitational lensing provides an important mean of studying the mass distribution of the Universe including dark matter. This is commonly referred to as the cosmic shear power :
S₈(

) =

₈(

_m/0.3),
where

₈ is the amplitude of matter fluctuations,

_m the cosmic matter density parameter, and

~ 1/2. Thus, if

₈ = 0, the matter distribution would be smooth without any fluctuations.

The Dark Energy Survey (DES) measured 300 million galaxies and their red shift to produce a 3-D map in a segment of 1/4 Southern Sky (Figure 02-10s,a). The number density n_eff together with gravitational lensing enable the determination of the fluctuation parameter

₈ and the shear power S₈ (Figure 02-10s,b). The result shows that the universe appears to be slightly smoother than the the value from CMB measurement by Planck, 2018. They also conclude that the dark energy density is constant over the eon.

Figure 02-10s Dark Energy Survey [view large image]

See "The most detailed 3D map of the Universe ever made" for more non-technical details.
End of 2021 Update.

See "A Novel Idea on Dark Energy (2022 Addition)" for a new scenario about Dark Energy.
End of 2022 update.

The effect of dark energy became dominant only at an epoch about 8 x 10⁹ years after the Big Bang. If the acceleration persists in the future, it will impose a horizon surrounding a galaxy like the Milkyway - a distance beyond which light cannot reach us. Figure 02-10t depicts the sequence of events for the future of the universe with cosmic acceleration according to a computer simulation (click image to obtain larger view). The model assumes that the dark energy permeating the vacuum has a positive, constant value - similar to the cosmological constant, as Einstein once posited.


	Figure 02-10t Future of the Universe

A study on the consequence of cosmic acceleration concludes that while most of the galaxies move away beyond the cosmic horizon, the

		local group of galaxies will collapse into a supergalaxy by gravitational attraction. Eventually, the universe goes black when the last stars burn out. Future civilizations (if there is any left) will have a very different perspective of the universe. Our descendants will observe an island of stars (the supergalaxy) embedded in a vast emptiness. It will resemble the de Sitter universe originally envisioned by Einstein. Figure 02-10u shows the sequence of events according to an artist's rendition.
Figure 02-10u Future of the	Milkyway [large image 1, 2]

Figure 02-10v shows the 2014 version of a "timeline of the universe" from a special issue of the "Astronomy" in September 2014. The timeline starts from 1000 years after the Big Bang (see the "Cosmic History Table" for textual explanation). This timeline contains more details on the epoch in which life is possible somewhere. Beyond the death of the Sun and the merge of the local group of galaxies about 100 billion years from now, the portray is mostly conjecture depending on the nature of dark energy which is still an enigma right now.

Figure 02-10v Timeline of the Universe [view large image]

Since we don't feel the effect of dark energy and dark matter around us except through the gravitational influence on large scale, a model has been constructed with no other interactions between each other or with ordinary matter. It fits the observational data such as the high-redshift supernovae, the microwave background radiation, the distribution of large-scale structure, and the dynamic of celestial objects very well. But if 96% of the Universe is in the form of unseen substances, does this not mean that there is the possibility of hidden structure? Might the dark sector be a fascinating place, with its own intricate interactions - perhaps even a kind of intelligent life? Is there a 'dark light' that we do not see, radiating and absorbing in the dark Universe? Such possibility suggests that human beings are extremely unimportant

Figure 02-10w Insignificance [view large image]

in the grand scheme of the universe as portrayed in a 1985 movie called "Insignificance" in which Einstein and Monroe explore relativity and our place in the universe (Figure 02-10w).

[2025 Update]

Here's the updated information via conversation with ChatGPT (in Italic text)

What is the status of the search for dark energy by 2025?

ChatGPT said:
As of January 2025, the scientific community continues to explore the enigmatic nature of dark energy, the force believed to drive the accelerating expansion of the universe. Recent studies have sparked debates, with some researchers questioning the very existence of dark energy.
A notable study analyzing supernova data suggests that the observed acceleration might be attributed to variations in the kinetic energy of cosmic expansion, rather than an unknown energy form.
In contrast, other research supports the presence of dark energy but proposes that it may not be a constant force. Findings from the Dark Energy Spectroscopic Instrument (DESI) indicate that dark energy could be evolving over time, challenging the traditional cosmological constant model.
Observational missions have made significant strides in mapping the universe's structure to shed light on dark energy. The European Space Agency's Euclid probe, launched in July 2023, has begun unveiling detailed images of the cosmos, aiming to understand the distribution of dark matter and the influence of dark energy.
Additionally, the Vera C. Rubin Observatory is conducting the Legacy Survey of Space and Time (LSST), which will provide extensive data on cosmic phenomena, including insights into dark energy's role in the universe's expansion.

In summary, while recent findings have introduced new perspectives and challenges, the quest to comprehend dark energy remains a central focus in cosmology. Ongoing and upcoming observational projects are expected to offer more data, potentially leading to a deeper understanding of this mysterious force.

Why is there such a huge discrepancy in observational data and theoretical calculation (based on Vacuum Energy Density)?

ChatGPT said:
The discrepancy between observational data and theoretical calculations of the vacuum energy density (often referred to as the "cosmological constant problem") is one of the most significant unsolved problems in physics. Here's an overview of why this discrepancy exists:
________________________________________
1. Theoretical Prediction from Quantum Field Theory
Quantum Field Theory (QFT) predicts that empty space (the vacuum) should have an intrinsic energy due to quantum fluctuations of fields. When you calculate the vacuum energy density by summing up the contributions from all possible quantum states, the result is enormously large-as much as 10¹²⁰ times larger than what is observed in the universe.
________________________________________
2. Observed Value from Cosmology
The observed value of the vacuum energy density, inferred from the accelerated expansion of the universe (attributed to dark energy), is incredibly small compared to the QFT prediction. Observational data (from sources like the Cosmic Microwave Background, Type Ia supernovae, and galaxy surveys) indicate that dark energy constitutes about 68% of the universe's energy density, corresponding to a very small value of approximately 10^-29 g/cm³.
________________________________________
3. Why Such a Huge Discrepancy?
The problem lies in the scale mismatch between the two predictions:
a) Cutoff Scale in QFT (see Dark Energy Calculation)
The theoretical calculations in QFT integrate contributions from all possible quantum fluctuations up to the "cutoff scale," which is often taken to be the Planck scale (10¹⁹ GeV), where quantum gravity effects are expected to dominate. At this scale, the predicted energy density is catastrophically large. It seems nature somehow cancels most of this enormous energy, but the mechanism for this cancellation is unknown.
b) Fine-Tuning Problem
If the vacuum energy is indeed the source of dark energy, then there must be an extraordinary level of fine-tuning to reduce the theoretical value by a factor of 10¹²⁰. This extreme tuning is unnatural and raises questions about the validity of the current understanding.
c) Misunderstanding of the Nature of Dark Energy
It's also possible that the "vacuum energy" explanation is incomplete. Some alternative theories propose that dark energy arises not from vacuum fluctuations but from:
" A dynamical field like a quintessence field.
" Modifications to gravity (e.g., theories like f(R)f(R)f(R) gravity or extra dimensions).
" Something fundamentally different that we have not yet conceived.
________________________________________
4. Lack of a Unified Theory
The inability to reconcile the theoretical prediction of vacuum energy with the observed cosmological constant is partly due to the lack of a unified theory of quantum mechanics and gravity. Without a complete understanding of quantum gravity, we might be missing key physics that would explain why the vacuum energy is so small.
________________________________________
5. Anthropic Principle
Some theorists invoke the anthropic principle, which suggests that the observed value of the cosmological constant is small because only in such a universe can complex structures like galaxies, stars, and life exist. While this idea doesn't provide a physical mechanism, it offers an explanation based on the conditions required for observers to exist.
________________________________________
Current Efforts to Resolve the Discrepancy:
" Improved Observations: Missions like the Euclid probe, the Vera Rubin Observatory, and DESI are providing more precise measurements of dark energy to refine our understanding.
" Advances in Quantum Gravity: Efforts to develop theories of quantum gravity (e.g., string theory or loop quantum gravity) might provide insights into the nature of vacuum energy.
" Alternative Theories: Exploration of dynamical fields or modifications to general relativity continues as potential solutions to the problem.
________________________________________

In summary, the discrepancy reflects a fundamental gap in our understanding of the interplay between quantum mechanics, cosmology, and gravity, making it one of the most profound mysteries in modern physics.

Any form of dark energy, whether constant or variable, will doom the universe to frozen (heat) death?

ChatGPT said:
Yes, dark energy - whether constant (a cosmological constant) or variable (a dynamical field like quintessence)-will ultimately lead to a "heat death" scenario for the universe, though the exact pathway depends on the nature of dark energy. Here's how it works:
________________________________________
1. Dark Energy as a Driver of Eternal Expansion
Dark energy is the name given to the mysterious force driving the accelerated expansion of the universe. This accelerated expansion means that galaxies, stars, and other structures are moving away from each other faster and faster. Over time:
" Matter becomes diluted: As space expands, galaxies and other structures move further apart, and the density of matter (both normal and dark) decreases.
" Energy becomes unavailable: The universe's temperature steadily approaches absolute zero as energy becomes more evenly distributed and less usable for work (a hallmark of thermodynamic entropy).
________________________________________
2. Scenarios of Heat Death with Different Forms of Dark Energy
The specifics of the universe's fate depend on whether dark energy is constant or variable:
a) Cosmological Constant (?)
" If dark energy is a true cosmological constant, its density remains fixed over time, even as the universe expands.
" The expansion will continue accelerating forever, with the universe asymptotically approaching a state of maximum entropy.
" Over trillions of years, stars will burn out, black holes will evaporate via Hawking radiation, and only a sparse, cold, and dark universe will remain-this is the classic "heat death" scenario.
b) Quintessence or Variable Dark Energy
" If dark energy varies over time, the outcome depends on its dynamics:
o If dark energy weakens over time: The expansion might slow, but the universe would still approach heat death, just at a slower rate.
o If dark energy strengthens (phantom energy): This could lead to a "Big Rip" scenario, where the accelerated expansion becomes so extreme that it tears apart galaxies, stars, planets, and even atoms.
________________________________________
3. Why Heat Death is Inevitable with Dark Energy
The key idea behind heat death is entropy:
" The universe is heading toward a state of thermodynamic equilibrium where all energy is evenly distributed.
" As the universe expands faster, it becomes harder for gravitationally bound systems to exchange energy or interact, accelerating the loss of "useful" energy.
Even in the absence of dark energy, the universe would still eventually reach heat death due to the natural progression of entropy. Dark energy simply hastens the isolation and freezing of the cosmos.
________________________________________

However, in early Universe :
Dark energy's effect was negligible during the radiation and matter-dominated eras because their energy densities vastly outweighed that of dark energy. Only in the relatively recent cosmic past, as radiation and matter densities dropped due to the universe's expansion, did dark energy become the dominant driver of the universe's evolution.
Depending on the cosmological model, the universe be closed (meaning collapse back to zero) or expanding to heat death as depicted by the most up-to-date observation with 70% dark energy.
See evolution of "scale factor" below :

[End of 2025 Update]