Bottom Line: Apparently this Chayei Adam is based on the Arizal, so presumably it's kabbalistic. If you try to give a rational explanation to it, it creates problems. There's also a dispute if the Arizal means at the beginning of the fifth hour (two hours before chatzos), or after the fifth hour (an hour before chatzos, like the Chayei Adam understands).
I heard a recording from Rav Hershel Schachter on the International Date Line where he discussed this Chayei Adam. You can listen to the relevant part here (1:03:48-1:10:44). The following is a summary.
The Arizal is quoted by his student Rav Chaim Vital in Eitz Chaim Sha'ar HaKavanos Inyan Vayehi Noam s.v. ובתחילה and ואחר קריאת as saying that on Friday, already starting from the fifth hour of the day, the holiness of Shabbos is present in the world. He says from that point on a person can go to the Mikveh and accept Shabbos upon themselves.
Rav Moshe Shternbuch in Moadim UZmanim 7:236 in footnote א quotes from a Kabbalist named Rav Yosef Shapira zt"l, who explained the Arizal to mean that since in the far East parts of the world, namely Australia, they accept Shabbos eight hours before Jerusalem (12 hours into the day for them), that means at the beginning of the fifth hour of the day in Jerusalem (as the Arizal lived in Israel, so that was his point of view) there is already somewhere in the world that Jews are keeping Shabbos. That brings the holiness of Shabbos into the world, allowing everywhere else to accept it as well (this all assumes that the Arizal held like the Chazon Ish about where the International Date Line begins, which is a whole separate shmooze).
Rav Hershel Schachter points out that this idea also appears in the Mishneh LaMelech's book Parshas Derachim § 23 s.v. עוד נקדים. He brings a Midrash מעולם לא זזה שכינה מישראל בשבתות וימים טובים ואפילו בשבת של חול, The Divine Presence never left the Jewish people on Shabbos, Yom Tov, and on the Shabbos of the Weekday (Zohar III parshas Korach p. 179b). What does this latter term refer to? He quotes his father's teacher Rav Zerachia Guto who says that in Heaven, Shabbos starts when Jerusalem starts keeping Shabbos. Since there are parts in the world that start keeping Shabbos before Jerusalem (i.e. Australia), that's called the Shabbos of the Weekday. Although it's still a weekday in Jerusalem, since there are Jews in the world observing Shabbos, the Divine Presence is with the Jewish people. We see that there's a holiness brought into the world when somewhere is keeping Shabbos.
Rav Shternbuch adds that according to this explanation (which he adds has no support in the poskim), people in Europe can accept Shabbos already after only a few hours into the day (8 AM), and more surprisingly people in North America around Midnight the night before Friday. He doesn't discuss it, but I wonder what would be in Australia itself? Maybe this is only lekulah, for a leniency, but it doesn't create stringencies (as according to this, they can't accept Shabbos until it's sunset).
Rav Hershel Schachter uses all of this to explain the Chayei Adam. He says the Chayei Adam is coming from the Arizal. Even though we don't say you can accept Shabbos so early, we do say you can go to the Mikveh starting then. Since the Arizal says the holiness of the day starts already from the fifth hour (although the Chayei Adam understood this to mean after five hours, which is an hour before chatzos; if you look at the Arizal the way we have it, he explicitly says the start of the fifth hour), therefore that's the time one can go to the Mikveh in preparation for that day. The same is true for Rosh Hashanah. Rav Hershel Schachter points out that according to this Rav Yosef Shapira, in America they wouldn't need to follow the Chayei Adam/Mishnah Berurah not to go before an hour before chatzos. Rather, they could go to the Mikveh even in the middle of the night, since somewhere in the world it's already Rosh Hashanah. Although, that's clearly not the practice.